Hello, what can everyone thank you for attending, so I'm Fernando, I'm a software engineer
at Rayhead and today I'm going to talk about how do we perform link aggregation, balance
SLB or source, what balancing, income and space within a moment of year.
So just a quick introduction, so your quality familiar, but it's always good to remember
what is link aggregation.
So in essence is a technique used in networking to bundle multiple interfaces into a single
device that acts as a single link.
So usually you have ports that are physical links or there might be balance if you have
a virtual network, but in this case for example you have tunings and you have a bond and
you code use the that bond as an endpoint for your application and this can use to have
a full trance environment doing a lot of balancing and some other operations.
And usually these links are connected to a switch or they are connected to a different
ISP or things like that.
So this is usually called bonding in Linux because it's the driver implementation that
we have and there's also a team but it's mainly bonding.
Then you have multiple nodes and options which is what is going to define, how it's going
to operate and what kind of behavior you're going to get from the bond, we are not going
to focus on the options because there are a lot of them and for our talk is no they are
not very very important but we are going to focus on the mode.
So yes, we have around seven modes officially inside the kernel so the first one it's
balanced from Robin as it sounds it just do wrong running so it goes sequentially picking
every port.
Then you have active backup, this is pretty straightforward, you have one port that is active
when there is a fail over it's switch to another of the port's configure.
Then you have ballang XOR with this is a very important one for us because it has a trance
meat hash policy so in essence it will perform blood balancing but it does based on a hash
policy that you can define as an option and the default policy is basically the sold
Mac address, XOR, the destination Mac address, XOR, the packet type ID so we did you get
identify of the connection and you make sure that connection that connection that matches
the hash goes the same way the same thing all the time.
Then you have rooks, you're pretty obvious it's a rooks so it trance me all the packets
through all the ports all the time.
Then you have 8, 0, 2, 3, A, D so this is a standard it's a dynamic link aggregation
and in order to operate in it supports on the switch side on the remote switch otherwise
it's not going to work.
In essence it's dynamic link aggregation based on the doublex and speed settings on the
mix and other algorithms and stuff that can be configured on the switch side.
Then we have balanced deal with which is basically adaptive the trance meat balance in
so based on the trance meat blood balancing that the neck has it's going to measure it
and try to have even lowered on the different ports and then we have all adaptive trance
meat balancing which is basically the same until we plus it has adaptive it also adapts
on IP4 in the receive transmission.
Let's create an interaction to link aggregation and now I want to do another important
interaction which is OBS I'm pretty sure that you have heard about OBS in this room
a lot during the day but I think that's what is OpenBSwitch so OpenBSwitch it's a software
switch in communication which is usually used as a switch into VMs environments so you
usually have a lot of VMs and you have OpenBSwitch connecting all of them and doing
other operations like bonding which is also supported but in this case OBS and OBS
bonding so OBS bonding only supports three modes LACP so LACP it's a protocol that
allot the switch to cooperate when you are performing bonding or link aggregation so it requires
LACP support on the switch in the remote switch once the negotiation is done there is
no special treatment for the packet then we have active backup same as the kernel one it's
one port up when there is a failover another port comes up and then we have SLV so that's
important one for us today as you can notice in essence the two first modes are supported
in link in Linux implementation but not SLV what is SLV so SLV is the lot one
seen without the remote switch knowledge or cooperation and it's basically using a
hashing like plexx or but this hash is based on the source MAC address plus a
BN per so this is usually used in spying live situations where you have when the
leaf has multiple VMs inside and you want to perform you want to perform what
one C between the VMs inside the every machine so what is our goal as you could
see OBS support this but the Linux kernel doesn't so we thought the number of
managers could help here and find a way to implement this in conversation this
can be useful for situations where you have a system where you don't want to use
OBS or OBS is not available or maybe you are ready to implement everything with
Linux pages and bonding for Linux and you are using some options that are not
supporting in OBS and in essence you don't want to move everything to OBS the first
thing that we need to do is of course take a look to the driver in the kernel so
we need that some changes initially we thought that balance XOR was the ideal
mode for S because it was doing already some it was based already it was the
balancing based on a transmit hash so it was ideal but there is no hashing policy
that matches what SLV is doing so we asked for help and we get some kernel people
working into it and they did an introduce a new XMid hash policy option which is
Billan plus SRC Mac and this is what it does is you get the Billan ID XOR
the source Mac vendor XOR the source Mac dev and if we go to the code this is
indeed quite neat I hope you can see sorry for the code but in essence what it
does is that it gets the source Mac vendor it gets the source Mac dev checks if
there is a Billan ID if there is a Billan ID it just XOR them if there is a
Billan ID gets the Billan ID and XOR all of them together just that that's
policy the rest of the implementation for this bond on kernel was just the
Nellink bits to be able to configure it and see some effects on all the
configuration options so that you will we are ready at shift
balancing SLV and this is what we thought but no that's not the case we noticed
that all BS did and also work solving some challenges that we found and we
took along today to how they solved it and it was impressive so we started
just to think on how we could do that we could do that and we thought that
Nephid RF tables is the right candidate to help us here so let's take a look
to the challenges that we found so first of all there are two situations where
we can get packet application so the first one is when you get a broad
multi-guess from the remote switch so in essence the remote received from a
non SLV port broadcast or multi-guess packet and therefore the switch forward
it to all the ports in SLV and then you have in your bonding you have
duplicate the packages because all of the all those are going to reach the
bond to solve that and if they were was a good fit because we could
radiate everything to the primary port and make sure that the broadcast
a multi-guess if the connection is already hash get you can check if this
broadcast or multi-guess check if the connection is already hash and then
send it directly to the primary port and if it is not the primary port we could
drop the packet and then a situation is that the broadcast multi-guess
from the bond port so the remote received the remote switch received a broadcast
multi-guess packet from SLV port and it forwards all of them to all other links in
the SLV so you get the same packet again and you get the application same again we apply
the same technique and f-tables with check if we hash the the belong ID
on the source Mac and if we did and it is not the primary port we can drop it and
this is in essence I think yeah so the first thing that we have is we use dual
sets of NF tables the max set pack so if we have a belong ID or we don't have
a belong ID use two different sets so the first set is a combination of the source
address plus the belong ID and the second set is just the source address then we have
three chains so the first chain is in essence just updating if we
the sets that we define above nothing else the two sets is basically one of
then is dropping loop packages and the other package and the other one is dropping
a broadcast packet so if we locate a detail we check if the set already contains the
the combination and then we do some drop in and yeah on the yeah and then on the other
one we do basically the same which if it is a broadcast or a multi-guess and then we
drop one if you take a look to it the first one the one that is updating the set is on
agres because we want to update when the interface is transmitting and the other one is on
ingres so we check when the packets are arriving so we found one more problem
IGMP and NLD so this is usually used with top of the rack switches so we noticed that
top of the rack switches may prune the multi-guess tree and they sent their announced the
different multi-guess group and so on and we noticed that there are problem because if the
belong SRC Mac happens and the prune multi-guess is sent to a board member where the
RX filter will drop it we will lose the the prune and therefore we noticed that the only
the primary member board should accept this again NF tables we created a close set it's a chain
basically it checks agres and it checks the IGMP type depending if it checks it sorry
it checks a membership report and then send it to the primary board so in this case for
example we have this rule for ATH2 which is not the primary but ATH1 is so before what the
packet there and the same for ECM PD6 we check if we have NLD listener and then we do a forward
by the way the counters that you might see there is because I was running this in debug in
one manager showing debug it at some counters so it's easy for us to debug any bug that we might
find and we found one more which was a little bit troublesome and if you're familiar with
the link aggregation you're familiar also probably with this so in a spinoleaf environment
architecture when there is a link fail over and you are doing balancing it's probably
it's very probable that the switch is not going to update the Mac table on time until there is an
outcoming an outcoming connection from the bond and there is some moment where you lose the
connectivity between the switch and the bond for that specific port and the traffic is not
already detected we found that OBS already fixes I'm wondering why we don't do this on canal
I think I need to talk to some people because this should be doable on canal and it will work
for all the bands mode and yeah so by the way this obviously happened if you have if you don't have
an ACP because if you have an ACP the switch will coordinate and this won't happen so with all
of the with all of this by reading the Linux switch FDB table sending a reverse ARP on the link
fail over on link fail over event and we were updating the Mac tables on the switches so
in essence what will happen is that you have usually this architecture with different
balance in the bridge connecting to the end for example only can be best does a matter
then you have the bond and the trivial next that is connected to different switches
that may or not or may not talk to each other and then you have something as in this case I
define the same architecture but it could be something else if there is a connection a string of
connection going from link three through the switch one to link one sorry from both one to link one
and link one fails for whatever reason switch the bond and the switch two does not it that it can
reach the Mac address of the bond through switch two and therefore the connection is not
redirected it's a little bit cumbersome but it happens and we got back reports that people
that were experiencing this and we initially said yes this is supposed to happen it's a proven
non-bonding and they replied it doesn't happen in OBS we investigated and OBS fix it so
coolest OBS so we fix it and how we did all of this so we integrated everything with
never manager so we introduced a new bond option in a manager which is called balance SLB so in
essence you just need to configure balance XOR and the transmit policy to BLM plus SRC Mac
and then this option to balance SLB one when balance SLB one is set to set in the configuration
never manager do two big things so apart from configuring the the bond that is not a special
it also monitors the link uh or carriers status of the different ports and according to the
need it's modified NFT all rules said that I show before so if there is for example you
value your primary and you're already connecting the EGMP all the melody to the primary
and the primary goes down for whatever reason the rules said is going to change immediately
to redirect the queries to a new primary and then basically it's an NMCA command that you can use
so it has an NMCA LI connection a type bond connection name whatever if anything whatever
and the bond options mode balance XOR XMB hash policy balance SLC Mac balance SLB one
and with that you should be all set of course you need to configure on the other side the ports
as you might want but this is the bond configuration for simplicity I don't know if you are
aware about what is NMCA state but in essence it is a command line tool and also with an
accompanying library that's added you to perform a host networking configuration in any declarative way
so this will be integrated with the NMCA state it is never manager as a backend so this is the
configuration that you will apply and with this you will have everything set of course notice that
here the controller is BRT0 and BRCO you should have configuration before maybe you don't have
a controller or maybe the controller has a name or whatever bonus point so while we were doing this
we noticed that when updating the NF table rules set sometimes we need to make sure that we
clean up the whole table and for that we did NF tables delete and try to delete the table for
every interface that we have and we were getting an error and within understood why until we found
out that NF tables delete fails when the object doesn't exist and that's a problem when you are
scripting so our way to fix this is that we implemented in NF tables a new destroyer
operation that a boy doing at and delete all the time to avoid this delete problem so if you
are not sure that an object exists in NF tables now you can do NF TV destroy whatever and
it's not going to fail so for example you have here NFD destroy table IP and amissing table
it's going to return CO error code and if the table exists it's going to delete it if it doesn't
exist it's not going to do anything at all so yeah that's an extra contribution that we were able
to make and it was pretty nice that when we did it we were reach out by other people that it was
really useful and people was looking for something like this before it was pretty simple to implement
so that's a nice thing important remarks when use connected to obvious breach the link failure
issue will be there and this has main reason is that all BS maintains its own FDB table so
and right now there is no way that we can delete it from the API so it's only possible to
delete it with a command line to OBS APP CTL but not with with OBS DB JSON RPC so we cannot
read it at this point also we think that's kind of fine because we are expecting that if you
have OBS in place use the OBS bonding because probably it's going to work better together
than mixing the Linux bond and the OBS breach so yeah that's mainly in tiny important remark
the bonding itself with work that the failed or bear situation will be there if you don't mind
it's completely fine to use but again I when using OBS I personally recommend to use
I not an OBS developer or anything like that but from experience everything works better
if you use the whole suite as much as you can instead of mixing things
yeah thank you for attending thank you for the organizer thank you very much