Thank you all for coming, so my name is Mike Bullard, today I'm going to be giving
a talk about the Neckhead device, okay, the Neckhead device is a new networking device
that was introduced in the Linux kernel that allows isolated networks in containers without any
performance benefits, so I'll get into what that means in just a minute. I'm going to try to
go through this pretty quick, I wanted to say thanks everybody for coming out today and especially
thanks to the volunteers for helping to put this together and for staying all day in the room today.
So some background, kind of why do we want this Neckhead device? So I'm a software engineer at
Meta and at Meta we tend to run all of our services in containers on a host and so in this
diagram we have a pretty typical setup, the host is connected to a physical ethernet card which
is connected to a network switch and then we could have containers running, dividing up the
resources on that host, more and more we want to run multiple services on the same host.
This is more efficient if we have like CPU that we're not using container A is not using it,
we can load up container B which might have like a batch processing job or something to use
that resource. The issue that we can run into here is like what if container A tries to buy into
this IP and port and container B wants to buy into the same IP and port that's going to cause
a conflict. So ideally we wouldn't have to deal with this at all and this is where network
name spaces really help us out. So a network name space basically gives a container its own view
the network basically its own network interfaces, a set of ports and its own IP tables so it kind
of has its own network connection completely separate from the other containers and the other
network name spaces on that host. And so here's what that diagram looks like now with network
name spaces implemented. So container A is running inside network name space A, container B is running
inside network name space B and the way that we kind of like hook these up so that they can see the
world is we create a virtual ethernet device in the network name space and we create a matching one
on the host and then we basically run like a virtual ethernet cable between those so we wire those
together and that gives us network connection to the container and now this works we can now buy
into any port on here any port on here they don't need to know like what the other container is
doing which is great but like many things and engineering we've solved one problem but we've
created a new problem which is there's a performance implication here so if you kind of like
walks through the path of a network packet coming out of this container first it's sent you know
the job sends some some network packet out it's got a transmit from that virtual ethernet adapter
it's going to go through that virtual ethernet cable it's got a land on the host like virtual ethernet
gets received in that VEV and then that gets received by an IRQ and then like finally it gets
transmitted out of the host so there's many extra hops here compared to when we didn't have
network name spaces and we're just using the ethernet adapter directly and so what we see in practice
is when we enable network name spaces we're not able to like utilize the full network performance of the
host and I'll get into a little more a bit like the specifics but this has caused you know issues
with jobs that are very network intensive you know of course they they need to use the full you know
the full available network performance so come come come to play this net get device this
was introduced and I think November of 2023 and it was really designed around solving this
exact problem so this is a new type of virtual network device that is built with like container
ization in mind and what you see here is when you create a net kit device in this host network
name space it gets a kind of a built in peer net kit device you can almost think of it like
a backpack attached to that primary device those are linked together you know behind the scenes
and it kind of eliminates that extra virtual ethernet cable so and something that's kind of
important to note here is whenever we manage this net kit device it's done from the host side
that's a big part of the reason why it works you know it's it's all being managed as a device
on the host but it's able to kind of peer into this network name space and interact with the
traffic over there and of course the other thing that's important and relevant to the stock is
this is all done using BFF programs which we attach to this net kit device so there is a
net kit attached there's a BFF attached point at the PO side and a BFF attached point at the primary
side and by attaching BFF programs we can direct that network traffic tell it where to go
and this is where it's going to save save us some performance so it's just so kind of what
that looks like now the packet gets sent from the container again like the job just transmit the
network packet it lands on the net kit TX side we have a BFF program running there which can inspect
the packet if it determines that the packet is going to be routed somewhere outside of the host it
can directly just place it into the transmit queue for the physical Ethernet device so it just removes
a lot of extra hops in there and the way that we see it show up is in like the soft IRQ usage
and so with the what we see is the soft IRQ usage kind of gets overloaded for jobs that use a lot of
network traffic but with net kit that's greatly reduced and much more network traffic can get through
in addition to the performance benefits we also see a lot of benefits from using BFF a big one is
just it's much faster to develop and debug these BFF programs so it really gives us like a low level
view of what's happening to that network traffic we can set up you know detailed logs and monitoring
in that BFF program and kind of have like I think it like an x-ray view of like what's going on
in the driver and then we can identify if there's any issues or if there's any performance improvements
you want to make you know it's very quick to write a new BFF program load it tested out and then
deploy it so this is a huge benefit over you know in the past where we had to write a curdle driver
and then recompile the curdle reboot into the curdle this is much faster and allows us to like implement
new features and fix any issues much faster. So I mentioned before the performance implications
so this slide here is showing some you know like experimental benchmarking that was done
so set it to ISO variant in this blog post they used this iper tool to just send as much
network traffic as they could over these different types of network adapters. Here the first
chart here is the VEF that was only able to get about 60 megabits per second compared to the
baseline which is you know around 100 megabits per second so really we can only get about 60% of the
host performance we can also use BFF with VEF and we can apply some of the optimizations there
even with that we can only at best get up to around 90% of the performance of the host
but with net to NBPS we actually can match the host performance so this was proved out in this
experimental testing we've also seen this happen now in like real world jobs so this was one service
at meta that was experimenting with this you know stacking multiple tasks on a single host
and so this graph represents the you know latency of the requests coming out of that host
with VEF the white line with VEF enabled they were seeing around like 12 seconds of latency
compared to the the baseline what we would expect is around 100 milliseconds and when we looked into it
we found this was caused by that high soft IRQ usage so essentially the soft IRQ the queue was
filled up and it just wasn't able to get any more network traffic through when we switched it
over to net kit we saw return to that the baseline that we expect 100 milliseconds so this is kind
of a real world demonstration of the benefits of net kit so you know in comparison we're in the
process of rolling this out I wanted to give some shout outs of course from ISO valen they were very
key in getting this authored and merged into the kernel and also sharing all of that knowledge
so there's no way we could have put this together without the help of this open source
community and having all of the code available to read through and there's so much you know blog posts
and information out there that we can refer to and within meta talks up and Martin are both you know
key members of this network and team that that really helped me through the work in this through
and so yeah and conclusion the network name spaces it's really a key feature that we want to enable
in order to be able to isolate tasks so these tasks don't need to know anything about the
environment they're running on they can use the network as they want to and with net kit we can
turn on that network name spaces feature without incurring any performance penalty and finally
BPS has really allowed us to move much faster and implement new features much faster
as we you know work through implementing the features so this is the conclusion of my talk here
and just wanted to open up for any questions I put up this quote which supposedly was said by
Beethoven I couldn't confirm it but it's that was pretty good so with that that's all I had
I don't know if you'd like to answer any questions
So further received path we we didn't have the performance issues you know mainly because we can
we can pick up that packet right away as soon as it comes into the host physical adapter
and we already had BPS programs running that would redirect it to the current container
you know so in that case like the host is aware of all of the containers and can redirect it properly
but the traffic coming out it was that VE in the container that didn't really have you know
knowledge of where that packet was going so it had to go through the the that software Q pass
and yeah just to repeat the question was do we have the similar performance impacts on the
ingress side for traffic coming into those
other person
the VPF code that you have for next kit to be extended to expect packets for purpose
yes yes I mean this is the big this has been a big benefit for us is like when teams come to us
with you know a specific use case like they have this specific type of network traffic
that they want routed in a particular way we can work with them to you know more quickly
get that implemented you know through these BPS programs and you know one thing that we do is
you know like for specific packets we can redirect them you know directly to the to that
networks which so kind of like bypassing all of the you know internal code that's running on that
that host that may be affecting that that packet we can descend directly to the to the
the racks which so yeah so the question was can we with the packet device can we implement
business logic in the BPS programs and yeah the answer is yes so the packet device has new
attached points but they're very similar to XDP attached points it's just a different place
in the driver where it's attaching so kind of any logic that you could do an XDP program
you can very easily change it to be like a net kit redirect program and then attach it just like
a normal BPS program.
So not not that we've seen so oh yeah so the question was were there any impacts to the container
tasks like did they have to be aware of this and the basic answer is no so we have we have made
not getting out of the default choice for tasks that enable network name spaces and we didn't have
to work with the teams to you know make that switch it's basically a transparent switch the it's a
transparent change for them to just swap out that device I would say the one team that has come
come to us is they had a specific debugging thing they were trying to do where like then they just
knew more about VEF because it looks like any other you know network adapter they specifically
wanted to go back to VEF because they were more familiar with how to attach BPS programs to it
and how to debug things but that's really been the only time that people have come to us
to be requests to go back and I think that will change you know as as there's more knowledge
and more internal tooling for the net kit because again like within that kit we actually can
add more monitoring and like more debugging to is there any impact on inner container is there any
impact on traffic going between containers and the answer is mostly no I'll say there's one
exception not due to net kit but due to network name spaces which is like if tasks were
communicating over local host like if container A and my example wanted to communicate with
container B specifically because they were related somehow in the past teams I've got new
using local host to be that communication but we can't do that with a network name space because
local host is its own local host in the network name space so that's been the only thing that we've
had to you know move teams off of that so yeah perform with what is very similar like it has the same
we have the same kind of redirection happening that we had before where we'll detect if something
is going inside the container and it will get routed the same way it was before not yet for the
performance benefit is it because of the PBS program or the net kit device like what brings which
them in the performance and the results you have so the question was about where does the
performance benefit come from is it the BPS program versus like the net kit device so there's
many people that could speak more qualified end of this question my understanding of it is
it has to do with like where the BPS point attaches so we're able to pick up the traffic at an
earlier point in the transmit process and so with like existing BPS at test points in like
attaching to like the TC attached point the packet has kind of already gone through some of the
network processing on that VE where as now we can kind of like pluck it out of that trend like
a hook into the actual driver is transmit call pick up the packet and put it onto the host
so
I think it was like what are the requirements to enable that kit
so yeah I said that you know like the requirements to enable it is you know that the big one is
having the kernel support that's really the only kind of hard requirement you would also you
would also need to have the BPS programs written so one difference with BPS is if you don't have
or it's like one difference with net kit if you don't have a BPS program attached by default
it will just drop all traffic you know the idea is to prevent traffic from accidentally going out
before the device is configured so you would need to have a BPS program set up to do the
forwarding of the traffic without that BPS program it would just by default it would just drop all
of the packets so other than that there's really no like technical requirements there is like
you have to make sure like the IP tool has been updated to a new version that have
net kit support and like BPS tool those tools they've all been more recently updated with net
kit support so to run a lot of those like command line tools you just need to make sure that you
have the most updated version so the question was like did we encounter any limitations
using net kit compared to like XDP so far I would say no you know we're still kind of in the early
stages like we may discover more as we go on the only kind of got to that we found to look out
for is like making sure that if there is like additional BPS programs that were like in a chain
on the TC like attached to the VE if those may not get run if you if you have the net kit
program like forwarding things off so that's the only that's the only