All right, hello everyone, hello, hello, my name is Jacob, I work at CloudFur, thanks for
joining me on this talk about attaching metadata to packets.
Here's our agenda for today, we're going to first start off with what's our motivation
behind this project, and then I'm going to take you through what's it's not quite a
saga yet, but a trilogy where free attempts we had so far to get this upstream.
First, quick, probably announcement, we're looking for a lot of interns this year, so
tell your friends, look up the blog post that tells you all the details about the program.
Right, so what are we trying to do? First of all, how many of you know what an SKB or a
socket buffer in the Linux network stack is? Okay, okay, that's not bad. All right, so our goal
is to allow users, Linux network stack users, and their BPS programs to be able to attach
hundreds of bytes of metadata to their packets as they travel from an network stack to
do different cool projects. Whereas today we're limited to just, you know, a few bytes.
And this work is driven by a few very concrete use cases, we have in our production.
This first one being that on our receive path, we need to reclassify the packets at every stage,
and the configuration propagation time for each of these stages is a little bit different.
So there's some inconsistency, trends and inconsistency that bring us operational pains.
So we would ideally would like to classify the packet only once, early on, perhaps in our XDB program,
take it with a service identifier, and then all the later stages of the network stack
would just look up the service identifier and say, oh, I need to run rules for service X, right?
Our second use case comes from the fact that we have many, many data centers,
and we do traffic engineering between them sometimes. So if we're one of the data centers
is getting overloaded, we're going to take part of the traffic and encapsulate it and forward it to
another location to relieve the overloaded color. The problem that stems from that is when
when we lock through which color, through which data center request came through,
we get them misattributed, right? And that messes up our capacity planning.
We could easily solve that if, because we know from which data center of the traffic was for
that we could extract the information, attach it to an incoming request, and then if it could
travel together with a packet up to the application that actually processes the request,
the application could read that information out and log it and it will be golden.
Our next use case that we have in mind is just making a service life easier.
Cloud for in a great simplification is a proxy that sits between users, which recall eyeballs,
these will be your web browser, your apps, calling KPIs, and we just forward the traffic to our
customer service, which we call origins. Well sometimes we inadvertently drop packets,
and it would be really helpful for us to know whose packets are we actually dropping.
So in an ideal world where our proxies, when they are forwarding messages, they could label them
with a customer identifier, put them on the transmit path, on the network stack, and then if the
packet happens to get dropped, we could have a BF program attached to the function that goes to a
packet that would just read out the customer identifier and count it log it.
And our last use case probably most complex one is keeping track of how the requests are
traveling through the network stack within one's one host. So inside our machines we will
actually usually have a chain of proxies, and packets might get encapsulated, redout user space,
kind of over a unique socket connection, and it will loop back across network namespaces.
It's really hard to keep track how a very precise traveling. If we could tag it with some kind of
trace and identify it with survive, this journey then it would make debugging problems much easier.
All right, so how do we get about it? How do we attach metadata to packets? Well the first idea
we had was called SKB Traits. But before I can talk about SKB Traits, we need to talk a little bit
about the layout of the packet buffer. So when you have XDP programs attached to your Nick device,
the driver will allocate a buffer for the packet data, but it will leave a bit of a
headroom in front of the packet data for very stuff. It will be usually 256 bytes.
Now this packet buffer later on when we allocate VSCB will actually become VSCB head buffer.
But before it becomes that XDP will put its stuff in front and your XDP programs might
write some custom metadata in front of the packet data. The problem with that existing
metadata feature is that you can't access it, pass the TCPPF hook. And in addition to that,
it will get just crumpled overcrop that as you push heavier on your packet.
So our first idea was, hey, let's fix that by introducing a brand new area for metadata.
We call it Traits. That would be accessible from all the stages of network stack.
And it would be a new reference of the headroom. So it wouldn't be in the way of pushing new
encapsulation headers. But the feedback we got from for that from upstream was that we don't need
a second metadata. We already have one. It would be maintenance burden. So fair enough, but of course
sadness. So but we've worked out and since we're told that we already have a metadata area,
it's only made sense to look into whether we can make it work for us. So how do we make XDP
slash XDP metadata work and not break any existing users? Well, to talk about that first,
we need to learn a little how your BPS programs, TCPPF programs, access this metadata.
They actually do, they'll do it by accessing it through that data meta pointer,
which is not a real pointer that exists in your SKB. It's actually just a pseudo pointer,
that we compute for the benefit of your BPS programs. And it's relative to the SKB data
pointer, which points at the beginning of the packet payload. So what's the problem,
well, as your SKB travels through the networks like layers. And we move the SKB data pointer,
your data meta pointer, style of sporting in the at the wrong place, right?
So to make this scheme work, we would need to keep copying bytes over every time we're
in both SKB data pointer, which is not ideal. Solution, well, let's just hide away the fact
how we access the meta data by introducing an abstraction. And in this case, a useful
abstraction that we applied was BPSB pointer, which you can think of as a slice,
which underneath has a pointer and a length. In our case, it was a pointer that was relative
to SKB hat, or actually relative to the MAC header. Why relative to the MAC header? Well,
that's simply because of the packet that Linux network stack already tracks the position
of the metadata like that. So we didn't have to do anything.
That got in. We made some progress. Then we had to fix some BPS helpers. I'm not going to
talk about that, but we made some more progress. And then we encountered another issue. Well,
if you have layer two tunnels in your network setup, guess what? We're going to change the MAC
header offset as they encapsulate the packets. So now you're slice to your metadata, again,
points to the wrong place. So what's the solution? Well, we can keep track of whatever metadata
is located separately from the MAC header. We need to store more offset for that in SKB.
And that's quite an ideal, because this allows us to relocate the metadata
anywhere we want with the header. We could push it to the top of the header, which looks a lot
of like threads, right? It's not ideal, because the data make a pointer is not going away.
It's part of BPS API, and we still have to support it. So for some programs, we'll need to keep
copying the metadata. But overall, we can make this work. So this is what we are proposed that
metadata could just leave anywhere in the header. But that feedback we wrote, this feels like
duct tape. This is one of solution, please use SKB extension instead. All right,
sadness again. I guess not the XTP metadata, XTP, the SKB extension. So what is SKB extension?
So adding new fields to your SKB socket buffer is a big no-no, because of performance reasons.
So smart kernel engineers do the pointer at the end. That can point to an additional buffer
where various features can store very stuff like MPTCP or crypto flows.
So we could just store our stuff in an SKB extension, but probably not the metadata directly.
We don't want to embed it in a SKB extension, because this buffer for SKB extensions gets allocated
as soon as any of an extension gets activated, right? So if you're using MPTCP, but you're not using
metadata, then you would have just wasted an allocation for VET memory, right?
We could allocate space for metadata separately, and I just point to it from an SKB extension.
But this feels just like another metadata, which we're already tried with trade, and it's
we would, it feels like we would be repeating the same mistake. So let's go for something more
general, and here's where BPS local storage comes in. So BPS local storage is a established concept
with a BPS ecosystem and a Linux network stack, which allows BPS programs to attach
arbitrary data to be common entities that we have, like C groups, tests, sign-out,
sockets, and each of these objects has a pointer to BPS local storage, named one way of the
other, that will actually point to a list, not a byte array, which will have a list of storage
elements, and that's because you can have multiple users attaching their data to tests,
C groups, and you want to accommodate them all. And the way your BPS programs access the storage
is through a BPS map, abstraction, which identifies your storage and defines your storage geometry.
So you can have like users of storage and two corresponding maps, right? So let's have a pointer
to BPS local storage in our SKB extension, and your programs then can access it from new
map type called SKB storage. And that's all neat, that needs our requirements, but to actually create
such storage, you'll have three different locations that you need to do. And because of that,
we expect it won't work out of the box for all the use cases we have, especially both that
involve tagging or labeling every packet that passes for very C for transmit path. So expect
like a long tail of optimizations before we can actually make this work for us.
Number of less, I've tried it out, I've tried it out just this week, and a prototype seems to be
working, passing some tests that implement some of these use cases, I've talked about. So please
watch out for the patches appearing on the net depth and BFF mail in place. And we'll see,
as this is this finally, when we get rich pocket map data, I guess only time will tell.
All right, I'm leaving you with links to everything I prefer to hear. If you have any questions,
feel free to shoot me an email. And once again, I'll remind her that we're looking for
interns. Thank you. Any questions? Okay, I just want to dare. Thanks for your talk, just the
remark, keeping eye on the number of extensions because we from the Ken subsystem are about to
add one soon as well. And I think there was a limit of it or something. I'm not sure how much
is used. So the nice thing and something that also needs improvement about extensions is that you
can select which ones you include a build time, right? Yeah, but the conflicts are not very
granular today. Like say you, you're not interested in using the crypto offloads because you don't
have neck hardware that supports it or you just don't use it. Yeah, but you and all of
yes, conflicts should work. And it should not break because nine bits do not fit in in you eight.
So yeah, that's true. I mean, maybe it will come to that, but you know, you have to pick and choose
with which extensions you build your kernel or maybe we just add more bits. I think we're still
whole in the extension like header. So before that proceeds with chunks. Just just a heads up.
Thanks. We were discussing if some extensions are mutually exclusive just use the same.
Makes sense. Thanks.
How about casual quality if you're having the two drivers, the pointless. I guess it's pretty painful for
cashers. Yeah, I would imagine so as well. I don't have any branch marks as far as
what overhead is. We just expected to be painful, but we'll have some benchmarks for RFC.
But you know, we've tried doing it without allocations, reusing the free space and the pocket
buffer. We couldn't make it work. Any other questions? All right, thank you. Thank you.