But yes, let us introduce ourselves. I'm Daniel Medado. I'm a software engineer. I
re-hat and I'm also training training. I'm trying to run up the Fedora's EBPF
secret. We will discuss another later. Basically, what we want to just speak about here
is about the sciences that we have been dealing with when we were trying to package EBPF
programs and Fedora. While this may see no, I thought when I first started to
this, it has another program. Let's go package it. It's going to be a spec
file and that's it. It was necessary forward. So, our plan is to go through a little
bit about how this works. It's time to have fun while doing so as some solutions.
And afterwards, we'll be happy to have some discussions probably offline because of the
time constraints. This is my colleague, Mikael. He's also involved into
the packaging and also working for Rehat and I'll let him start. Thank you.
Thank you, Danny. So, good morning, everyone. So, our plan is to talk a little bit about
why we think that packaging EBPF can be hard. How we plan to solve it and we will
be giving you a list of checklists that we created for when we work packaging EBPF
applications. So, EBPF is here. We think it can be a packaging name for some people.
So, the thing is that EBPF is no longer optional. EBPF is everywhere.
EBPF is already running in the most of our systems through system D or using
different CNIs or whatever that we are using. But the thing is that the
use cases are growing a lot. And we want our
distributions to become part of the EBPF platforms. I would say like,
how we as a distribution packages can take EBPF applications to the users. So,
usually the, okay, the packaging patterning for traditional software was to take the source
to compile it, create a binary and ship it. We were relying on non-library, like
lip-c, lib-c, lib-accessary, cool or whatever, right. But in the case of EBPF
software is a little bit different, because the kernel itself is the library. It's a dependency.
And the kernel has not a stable ABI so it can be a problem, okay. Also, while we are trying
with EBPF is to try to inject kernel models that behave as a user space application. So,
that's something new for many people. But at the end, we think that many users,
expect that they can just run DNF or whatever tool they use to install packages in the
distributions and get the same feeling or experience with installing it.
Their reality behind that is that, as I said, kernel structure changes, but if
structure changes and so on. And they also, we may have some capability issues that we need to
define that this can make a C, or a Selineux policy, I think, like that, that they use as
might find complicated to do it by themselves. So, the stacks are high. If we do it wrong,
since then others will break whatever software we are running, monitoring, or whatever the EBPF
application was doing, we will fail and users will lost the trust in the distribution.
But if we do it correctly, we think that EBPF might be kind of another boring infrastructure
technology, like, and like we are doing with other languages like could be go or asked in
Fedora, for example, we think that it should be installed once and run forever, and our goal at
the end, we will show with that. What we are trying is that this will become EBPF friendly, like
another C descent. So, the main highlights is that the old way, the EBPF compiler collection that
was supposed, you were meant to use that at runtime to compile the applications that were
that you were planning to use, the EBPF application you were planning to use at runtime,
and the problem with that is that that requires a lot of software to be installed in the system.
Like, we are talking about C language, LLVM, and so on. And usually you don't want to have that
kind of applications in running server. It takes up to, in the case of Fedora, it was more than two
gigabytes, and it's a lot. So, as a distro, what we want is to have just a small pre-compile binary
that has minimal runtime, that will check everything for you, that will make you update and all the
stuff easy. So, we need to go to the second part. The second headache is that, as I was mentioned,
the cannot doesn't have an ABA stability. So, extract that, we might depend on my change between
between releases. This is one example of how the process is. The process extract that is defined
by the cannot changed between release 5.15 and 6.8 with PID, feel that change the position.
So, if your application were running, what's really in that position 16, it will
re-guarbet with newer kernel. So, we need to take care of that. And then we need to make sure
that the matrix of toolings that we are using is the correct one, that's something that
distributions helps a lot to do, because it's not easy to maintain the compatibility between
the application, the BPF, BPF tool, C-Lang, and the kernel. So, that's where we think that
the distribution matrix can help. So, for that, we will rely on core and Daniel will talk more
about that.
Okay, so, as my colleague was saying, like, this is not really a stick. So, I would like to
be in this distribution, not even again, to have in it to recompile every BPF program that
I'm going to be packaging, every time I just go and upload the kernel. So, BPF provides
the new tooling, which suits work in every model, Linux kernel distribution, starting in
67, which is called BPF Core, which stands for Compile once around everywhere. So, I want to
know if it can be seen there quickly, it doesn't click it well. But basically, let me try and
be brave. So, here we got the source code. This is our, yes, BPF program. We are going to be using
C-Lang and BPF, which stands for BPF type format. This is just some kind of metadata, which
would make use of kernel header information, and that would stop us having to rely on kernel
headers on their running system. What does that mean? That means, let's say, we are going to
be compiling this using, you see, see, or whatever, and the end we are going to be having
this file, which would be workable in any kernel. So, not only, like, I wouldn't have to
go over the compiling dober, and this is also really important when we think about the
Fedor infrastructure, we got codians of course, and I would like for the codi on any build system
to have it to go and download a lot of dependencies, yes, for even building, and even while
building is fine, I would like for me to have it to go to a production cloud, and having to
have all those dependencies in runtime. So, then goes BPF, and BPF would read this, BPF, which
I said, you just submitted data, and would rewrite all these issues with the headers that we
were seeing in all those lights, and so then we wouldn't have a kernel pane, basically.
Was that happens? Our binary would be working into all the kernel distributions.
So, one else. Yeah, so here, as I was saying before, a compiling time, if our codi is using
core macros, we can just rely on here. Let's say, again, getting back to the example of the
we were speaking before. If we are needed in the pit here for the data struct, that's not going
to be here coded. It's not going to be a stored as the BTF header here, we made a data, and in the
end, it would just work the same Fedor after nine, forty, forty one, relying or open success,
what probably? Basically, any Linux with a modern kernel, I guess it's started in Fedor
specifically in 39, but I need to check probably in 32. And how does it work? We'll be BTF.
It would reinvent the metadata, which is stored here, but away. It would find a new struct here,
and it would be fine, and wouldn't do any kind of kernel pane. So, this is a little bit
the same as I was saying before, like probably better explain here. So, we have our source code,
and we can just put it here. The PTF has a score read. It would be read by Siulang,
it's going to be compiled into our final guide. And this byte code would be now fine,
because it wouldn't be just going in trying to serve this specific pit. Then, when the
hour BTF code is going to be logged in, relying on BTF, also the kernel would be fine, and our byte
code would be just passed and it would work across any kernel. So, before,
BTF core or BTF, we were, again, relying on all these dependencies, even in run time.
Now, we all need them for in build time. And in our packages, we will need to do any kind of build
requests anymore, and yes, our requires. This is the, sorry, again. Awesome. Well, I'll be going
over there because I can't read. So, it's not there, it's the, it's a, it's a, it's a,
a tournament. So, this is like, basically, the equivalent of dynamic linking, you'll see that
on those slides, it's going to be available and post a link. But anyway, it's also helps us
like if there's any CBL, say, for the BTF, we wouldn't have to write that, it's going to be dynamic
how does it work for Fedora, specifically? So, all the interest ready, as I was saying,
this works Fedora 39 onwards and so forth. And if you, well, let me see here, please
stop flittering for a second. Anyway, anyway, anyway, so here is, well, if you can see that,
BTFL system is a BTFL system, so it's exposed as it's off there, and you'll be able to
get that it's here, and you can just write that, and you're going to be happy and like the
BTFHeaders here on the kernel, it'll be online. And as I was saying, the dynamic, well, how does this
relates to our spec file getting back to our, so we shouldn't have to rely on this static
table, it's because, again, when I'm just shipping any RPM in the absence case, I wouldn't like to
see this specific metadata here on as a table there with a spec file, but as we are relying on BTFCore,
we can just go and use kernel call and kernel labeled here internally, and this would be just
getting the one for the proper one and pass it.
All right, same thing for like the environment, so in Koyumog, slash Cs would reflect the
build stop journal, and as I was saying, I would like for these to be happening in Fadoris
infrastructure, because I would like for these to be having to, let's say we got this build note,
which is Koyi, it might be Fador 43, run like whatever, I don't care, whatever we happen there,
but what happens if we've got a target of Fadoro height? I mean, this kind of wouldn't be fine,
because this would be pre BTFCore, and this is direct kernel, and the code used,
would use in that, which is good at it, and 60, what happens? Yes, it would fail, and boom,
how do we do that? Again, using BTFCore, we would be able to get the kernel version directly from here,
and you just start to compress kernel, and you might use in this active version,
and we would be generating this header, using BTFCore, this BTFCore command, basically,
regenerate the BNLunus.header that we were speaking about before.
And then we had him back to Mikael for some examples, and we even got a demo, let's see if the
Flictros allows us. Okay, so for the sake of this presentation, we created a small BTFCore
tool that is called ProcessSnub, that all you will do is print in the screen, all the commands
that are launched in a system, okay? So we are not doing in the scripting way, I was telling before
the BCCA way, like we are not doing it on runtime, so we are doing this tool mode, we will have
several dependencies, so that I'll lip see a BTFCore, I will generate a small 20k binary, and
I will have everything on it, okay? It will be, it's just this simple script, it relies on the
bridge, the headers that we are generating in the script, that I will, the spec that will show you.
Now, okay, but that's the important part, okay? The first line, but the script is just this
to create a BTFCore application, so what we call the goal and stand out, or what we do
the whole somehow, expect would be this one, okay? Everything is linked in the presentation, so
we will have the bill requirements, bill requirements, to build the application, that will be the
compilers and the kernel core kernel level, and we will require just libbpf, as said to be able to
launch the Bbpf applications, then as Daniel said, we are going to look after the kernel that we
need to use and generate the headers there, and then we will run this bpf tool to create a
code for the, sorry, to generate the headers, I wonder now, sorry, sorry, we will compile the
bcode of the Bbpf, then we will create the headers of that Bbpf applications, that when we will
use GCC to link against the libbpf and create the final binary that will be running. Then
there is a full script after we have two options, I mean we can just create the install the file,
the binary file has been created, we can't say the capabilities while creating the
spec that my allow regular users to just run it, the option is just to launch it using system
integration with a regular service, the important part is that we need to define the capabilities
and the ambient capabilities, okay? The good thing about using this kind of spec is that we
will allow us to run the same spec and the same application against different many different
kernels, okay? So sometimes we might be tempted to to pin whatever releases we want to use to
create the bpf because we might have read a C-lang by code might change between versions or whatever,
but we think that is good idea to have us open spec as possible and they rely on things like
Koshae or Koji or whatever to find the problems I would call my half, okay?
So when that's the base, of course things doesn't work as smooth as we want to do,
and we found a couple of edge cases, first of all, like some stuff it depends on the
bpf problems themselves, so bpf uses the map which is a memory structure that is basically
using between both user space or internal space, and what happens if I just go and update
maybe bpf tool? It may be the case, that's kind of interesting here anyway, not really, but yeah.
So if we stop a service, my bpf service may still be pointing to the old map, the new service
starts, then boom, how do we handle that? So click there, yeah. So this is an upstream requirement,
it's something that it relies on not only of the spec file, but on the bpf code or the developer
application, so bpf programs must be written to handle that and reuse existing maps.
It's, this is an example using yes cbpf library, bpf, but yes a requirement, not only for spec files,
but just over lbpf programs. Then I think this is important, and I was having another set
in another session yesterday, which is like, you cannot get the ability to use. So when we first
started doing bpf programs, we were using basically, copy-satmen, this is legacy, don't use this,
is basically covered, you're giving root access to your bpf program, which you may not want to.
Another option would be, let's assume we are going to have a packaging and xdp program,
we would need like, capperf mom, kbpf, and capenetalmen, okay pretty much like root,
there's a new change that we are going to be proposing to fedora, which is like the
system-wide usage of bpf tokens, which they don't only use kernel capabilities, but anything
called delegations, and that means that as a root user or a set admin, you would be able to
allow like non-root users to basically run a specific bpf programs, which is specific and
more, much more grain like permissions. I'm happy to discuss that as we are also running the
bpf system group and that's something that we are going to be proposing as a system-wide
since we're fedora 44 or 45, but anyway, just FYI, this may be an issue, just go ahead.
Also for, as is the disservice, we would need to take that into consideration just for the
amount, this may be enough, but again if it's just an xdp program, it may not be enough,
we may also need to get different capabilities for both time,
positive and of course as elinux, which I guess is the next slide.
Yeah, here we got as elinux. So far by default, if you're too advanced in a confidiment,
as elinux would just deny all your bpf operations, and depending on what you want to do,
it may be also the next. You would need to go and add a companion or another option that we were
just thinking about is to add some global system-wide system policies for bpf, but that's a
no topic. Yeah, this is just an example. Well, of course, we would also need to detect the
kernel. I was speaking about bpf core, which I said it was totally okay for different kernels,
but what happens if your kernel doesn't support bpf core? As I was saying, you would need to go
and recompile your application for every different kernel pre-core, and so forth. And even if
that's somehow fine, basically you need to be aware of that when you're packaging, if you want
your package to support lex resistance. So, basically using require kernel higher than 6-1, should be
fine. Um, this is what happened to one of the kernel updates. Again, core comes to the risk you,
anyone just worked, but if we got some lexic kernels, it may or may not work depending on how
they maps on headers have been changed. If you're just lucky, it may just happen to work,
but it doesn't have to be. So, this is something that you should take into consideration
we created your spec file for lexic systems. So, um, I'm going to be handing over down to
Michael Lee and let's see if we got them of them. Okay. Yeah, that's right. Okay. So,
this is the checklists we are following, uh, spark edges. So, before packaging, we need to check
if the application supports core. If, uh, make sure that, uh, it doesn't require any specific
LibbPF to, uh, that is ban the load that is statically, uh, require, request statically compile,
uh, LibbPF. It's important also that, uh, LibbPF will check what kind of license the application
have. So, it's important to check that everything is right at the level. And also that the headers
are generated at bill time. So, make, we can make sure that we are using, uh, every release,
every version, every kernel version generated here. This is really important. And then,
after we create a package, uh, it's important that we serve the correct, uh, minimal capabilities
that, uh, say Linux, uh, policies created if it's required for that, that applications that we
have to comment in any change that is, or any change that might affect, any change in the kernel
that might affect the package and so on. So, it makes maintenance much easier. And then, as we said,
it's important that, uh, we tested it in many different, uh, releases to make sure that we are
not breaking anything. Uh, quick reference card would be the spec that we already show on the
system, the essential just as a one as a light of, of how we should be done. Uh, yeah. And last,
and just quickly, uh, we as a LibbPF's group, we are working in both adding new macros to the spec
to support directly these ebbF's new tooling and spec files and bbf token as I said. And we're
going to be proposing a new changes for, uh, like for 244 onwards. So please join our group if you
want to help us into that and we are totally open for contributions and, um, you know, basically
feedback about that. So, with that, we are running out of time. So, thanks for attending and we are
up and through any questions. Thank you.
So, uh, you move to another solution is we are actually seeking the new stage that corresponds
to the kernel of the machine as if a package. I think it's a lot of those the same and we're using
the system to give you. So I'm wondering why you are advocating for everything that everyone
can. Yeah, I was going to answer the question. So in the book, he won't do, uh,
is already packaging bm Linux headers. Uh, why are we advocating to create, uh, in, in the
spec itself again and again? Um, that's basically, uh, I mean, that could be also done,
but in, I was speaking specifically about the use cases for ebbf program packaging when
not using bbf core because basically it wouldn't just be able to reuse that directly or not
without some specific things. What may happen is that if you don't use bbf core is that your
change in that and you might need to renew that for your specific packets. I mean, uh, I can give,
I mean, we can just discuss that offline because we're running out of time, but that's also an
option that we would like to discuss in this new proposal. We are speaking. Thank you.
Any other questions?
Yep.
So is the are we already shipping this? My question will be which part of it. So no, no, you can.
So we, uh, we created already as I said, uh, let me check. So call this already available in
a small repo. So this is the bare minimum. Sorry, uh, it's not okay. Yeah, so we have an example.
We have also created. So it's not that one, but we also created a copper repo to build it
against different versions and so on. So it's already, you're already able to use it. The thing
is that the macros are not there here. For example, the mixed thing is here, but if you work,
you can create a spec and the applications and so on, if you'd work. For example, this, this
little application, we just compiled yesterday. We built it against different federal releases
and Apple releases and we are able to run the same application with the same spec. In federal
43 and in, I was in all my nine or something like that yesterday. So we have a lot of time, but we
get to be on site and if anybody wants to keep it that, then we'll be right back. Thank you all.
Thank you.