I will be talking about python bpf, what is it and how it works, but before that yeah a quick
introduction about me and Varun. So, I am Pragyansh, I work at canonical in Ubuntu engineering,
I am also the co-mentainer of python bpf which has nothing to do with my day job and Varun
engineering student who works with me on python bpf, he could not make it here because
of some reasons, but yeah another person I forgot to include in this slide, but I have
a master acknowledge a scarthic here, a PhD student at e-mfl, who helped us with adults with
ebpf and was very helpful in guiding us through this project. So, yeah the project python bpf is
a python front end for writing ebpf programs. It allows you to write both ebpf and its corresponding
user space code in python and they can be in the same file, they can be in different split
across different files, it can also be in a python notebook across multiple cells. Now writing
ebpf and python is not a new idea and might sound somewhat outdated as well as bcc or
already existed in like what 2015 16, but how python bpf does this is very different from bcc.
So, if you have used bcc for writing ebpf programs before, you might know what the might
know that you still have to write the bpf specific part in a multi line string if you are using
it in the same file or have to have a different file and then reopen that and read that
in your python file and your ebpf specific code is written in a flavor of c and only the
user space processing part is in python. In python bpf the users use a subset of the existing
python grammar to write ebpf code and this has some benefits like python dev tools like
lenders and pre-setters are useful for the whole file now. This also potentially lowers the barrier
of entry for people to learn ebpf by hiding a lot of typing details and hand me in common
verification failure mitigations behind the scenes. So, this is a side by side comparison of
the same code written in bcc and python bpf. This was one of the examples in the bcc tutorial.
You can see the dev tools at work with proper syntax highlighting as compared to the bcc
example given on the left side. This is somewhat superficial when it comes to benefits as we will
discuss what else python bpf enables in our latest slides. Let me give you a basic overview of
this program. At the top we have the various imports of python bpf the decorators, the helpers,
the maps and the type hints which are necessary. Then we declare a custom data type data t
which has a pid timestamp and the command which is a caraday. Then we also have a perf event
array map type bpf perf event array and then there is the function which attaches itself to
says endoclone trace point and then outputs some data to the perf event array map. So,
yeah that is the basic that is what a python bpf program would typically look like. Let us move on
to how do we compile this. So, this is a big image and I wish I could employ you all to remember
this for the rest of the presentation but we can come back to this slide again if we need it.
This is what a comes firstly as a one time step we have to generate the VM Linux.py VM Linux
header. So, we generated using a script that we have written that runs bpf2 to create VM Linux.h
and then runs crank topy on it with some tweaks to give us VM Linux.py so that we can use your running
kernels data structures in the bpf code that you are writing. Then we can take the input as a python
file or python notebook source and then bpf chunks are marked out and processed by python bpf.
What are bpf chunks these are just code snippets which are which will be like compile to a bpf
object. So, if there is a file that contains both if bpf code and user space code
we just we are just concerned with the bpf chunks rest of the code is run through any python
interpreter. So, here and then we first look at VM Linux imports and how they are being used.
So, if there is any kernel data structure or kfong or something that is present in VM Linux.py
imported from there and being used there. So, as a first step we create the llvmir for
that said import and then we create the symbol table for those imports and
like inject them into our local and global symbol tables for the input file we have
which will be populated with other program logic later. Then we do struct processing
our custom data types you might have declared and then comes map processing where
we generate llvmir for any maps you might be using we do this in this specific order because
you might be using those custom struct types in your map declarations and might be using
of course, always using those maps in your functions that you will be writing.
So, this is the order we follow. Then comes the real part function processing where we
do a pass to kind of guess how many local symbols you need and how many local
how much local stack space you need and then assign those stack spaces to symbols at times
you can have the same stack space for multiple symbols fill in type metadata and then process
all of the expressions one by one here. So, conversion of Python ast to llvmir is done by a
project called llvmir which is number you are thinking of using just llvmir but we figure that
let us try with llvmir it allows us to write everything in Python and I do not know this goes
in with the spirit of the project. Then now that we have that i of file we can just pass it
well c and it gives us bpf object file. We also have a sister project called pile bpf which is
essentially just python bindings for llvmir and we can pass on the
struct definitions that we created during this pass and pass that on to our user space code
which is using pile bpf. So, you do not have to declare the same structs twice and then use
those map and struct definitions in your user space. So, yeah that is the compilation flow.
Now, I will go through what is the anatomy of python bpf program through work through. So,
this is the disk snoop example which we have ported from bcc to python bpf and so yeah
first we take in the c types import. These are necessary even though type hints are
optional in python for python bpf they are necessary because you need to compile it down to llvmir
and there are some places where you just cannot guess types after some point. So, these are
common and also like when you are creating suppose this hash map which is a map of type bpf hash
it is not the data structure hash map. You have to give the key and value types and they could
have been like instead of equals we could have used code in but that is something that we
talking about in the next slides. Then there are some imports from rvmreenox.py struct ptx which
is the type of the context argument pass to functions attached to k probes then there is a request
struct which is a common data structure in vmreenox header. Then we have the decorators bpf bpf
global compile is not a decorator but map and section. These necessary to point out the bpf chunks
and what kind of bpf chunk there then we have the k time helper which maps to bpf k time and the hash
map type is imported from bpf.maps there we have listed other map types there as well. So yeah
next we have what a function body would look like. So first if we have to attach a function
to like section we provide that section using the section decorator of course the bpf decorator
should come first to mark that this is to be compiled down to bpf object and I am not quite
happy with these syntax we use in the section as the argument for section and I will come to that
later but yeah then we have trace completion this is I think I have shown this before and this
zoomed in part of that. So struct ptx as I said is the type 4 context and this function returns
an n64 at the end I have to add a type ignore return value because my pie complaints that 0
is not c n64 but Python bpf internally would assume a constant n64 if no type is specified and
so returns 0 is actually c n64 my pie doesn't know it so for satisfying it we have to add it
you know so yeah then we have context or d i context if we just take the type of context or d i
in lvmir it will come out to be n64 so but the user knows that it can it can we convert it to
a pointer to struct so you just do struct request get the request object from it and then
data land command flags request or data land request command flags there is also this
interesting bit that we do not know if data land and command flags are pointers or pointers
to pointers or integers we will just use a dot to d difference it to any depth level and yeah
this is part of the type deduction we have to do then another interesting bit is start dot
look up so in c for example if you were looking up if you were doing a map look up that would have
been a stateless function where you have to pass the map as well and the argument but I think
for pie then it makes sense to make it look like start is a data container and you can have a look
up on it you can also overload the dundam method for indexing and then you have start square
brackets request pointer and that does it pretty well then we have the usage of k time minus
it equates tsp and this would be interesting because the type for k type k time and request
tsp does not match here what still it works and then we have print which allows you to use python
like format strings and print is actually bpf print k so you would see there are only three
arguments because we cannot do four so that is the and I did last the I won't explain trace
start because I think the an explanation of trace and covered it the things happening here pretty
well there is this part of bpf global where we specify the gpl license because it is used you need
to do this to use much of many of the helpers and k functions in the Linux kernel we haven't
like automated it here it has to be done by the user itself at the end is the compile function
and now this is important that the compile function has to be at the end of all bpf chunks
and instead of and what compile does is spit out an object file which contains a bpf object file
there are some variations of compile that you can use you can use compile to IR to only get
the LLVMIR instead of the object file if you need to inspect something while debugging yourself
oh nice instead of compile and compile IR you can also use the bpf object from pile of bpf
which works pretty much like what bcc user space code does so and you can have all of your
things in the same file and that's pretty neat then coming to demos and examples this was presented
at Linux Plumbus Conference 2025 where we presented some demos I don't want to do that again
because of two reasons a the demo was like 10 minutes showing four different things will be
there are some people who have attended LPC 2025 and this talk would have been like
a waste of time for them if I did this again but I do employ you to go see the recording
from this time stamp five minutes 52 seconds we did for examples a TUI based container monitor
syscola anomaly detection for Spotify and then a kernel symbolization using
base sim and vfs read latency example with different like in a python notebook then in a TUI
web dashboard and how you can leverage the python ecosystem to make your vpf tools better using
python vpf also I welcome you to mess with the project and the examples we have on get up most
of these are ported over from bcc we have a try it out section where we list out the steps you need
to take to set up python vpf and mess with it yeah I think I have covered pits in this slide that
the vpf decorator is what marks a vpf chunk and then you can specify for the if it has to be
a struct if it has to be a map or if it's a global or if there's a section that you need to
attach it to so those are the decorators the vpfs nvpf chunk will have at least two of these decorators
one for marking it one for specifying it now a big challenge while working on it and one
of the things that we are still working on that's why the slides are named the internals trying
to make typing work one of the reasons why we are still working on python vpf is to lower the
barrier of entry to learning vpf and writing those programs and abstracting away typing has some
benefits because now people don't have to know everything about what they're writing and
if we handle some vm linear x or verifier errors and typing ourselves behind the scenes it gives
for a much easier experience but so let's define an action on a data container or data to be an
operation which involves the set data or maybe a function call to which this data is an argument
of then we can find such action points in the input file and employ our type deduction on it
because these action points are the only places where we have to care about the data type of the
container in question yeah I try to read about auto and decrytype and how they perform type
reductions in c++ and the outcome is to have a set of rules and just wing it I don't know if we can
ensure that all cases will be covered by us that might need some formal proofing or I don't know
but for now we look at the expected type and the current type of the data at any action point
and check if we can actually convert between them according to our set of rules and then try to do it
one work which we haven't python vpf is there at times you need to convert some r values to
values suppose you have a map with a hash map with your n keys and n values and if you look
at the helpers at the dogs and see the helper signature you'll find out that the keys
have to be for for look up the key has to be an L value but python vpf allows you to do stuff like
map dot look up one or it can be one plus one or it can be k time
on this returns and 864 so and this can be any expression plus one plus I don't know or x plus one
now the outcome of these operations is usually in our value but you have to pass it to a helper
which needs the address of this thing so this introduces a lot of complexity in how we allocate
stack space to local variables another work of working with vpf is that you can only
allocate stack space in the first basic block of a function so if your function has any f statement
and you are creating a variable inside the body of that in f statement we need to see if we
will ever reach that if statement and then probably create space for that as well now another thing
is that map look up this function takes an L value but what kind of L value may be it takes
up not just this helper but any function which we might have already cleared it can be like
a pointer to a pointer or and you are passing just one integer so you need to create
a temporary two temporary stack spaces here which can allow you to first create a pointer and then
a pointer to a pointer and then pass it over to the helper so this because of this we have to create
like scratch spaces where you go through each basic block and see how many such temporary
scratch registers you need and then create them for each function so there is also the depth
problem that suppose we have I think this was what I was hinting to here K time minus request
tsp the here it is very easy because request time stamp is a pointer and you just need to
dereference it because we know that the outcome of this expression has to be and in 64 or
in 32 depending on what the widest integer in this expression is K time is in 64 so this will be
64 bit minus d reference a request tsp until you get a 64 bit integer if you don't then just don't
do it that's one of the things we do and we also have to do it the other way like
allocate as much stack space as you can until you get to a point where you can pass it to
with or right type to the function or helper you want to pass it to then the to
victor stock here was also interesting for string and calories and this is a problem that we
view facing here that there is no clear distinction between strings and calories in python bpf
so if a function needs a pointer to a string you just give it a pointer to the first character
of the character array but now if you want to have a character array at an action point so we'll first
do call to the helper of bpf corner read string and give the output of that string behind the
scenes to whatever action point that that car array is required at so yeah coming back to yeah
so this is somewhat difficult and we constantly test and improve our type deduction and this is
an area where we face most of our hiccups whenever we try something new in python bpf
this is an example which can probably I yeah I think I did go back to this example like here and
then if you see you don't need to worry about types then we have this core read which will read the
exact data for you but here we employ the same thing using just the D difference the struct member
syntax and yeah that's that's it for this right I believe and one more thing about this is
that you might see that this looks cleaner and bless complex and this is not about code
goal thing this is that it's it's fine if the user doesn't know about exact types as I don't think
they should be knowing that to write bpf programs then we have this this about vmnex.py and how it works
the quark here was that we have to generate a debugging for for this and we encountered some types
that we weren't encountering when we write programs without vmnex like function pointers or
structs within structs within structs and that led to more challenges which we overcame
and then where what python bpf does well it tries to shield users from typing and verify
bit pitfalls by basic verifier failure mitigations like auto generating null checks for pointers
or boundary checks for raw data from packets then a concise python x syntax
you can leverage the python ecosystem to create created data analysis and visualization tools
which we have demoed in the LPC recording which I urge you to watch and this can be ideal for
beginners to vbpf and for quick prototyping and there are some things that we still have to work on
we got some nice reviews after presentation at LPC from the attendees one of the things that we
have to work on is that it's not quite feature complete which is true we only have support for some
maps and helpers right now partly due to because we started by handwriting all of the helpers
but now that we can pass vmnex.py and have some what robust type inference system we can
probably create auto generate python signatures from vmnex.py and try to
not insert a writing it by hand just we use the exact helper okay from and pass arguments
and it should work barring for some cases in which we will still have to write exceptions
there was also this review that in vpf raise someone wrote a snake game can someone write a
snake game in python vpf and yeah you cannot write now because we don't support for loops
and this becomes a vacuum old game then that you present this project to someone and someone will
come with a feature request and then you rush to complete that feature request and this might work
but this is not how a project should work we should know like what things we have to implement
for this to work one idea for that was to try to put all the kernel self tests to python vpf
but that seemed like a pretty big undertaking through that it's a very representative of what kind
of vpf programs you can write but it's a huge a lot of code that we will have to work on so
another thing that we focus on would be making a like a simple minimum viable product
and perfecting it and that comes to the second thing establishing a feedback loop with our
early adopters people will come try this find one of the one or two features they don't like
and or maybe python vpf doesn't support and relieve it and we need to have a feedback loop to
our early users to be able to make this better so we need users whose needs and use cases are
very clearly defined and we need to have a wrap up with them to have constant feedbacks a good
feedback loop this can solve our problem we had with kernel self tests as well it's a huge thing
but what if we create it for grad students first for like courses where you have to do a lab
is the time done yeah okay then so so yeah this is something we're working on to make a lab for
my alma mater for the fall sim that would be but that would be a good feedback loop from
them from the students there's also how opinionated python vpf has to be so like for things like
k time that can be time and using print instead of print k because print already existing python
so why should we use the whole vpf print k thing and instead of having a string arguments to sections
having name couples so that your id hints can help you find if a specific trace point exists or not
so things like that can be added a goal is to make python vpf a crime choice and make this
accessible and this is the summary of whatever of this presentation the links I didn't have a QR code
but you can download and visit them and yeah that's all any questions
so you mentioned that we're producing the tools from bcc as example using python vpf do you have
some other tools like property python vpf as well that are not in bcc I mean not tools but we have like
kind of tests so like we implement a feature and then try to write tests for that and those are
not tools that people can use it's more like this we pivoted to using converting bcc tools to python
vpf because we thought that they might already have users maybe just a small comment if you want
to get grad students interested I think it would be good to support for example the vpf
m or the sketch to implement schedulers and start experimenting with that to make it really easy
for students to hike on it or like networking you know some cases like that yeah that's something
we are discussing with the professors that what they exactly need for their labs and since we have
a good timeline for the false them we can implement all of these and then try to see if they're robust
and because students would essentially be fuzzing it for us so yeah that's the plan
all right thank you thank you