All right, we're going to go back to it and we've got to talk that is a bit different
from the rest of the agenda.
This time it's about setting up EBPF using NIX.
I don't think I've seen talks about NIX and the EBPF before, so that sounds interesting.
So that's EFA presenting that for us.
So yeah.
Go ahead.
Thank you.
Hi, everyone.
My name is EFA.
I'm a PhD student in RIA and this talk will mostly be on the DevOps side.
So if you're expecting very technical EBPF stuff, then stay here after I finish the talk,
please.
Okay.
So this is how this talk is meant to me, like how I demonstrate how I use NIX to do EBPF
review at work.
And the overall cycle is I use this specific laptop to the collaboration and the cycles.
And then use Q-Move and our VD switch to do virtualized networking between the virtual
machines.
And then you use a specific deployment toward them.
I supervise the road and to deploy into the HPC system at the cluster where I work
at.
And then I click collect live metrics from the system as well.
So I started a project for using multi-developing a multi-catching system for network
file system over XDP.
That project is running late.
So why I'm presenting DevOps stuff right now.
And a little background about NIX, NIX is a declarative package manager.
It's also a functional programming language.
And the concept of it is to take your source code and then call a built-in function derivation
to transform your source code into closures.
Closures are just packages built on your system and put into NIX store path.
And NIX OS is an operating system as closure.
And if you interested in NIX and NIX of the stuff, you can head to the bedroom after the
talk.
So what it's a test bid, test bid is like a regular production system, but the code running
on there are a femoral, so it's short lived.
And for this specific test bid, it's called 5,000.
It's a part of slices afar and it's a big, pan-repean HPC project that are being most
of their stuff, it's open source.
And the little flow here, this graph, I showed a little bit earlier, you can take this
part and just swap this with there.
So users SSH into an access node, SSH jump host, and then you can interact with the back
bone at work and then the site specific front end.
So for example, since I work out of Grenoble, I log into the front end and then go to the
front end site for a Grenoble.
And if you want to submit a job, I use a tool called OAR sub.
So OAR is a HPC job management system similar to SLIRM and you use the K deploy to deploy
the actual in-n-rd and kernel to the bare metal machines in the cluster.
So the problems we're having is if you're doing a pretty big project and you need to
synchronize the headers of the compiler versions and sometimes LSPs won't even work with
their editor.
So if you have niche use case in some like guarded BPM health perfunctions like override
return, you have to tweak the compilation flags and for it you'll be working properly.
And since during the stage of development, most people use a QM to do the local testing,
but what if you have multiple machines involved like QM3 for and running these commands
can be pretty hard.
You can do it though and you want to see what if you want to see the networking between
them.
So better to use a wrapper tool for that.
What if you're doing booting inside a cluster and you want to clip a larger amount of data
and the automation scripts for those are non-trivial to write.
And also if you're doing academic work, you want to result to be reproducible.
So the artifact evaluation like evaluators can grab your stuff and then rerun.
So hopefully you get the SM badge, artifact reproduce.
So what worked for me is using a mixed with specific tool, but you can run this on non-nixed
system with mixing stalled.
It's called NIXS VMS, it's basically a wrapper around QMU and generates scripts and you
set up the virtual networking stuff and the starting script for QMU VMS.
So with this you can have multiple machines running different kernels with networking
between them and you can also get the benefits for just using NIX like for example binary
cache and everything.
So you'll be built one path exactly once and never again.
The deployment tool is called NIXS Compose.
It's a multi-favor, flavor deployment tool for firmware experiments and it works with
system the unspawn, Docker bare metal and some other stuff I don't use and you can substitute
this stuff with your own development tool and you get user space tooling so you can write
your own package definition in here and then inputs from will grab native build inputs,
building inputs, library stuff, directing to your shell so you don't need to worry about
the versions and stuff so that's really good and then the compilers and the tools like
LLVM strip will be including there like with this and now if you want to use a weird
kernel let's say multi kernel and it's still in the middle of this and the repo is public.
So to do that you can define a NIX file and you specify the name version and what compilers
tool chain you want to build it with in the source you can either use a local path or
file set or just fetch from whatever get for you use and then you can add your own kernel
patches like it's a list of files or a list of patches can be fetched from where we
remote and then the compilation flags and then you invoke that with a boilerplate plate like this
you're done you have the NIX system with the specific kernel ready to go and if you want
to do testing on one machine so you need to do boilerplate like the invocation function
run XOS test and the name for that and then the declarative NIX closure like with no dot
machine name or can be whatever name you want and then in here it's just regular and
XOS module so you can write whatever you want so here in in this example you are seeing
SCX the EVPF based external scheduler and with just this one line you can have that in the
machine and that here is a block of imperative Python statements you can invoke so this will
be run in the now interactive mode of the testing and this can also be invoked manually in the
interactive mode and what if you want more machines you just add a new block and then
declare whatever you want so this becomes really handy for me well I'm trying to deploy
like a benchmark with 15 machines and just add 15 machines in here or you can use a map function
to wrap the config with shared machine just different names and then it will just be there
but watching side the run XOS test result if you do a ripple you can see that it's the config
for the tested self and driver driver interactive the name and nodes. Nosing here contains the
actual evaluated mixed with system call from the next package lib and with what we talked about
drivers and the drivers will automatically include a script for starting human beings and a
release which to set up virtual networking between them and I wrote a very simple
BPM program to do statics mocking so in here I'm using the the bad way of string comparison and
as the last talk specified and then setting the replacing the entirety of the statics call
with a static zero like almost all every every field is set to zero and then you call on the
overwrite return function which requires a special allow error injection plan error injection flag
and BPM allow k probe allow return or something but you need to flex to for to work and then in
here we also have a k return probe so you can count how many times and get a histogram of the
data we collected so for user space programs we're using cloud flares ebpf exporter so it will load
the object file for you and then you have a shared yam profile where you define the bpf map so
it will read into the map and your bpf we're going to rewind the map and then export it through the
previous format and then you can collect and plot the metrics with we're funna and then
the same thing will work locally and in a large scale deployment but in here for the simplicity
I'm only using a tune-up setup so the program I'll show you guys we're living here the source
block of it and then you give a name and a version to it and then in the input and faces is where
you declare the dependencies and compatos needed and faces are how you invoke it and install into
your next store path and then you call the a wrapper function like packages that SDD and V
that make their version will call the built-in function derivation to build the package and then
derivation gets realized into your closure and in this closure you will see that if you have
lib or bin or whatever it can have dependencies and this can of dependencies forms a closure and
in my specific example the object files are put into lib music directly under the store path and then
in this portion you write an exclusive modules to define your kernel packages and in here I'm
overriding the standard kernel into your kernel with a low air injection flag enabled and also
configuring a system b service to start the bbf exporter to load the object files and export
the metrics and HTTP reverse proxy so I can access it and then done here after the system
configuration is evaluated you can either use a local VM or physical machine to test it out
so for local testing we've already talked about this and if you compile it you won't want
and the future compilation will be pretty fast and then you can push the cache to a remote server
or have a CI system build it so if other people want to do the development as you want to do
then you can just fetch the cache on the server so their build will be fast as well and if you
are working at a very big company your company might be interested in a software built material
and makes it pretty easy to track the dependencies and this morning add another dev room the creator
of system d mentioned a f v saw and next let's local test will use this to create an SSH
backdoor and with one single life code so super easy so now I will show you guys a demo for running
two machines on a local machine
so here is the repository this file is the bpf program I wrote in the same
there were three there's a Yamoconfiguration for reading the map sorry yeah is that better
okay so that's the bpf program that's the config and then in the test is where I set up the module
so this is for local testing for the local test in the non interactive mode I want to
check that on the export side I have these flags enabled and I need to wait for this to start
and then on the other note I can cover the end point and see everything works and on the
modified kernel machine I will make sure that I can get zero on the the time stamps
when the bpf project the object files loaded and then I'll disable the client program so when I
do the step again I can see the regular date again
oops so this is the interactive test driver as you can see the system the SSH proxies enabled and
you can direct the SSH onto the human environment with v-sock let's you start all two
start two machines so now the two machines are starting now can see the logs here
so after it's put it out SSH onto the collector node where it's not the collector node
to get a port forwarding so I can access the graphana on my local machine
on the exporter node
oh if we create a file on the tmp overwrite and then we start it you should see that everything is
zero because the the object file is loaded and if we disable the client side
the unload the object file you can see the data's back you normal let me restart it again
and then already infinite loop do start let's head to the Prometheus side
sorry the graphana side and logging with default password I've an admin skip this
dashboard make a new dashboard visualization permit yes let's do a time series first
to see the count the metrics are prefixed with FSD which means fossilm and let's do the count
around the query can see here is the one way you first put it up you can see some going up
and then we stopped it it's empty for a bit and then we're running infinite loop
it's going at the being precise if we go to histogram and change it to pocket
it can actually see the life it like coming in so that's it for the local demo let me stop it real quick
okay and I think it can see it's really easy to start a two-note setup and even if you add
more machines it will just be a couple lines of code and that's it and for production deployment
it's really nice that you can get bit perfect reproducibility on some store paths for example
kernel and the object file but for some programs compiled with okam or haskell you may not
be get bit perfect reproducibility and I think the maintainers are addressing this problem at the
moment so let's see what it goes and since everything is in the closure the entire like the entirety
of the system like the the configs they the kernels in there are everything and it will be
very easy to write a deployment harness and now let's do a production deployment demo
so in here I have essentially until the granobo node of grid 5000 and right before starting this talk
I deploy the exact same environment to a two-note setup and you can see dachu 1991
let's make the text bigger and if I start a new tab and do k-console k-console is a way to
see the what's going on the machine let's copy 19 here so this is the collector node
that's the export node and if we check the kernel config
error injections is indeed enabled if you go back to the regular node
and a collector x-border have function enabled but the function error injections in dd
is able so it's not touched and I'm the exported node sorry I'm the collector node let me give
the IP address of that and in here we should be able to log into graphana as well
and see the live matrix coming from the exported node skip this
and yeah we do indeed see data here and that's the enough for for the production demo
and in conclusion what you guys just see is made with less than 250 lines of next code so it's
very easy to implement to do and you also get portability with it so all the codes you saw
can be ran on your machine if you tweak a little stuff to fit your program
adding another deployment another bpf program to dd deployment will be as easy as adding the name
for the object file that's it and now I'll take questions
thank you
google chrome is not happy it's blocking something here sorry I'm curious too because the things
that you show here it's it's against kernels that have been either package in nixOS or in
github that you can compile have you tried I don't even know if it's possible but like
kernels that for example Ubuntu or fedora users so you can actually test against things running
for those you can yeah so you can grab whatever source you want so for example in this page
this page you're here yeah so if you change the owner repository or whatever source you want
if you tweak the configs a little bit you can get a compiling and then you should work okay
follow up question do you know if there's people building packages for nixOS but kernels for kernels
running for other OS's I'm not aware but if you search for a project called
system manager or something it allows you to want to like manage
configs and everything in other open system and also there is another project for rather name of
and allows you to run nixOS tests in not only nixOS v-s buying v-s arch Linux v-s or whatever
you want so I think okay awesome yeah okay thank you
no more questions so I had a question you mentioned earlier that some package
histories might be like completely reproducible like byte perfect yeah do you know what
the differences are between those that for which that works and those that for which it doesn't work
so bbf be affected by that so for now bit perfect store paths for example I believe oh
camel programs are not perfect reproduced because in somewhere during the compilation stage
the types the packs for the types you're going to strip in the library file header files will change
because that part is not different not differentistic and the O camel comparison so that's
a portion if you're why it's not 100% reproducible but if that one gets up address upstream they should be
all right thank you another question and you mentioned that one of the motivations for
working with nix for the process to help with sharing your environment with some of the people
like forgetting we preserve reproducible that well beats or results have you shared your environment
with colleagues for working with bbf before and how did that go so I've shared it with my supervisor
I'm not sure if he's actually running my code but he's happy with it and I have multiple machines
I have this laptop and another laptop running in the lab and a couple more pretty powerful
machine in the cluster and I just only need to clone the repository and run third and a lot so that
all my developing environment gets on the same machine exactly how I want including the LSPs and
everything it's pretty good it's a you see good at what your your took
your supervisor you see good at what you took I think so so you should definitely on behalf of the bf
organizing team you should definitely try it so that we can know yeah and I can show you guys one more
thing in this repository the reason why this file does not have diagnostic is because I didn't
run with bear so if I clean the build repository and then do a bear with it and then it generates the
the compiled commands and then I go back in again now everything's good plan these running
easy asset and concretely if I want you to try your environment I just need to install
nicks on my machine and then I can clone the repository and then run next developed with your
and perfect shell zsh whatever you want and then you should get the same advantage yeah all right thank you
the next door so yeah coming when you install nicks all is at some additional stuff
so if I use the install right set it up automatically okay thank you any other questions
like once going twice all right thank you thank you