I did unmuted well, hi everyone, thanks for being here and thanks for making the room free.
So who am I, my name is Victor, I work at what had in the kernel engineering and one of
the things I do there is I am a commentator of this tool BFF trace which I would like to
introduce to you today. So what is the purpose of this talk? First of all I would like to
introduce BFF trace which is a tracing tool and a tracing language based on a BFF.
And I would like to show you what BFF trace can do for you without its kernel capabilities.
I will slightly go into some like details on how we are doing things but I won't go too deep.
And I would like to also like to show you what BFF trace could do for you in the future.
So what we are going to be working on so that we make BFF trace the ultimate Linux tracing tool.
So let's introduce BFF trace first quick question who here has never written any BFF trace program.
Okay, good to see that many newcomers too too. So as I said BFF trace is a sorry.
It's a tracing tool for Linux based on a BFF which comes with a domain specific language which
we call BFF script. And one of the nice things about BFF trace is that you don't really need to
know internal of a BFF too much to be able to use it because the language is really simple.
How does it work? This is like a very basic workflow how BFF trace works. So at the beginning we have
this BFF script program which can be as simple as this and internally BFF trace once translated
into the BFF byte code which is then loaded into the kernel and then it's attached to some
event for instance in this case we are tracing a kernel function called VFS read so we attach it to the
to the kernel function and whenever that function is called in the kernel your BFF trace program
will execute it can call it from data, it will typically communicate with BFF trace runtime
and then you get some results. That's the very, very basic concept of how it works.
SLBBF script for language the main building work of the program are so called probes
where each probe has two parts. The first one is an attached point which says what event you are
attaching your BFF trace program to for instance in this case we are attaching to a kernel function
but there are other events provided by the kernel or like static trace point you can
attach to user space functions, user space trace points and many many more and then there's the
action work which is like the actual BFF program or BFF first program which you want to execute whenever
that event happens and so what can you write in this action work it's it's a program language like
any other so you get all the familiar stuff like variables, operators like arithmetic,
logic etc you have control force statements or you can do like if conditioning you can
do loops of course with the limitations that BFF provides what is a different from other languages
you have some built in stuff mainly which will mainly help you with the tracing capabilities of
the tool so two things which are interesting in language first is our variables we have two
kinds of variables the first one are called scratch variables which you can think of like a normal
local variables which you know from our program languages that works code which means the
only valid in the current block we have an example here like we are creating a variable x they are
always perfect with dollar and you assign it the current number of the cpu or the current
cpu where your event is running the cpu that's one of the built things which will just give you
the cpu number the more interesting are so-called maps which is another type of variables which are
basically key value pairs which are globally sculpt and you can use them for a native
proposes one is communication between the probe so you can just have one map and you can
store some data and you pick it from another probe to do something useful and you can also use it to
communicate between the the big bioprograms and the use space part of BFF trace
the implementation BFF has maps which is why they are key value pairs and as an example here
we have a very local start the key is the current PAD and we're essentially storing a timestamp
to number of milliseconds also you're essentially storing timestamp for PID in this in this
verb and the second interesting part of the language are built-ins which are as I said special
variables or functions which can help which are built into language which can help you with some
tracing tasks like accessing the kernel data type conversions printing string manipulation and many
many many more there's a month page somewhere where you can find or delete stuff of all of them
okay one thing I want to talk about is why I believe BFF trace is a really great tool
so there are a couple of reasons there are two the most important ones for my point of view
the first one is one lineers the language is designed in a way that it's very
terrors if you will and it allows you to write quite powerful scripts with in a quite short
manner which means that it's ideal for writing one lineers directly in your terminal for
instance debugging production systems because if you BFF it's supposed to be safe so you can
run it on your production machines and you can quite quickly write BFF trace scripts which will
help you find regressions or whatever problem you are trying to debug like for instance an example
here this is a BFF first program which will basically attach to the open Cisco and anytime in open
Cisco happens it will print the calm which is the current name of the current threat and it will
print the file name that is being opened so basically with this one lineer command you can
sniff on all the open cores on your entire system and you can see for each open core which file
is being opened and what's the name of the process which is opening it which is I think quite impressive
the other nice thing about BFF trace and I've mentioned it already is that you don't really
need to know BFF internals in order to use BFF trace because BFF as was mentioned in the first
talk has a lot of powerful features but sometimes they require quite a significant knowledge
of BFF internals to be used and the nice thing about BFF trace is it sort of abstracts
away the implementation details from you by providing the simple scripting language which makes it
a great choice as the entry point to the BFF world because you don't really need to go deep into the
BFF internals in order to use BFF with BFF trace like I have an example here for instance
many of you will know that there's this thing called BFF stack which is quite a limited space where
you can store some like local variables or like some temporary data if it has only 512 bytes
so often if you want to store some like large piece of data somewhere you can't just create a
local variable because it doesn't fit the stack so you would have to have to afford it to a map
or to build the variable now if you want to create a map in in C in the way standard big different
amounts of origin it's not entirely easy I'm not saying it's super difficult but it's already
something that you need to know how to write declaration this is an example of a declaration of
map in BFF trace you don't have to care BFF trace if you create a local variable BFF trace
will automatically check if the size fits the stack and if it doesn't if it's automatically
afforded to in this case to a global variable and we don't need to care about about these
implementation details of course there are strengths but there are so weaknesses which is
what I want to talk about a bit because we are actively working on eliminating them so
there are a couple of weeks but first one is the feature completeness so BFF as I said provides
a lot of powerful features and we don't support all of them we don't have we are missing
super for some helpers we pretty much don't have any cave function there we only support one
that type which is hash hash maps or a couple of those but there's a lot of my type which
you're missing we don't have co-ree and it's a strategy as well so they don't have features
which BFF forever it's which we don't have and another a weakness is that a BFF trace
has traditionally been targeting one line of scripts so these powerful one line there's
which you can like use to on the fly to back your your system however but it turns out that
people want to write large larger BFF trace programs which are they want to maintain for
long period and use them repeatedly etc etc and BFF trace has been liking support for
that for instance like you can't have sale options for your scripts or you can't divide your
probes into into support and plug into functions as you know from a language is because we are
missing support for it so these are one of the things we're actually working on and the last thing
I mentioned here and this is not specific to BFF trace is that there are some events and there are
some environments which are notoriously hard to trace like for instance tracing inline functions
it's kind of tricky because you need to know all the places where the function was inline so
that you can actually at that street which is kind of difficult or for instance running the
trace in a container or in a name space in general can be can be quite difficult so what do we do
to eliminate these? There's a lot of ongoing work on BFF trace these days and I'm just going to
go through all some of the most important things so for instance for tracing inline functions
were already have a solution for that if your trace binary B'd VM Linux or it's based binary
has debugging fault we use LLDB which is a tool from the LLDM ecosystem which can
which helps us to attach your probe to all the inline instances of the function which I think
is very nice it also resolves some fun of problems like it allows you to place the probe
after the function probe which means that you're no longer losing some stack entries that
are specific if you've had ever traced your space binary you could have noticed that sometimes
the stacks are missing one the first entry and this can actually help with that.
Another thing I mentioned was surrounding BFF's inside a container so
I'm so recently we didn't have support for this so if you're running it in a container
with bad namespacing some of the helpers like getting the PI deal with the process ID,
thread ID, order use space stack being worked correctly the reason is that you have to switch
between two different helpers depending on whether BFF trace and the trace binary are running
in the container or not. Now we have a solution it works for most of the cases unless
BFF trace is in a chat namespace and the target is in the root namespace then
we can't support it yet it's kind of a tricky situation to work with.
The rest of the features are here related to the support of those complex programs which
I've mentioned which is one of our goals to be able to support larger BFF trace creeps or
tools based on BFF trace which you can maintain for a long time and one of those are
make variable and net declarations because it's nice that BFF trace has automatic type
inference but sometimes you want to specify the type especially if the inference doesn't work
correctly or you just want it to be more verbable and more like readable if you get back to
it's clear after some time you want to know what the type of variable etc. So these has two parts
declarations of stretch variables we already have support for that it looks something like this
and then the second part are declarations of maps which we don't have yet and it's one of the
features which are being worked on. So for instance it would allow you to say that I want to
enable code hash it will be of a hash type and the key type is an unsigned 32B integer and the
devices are signed 64B integer. Another thing are functions so when your probe gets really long
you really want to split it into multiple functions you also want to reuse code obviously
you all know why functions are useful. So we've recently migrated to LibBFF which enabled us to
use these sub-programs and in future hopefully mere future we would like to support two kinds of functions
one are like normal functions defined directly in LibBFF's trick which will allow you to split
your code into multiple parts but another interesting part is we would like to support
important functions from externally LibBFF programs so you can write a LibBFF program in any
tool you want for instance like in C and then you can just import those functions from
LibBFF tricks and call them from LibBFF tricks which can be useful for instance because
people have written for instance stack workers for Python in BFF in BFF so this way we could
call them from inside BFF programs and for instance create create user space thanks for Python
binaries. Eventually this new functions should help us also creating something like a BFF
standard library which would be very nice. Last thing I mentioned here are common line options
again it's something that is not supported yet but we are working on it so if you have a large
tool BFF trust you want you want to be able to you want to allow the user to somehow modify the
behavior that's what common line arguments are for so we are currently thinking of something like
this you would define a simple helper or so we are simple built in where you would say for instance
I want this group to have one option which would be of integer type and it would be an integral
and then you can just use it for instance in this probe which will periodically print some state
of something like or the data you have collected and you would allow the user to specify the
interval in which you want the number of seconds when you want to print some information and that's
it so just to sum it up BFF trace is quite a powerful tool for tracing Linux which allows you to
use BFF without really knowing BFF internals which is which is quite nice there's a lot of
active development to overcome existing limitations mainly to support creating and maintaining
more complex tools and that's it if you want to know more visit our new website we have a
we have a new logo actually we have a new website so visit it it's it's getting new content pretty
much every day these days there's a bunch of tutorials documentation everything if you have any questions
or ideas for new features for BFF trace if you found any bugs feel free to interact with us via
our kit hub repository we'll be happy to answer your questions or of course you can
talk to me today and that's it thank you and I'm happy to your questions
this one
it was not it was not started by me it was started by Alastair Berton some five years ago
and you would probably have but I think it was at the beginning it was sort of a hobby project
of his oh no this is not like this is not created by Redhead it's just I'm one of the
commentators but Alastair is at meta and the rest of the maintainers are also from meta so
but it's not like bound to throw to a company
so obviously if you if you want to use BFF trace for writing your BFF programs you have to work with
what what what the language allows you to do so as I said for instance if you want some like more
powerful or modern features of BFF you you have to fall back to port port we see or maybe
rest which does support all all kinds of BFF features we only support like a subset we're trying to
like make it as as big as possible what's still worse to like have just a subset oh back there
oh the sorry I forgot to repeat the question so the question is if it's supported on on
which distros this supported as far as I know all of the large distros support BFF trace I know
even some guys are compiling I'm building in for Android so pretty much all the Linux distros
should support BFF trace there's not like technical issue of running it pretty much everywhere
where BFF can run and where you have our login all right I think that's it one minute left so
we're good thank you