I'm happy for that, very happy to be there.
This is my second facem, and I'm very, very excited to be in the EVP approved for the first
sign.
So I'm the Nia Shailu, I'm a software engineer at Isovatant, and I'm a community
oriented, and I'm going to present with.
Hi, I'm Chris Tarazi, a senior staff engineer at Isovatant at Cisco, and a contributor
on the Scaling Project for the past six years.
We're going to look into some common, yes, OK.
We're going to close it to the mic.
OK.
Better?
OK.
We're going to look into some common and interesting hookpoint gotchas.
So as a disclaimer, this won't be an exhaustive list.
So there's definitely other existing gotchas that we're not going to cover in this talk.
So he's a click-up review of what desktop will cover.
We're going to introduce what are tracing hookpoints.
I feel like you're quite familiar with them, but we're going to say the basics.
And we're going to cover some generic gotchas.
And we're thinking to more of a nice and interesting ones.
We'll see, you know, about it or not.
All right, so let's do a quick overview of the most popular tracing hookpoints
that we can use with EVPF.
And these are the ones that we'll also cover in the talk.
So starting with K probes and U probes, these are roughly the same except K probes
for kernel functions and U probes for user space functions.
What's unique about them is they can be attached to any offset from the function entry or exit.
And these probes don't have stability guarantees because they're dynamic.
For example, between kernel versions.
And so like if the function you attach to changes, you'll be required to rewrite the hook program.
They've existed for a long time in the kernel, so they're the most ubiquitous and easiest to use.
Yeah, so and F probes are similar to K probes, but they differ in a few ways.
First, they use the F trace infrastructure, which makes them more efficient.
Especially if your K probes are not running as optimized, we'll touch on that later.
And F probes can only be attached on the function entry or exit and don't support offsets like K probes.
And lastly, we have trace points, which are statically defined points in the kernel.
And these are designed for stability, of course, there's caveats here and there, but for the most part.
So they work by injecting five bytes of no ops at the trace point entrance,
which are then later re-written into jumps to your trace point program.
So here is some performance benchmarks that were done in various scenarios.
And here it's showcasing the different performance difference between these hook point types.
The most relevant number here is the overhead percentage that gives us like the high level picture of the performance cost.
Keep in mind, the amount of overhead is dependent on whether the function probe is in the hot path or not.
So it's not a direct translation to the overall systems load-up to overall systems load-up.
Unsurprisingly, the trace points here have the highest performance followed by F probes and then K probes last.
The K probes under the default intrap based implementation relies on breakpoints, which clearly is the least perform implementation.
So the first gotcha we're going to talk about now that we have this overview is about terminal versions.
So on most of the infrastructure out there in production, there are different terminal versions running.
I heard that meta has like 20 around 20 kernel versions running in parallel.
That's interesting.
How do you write EBPF programs running on different kernel versions?
K prob and IF prob have no stability guarantee.
Please just mention it.
And terminal internal functions can change with any new release, basically.
Function can be a new name, removed in line.
And then you're probably going to fail to attach because the symbol is not found.
So if you're not familiar, it's just like probably it's going to actually symbol corresponding to something in your program and then convert that into an offset.
And that's actually to that offset.
The function arguments can also change.
I mean, you're a factory encode every day, you know.
So your family always try to change it.
Like you're adding a new feel or maybe you're just moving around stuff because you want to reorganize your code.
And you can read the wrong arguments if you're running that program in another variable.
And you will just have garbage drive at the end.
So how do you handle that with F prob and K prob?
One of the alternatives is to actually use trace points.
There are actually more reliable because there is a guarantee of stability.
The arguments were documented and maintained.
It's basically done by Linux kernel developers.
So you can trust that.
And you can see all the available events there.
I have a question for you.
You're going to actually answer a participant during this talk.
It's interacting if you didn't know.
So what if there is no trace point for the function that you want to cover?
What is your option?
Who say A?
Okay, just have a look.
Who say A?
Hardcore.
The function from name and hope it doesn't change.
No one.
Right.
Separate BPS program for each kernel version.
No one.
Use core with BTTF for automatic adaptation.
Yay.
Some hands.
I've done an EDPF and use kernel module instead.
That is great people.
Yeah.
Okay.
Grace.
Yeah.
That was actually the answer.
See.
I mean, if you want to go for D, that's your choice.
So let's see what is core.
Core is like compile once run everywhere with BTF.
Core means that BTF programs can add up to kernel at low time.
And BTF means BPS type format.
And there are type information on your program that are actually stored somewhere in the kernel.
Not embedded in the kernel, sorry.
And you will have a location on the fly.
And the BFF example will add just offset automatically.
So you're attaching to an offset.
And those offset are stored for each part of your program.
And then you can attach wherever you want on the fly.
And it's going to work.
Amazing.
Let's move to the second gacha.
The architecture.
So K-Probs.
If you don't know, have infernal interface.
The context path to a K-Prob is like a structure.
Obscure one.
PDRX, which is a strike for presenting the registers.
Where it's going to store the function parameters.
So if you want to run a program.
If you're writing a program.
And that you want to run it on different architecture.
It's not going to work as is.
Because all the functions that are calling the registers are different.
Basically, the registers are storing all your information on different on the different architecture.
886, Aaron 64.
And the way to access them is different.
So the function to access them are different.
So if you write a program for one architecture, one another,
it's not going to be the same.
So what are your alternators?
You can, again, use other hook points.
Maybe F-Prob trace points.
No road trace points.
I actually don't want to do that.
But yeah.
And there is also the BPS provider that helps to access
the registers.
So basically, it's agnostic of the type of registers.
It's going to rot at for you.
And you're going to specify when you're compiling the type of architectures that you want.
So at the end of what you have is one program for several architectures.
And you're going to choose when you're compiling the type of architecture that you are.
That you want to.
So several binaries.
But one code to maintain at the end.
Okay, let's move on to the third one, dynamic links.
So we have a simple program here.
We have a library, which is doing an addition.
I think that we all familiar with C program here.
Is it okay?
Yeah.
Then we have a main.
I just put here the body's important.
So the rest of the main is loading, attaching the program and run.
The add function, calling the add function.
Okay.
What I'm going to do is to use this BPS trace to actually probe to the add function.
So I'm going to compile the library.
I'm going to compile the main.
And then add to the library and to the function add.
And when I take the symbol table, I can see that I have the function add with enough that.
And then I'm going to run this program.
I have a basic addition that is actually done.
And I can see that my probe is actually like heating.
We can see the log here.
Can you see my cursor?
Yeah.
You can see that.
Okay.
We're happy.
We have a trace that add function has been called.
So what I'm going to do now is that I'm going to update the library.
Because I don't know.
I want to update the library.
You do that when you code and add some code.
So I'm going to insert new function, subtract and multiply here.
I'm going to add them above my add function.
Then I'm going to call again my program.
I state I attached to my BFF trace.
Because I'm running debugging staff or production staff.
And I have again a question for you.
So your view probably is tracing add.
It totally gets a library and adds new function.
You're running a program again.
What do you see?
What is the output?
Is it a?
Everything worth fine.
You're going to see the add again.
Is it b?
You're going to have the subtract?
Is it c?
Nothing.
I'm going to stop working or not.
I'll be about.
C?
Yay.
Yeah, I think nothing is going to happen.
So why?
Yeah.
That's right.
So you're going to recompile the library.
And the offset are going to change.
So the add function is going to have a new offset.
But your BFF trace attached to the old symbol and the old offset.
So you have to actually reattach in the case of updating libraries
to make sure that your probe is going to actually
going to hit.
Such a gacha, right?
So we've seen that library compile updated.
Then in production you have library updates.
But it's very rare.
Yeah.
It can happen in case of security patches.
But that's not often.
And your monitoring system may break silencing
without warning.
You have different solutions for that.
You can monitor for file using, like, I notify weight.
You can use the provider program to reattach automatically.
If you have any other alternative, please share.
Thanks.
Let's move on to the fourth gacha about inlining.
So compiler have.
Advanced form of inlining depending on the level of optimization you choose.
And we're going to go a little bit more into detail.
But we're going to see only two gacha about inlining.
And as Daniel mentioned a few days ago, there is a zoo of compiling
like compilation gacha.
There are many of them.
But I found two very cool ones that I want to share with you.
So what is inlining first?
Inlining is an optimization where a function call is replaced by the actual code
of the function itself at the point of call.
We're going to go more into detail.
That's the basic definition.
Okay, let's have a look to code again.
Add function.
I'm not very original.
And main.
This time, if you pass an argument that is under five,
we're just going to return.
So early return.
Otherwise, we're going to call the add.
Then I'm going to compile without any optimization.
So 0, 0, 0.
And I'm going to see when I design symbol the program,
which is an M2 tool that I can see the offset my symbol add.
And the T just means that it's global external symbol.
Then I'm going to compile with a level of optimization of O2.
And I'm happy.
I still have my symbol.
So I can normally attach in any case to my add function and see my probes, right?
So let's see.
Okay.
So it's going to be tricky because I don't have my ID here.
So bear with me please.
Mirror ring here.
Okay.
So select even lying, right?
I have the non-optimized.
I'm going to trace the non-optimized one.
Okay.
Attach in here.
And then I'm going to run my program mute.
Okay.
Let's not mess up that.
Okay.
Then I'm going to run it here.
Okay.
You can see my history here.
Let's clean it.
This price.
Okay.
You can see the usage.
I have 0.
It's under five.
Then I'm going to run it with 10.
So you still have nothing is happening in the problem, right?
Because the function is not cold.
Then you can see.
It's actually heating.
Great.
We're happy.
Okay.
Now I'm going to do it with the optimized one.
So let me trace the optimized one first.
Okay.
Can you all see your book?
Yeah.
Okay.
Then I'm going to run again with zero.
And I have no problem.
That's normal.
Right?
Then I'm going to do it with 10.
Oh.
No problem again.
Oops.
What's happening?
Let's see.
Okay.
So if I go back to the slide.
So here what you're saying.
Okay.
Don't be afraid.
It's okay.
It's only this assembly binary.
We have a bunch of instructions here.
What you want to look at is the outside of the offset of the main.
And the offset of the add.
Okay.
I have on one side.
Not optimization.
It's branching.
You can see the BL here.
Just here.
So BL means that the main instruction are going to jump and run to the add function.
And it's going to attach to the other offset.
The actual programs.
So the offset is corresponding to the actions instruction that are going to be executed.
What's happening in case of optimization is that I still have my main idea of set.
Right?
Then instead of having this branching, I have all the instructions that was in the function that
are in line in the main program in the caller.
And you can see here that I have the add.
Add a new offset inserted.
All the instructions are started.
So when I was attaching to the symbol.
It's not the proper offset.
The actual one that is going to be executed by the CPU.
So how do you fix that?
Do you have any idea?
Okay.
What?
No, it might not go.
Okay.
That's an option.
I actually have another one for you.
I'm going to show you how you can attach to offset directly.
That's very little level.
I don't even know what else we could do that.
Okay.
That's funny.
So I'm going to use Perf for a funny reason when I was doing like a 2AM yesterday.
My final demos.
That you can not use 2bpftrace version like the 20 and the 24 because they're not
supporting the same like NM comments.
So my two examples were not working.
So maybe through regression if you want to file a bad.
I can give you information.
Anyway, I'm going to use Perf for that.
So Perf is going to trace it also.
Let's do it.
Bear with me again.
Mirroring.
Okay.
So I'm going to attach to the offset here.
I didn't tell you about this offset is just like the instruction of the function.
Sorry.
The offset of the instruction at minus the main one.
So this is a variety of offset on the main.
But I computed here and I'm going to attach to this offset.
So main plus the relay relating offset.
Okay.
Let me.
Okay.
Let me do that.
Okay.
So I'm going to do it on target zero.
So we're going to see what we see.
We got it.
Yeah.
I mean, yeah.
It's done.
Then I just.
We got all the even start to have been captured.
And there's nothing.
Right?
Because I'm calling the not optimized function.
Then I'm going to call the optimized program here.
And.
Yeah.
Let's see what's happening.
Yay.
We have actually a problem.
So you can see that by attaching to the main with the offset.
We are actually capturing at the proper instruction.
That is doing the ad.
So yeah, let me know if you do that sometime.
Well, they're begging.
That would be funny.
Okay.
Confision.
So we've seen that.
About selecting the lining that the compiler can actually have the symbol existing in the binary.
It's visible using an m.
But then it's in line.
Into the color.
So you probably have such a symbol on the symbol.
But it's never going to fire.
Because it's like the executed instruction or all in line in the code.
How you can detect that.
You can use object dump.
You can use LLMium dwarf dump.
There are not very easily friendly.
I might admit.
But when you get used to it, that's okay.
And you can check if the function is in line by seeing if it's branching or law.
Or if you have directly the instruction.
So be aware of one thing.
It's that there are some caveats.
Here we have only one call.
But you can have multiple pops for all call site.
So that can be a problem.
Okay.
Let's move on to the next one.
K-Prob and you probably lining.
So we have a program again.
And how many symbols does this generate when compiling is the following come in O2?
One, two, ten.
But the whole symbols.
You don't know.
It's actually two.
So we have a part.
What's happening with GCC?
We have a past path and a slow path.
Basically we have before and after.
Before we have one function.
After we have two parts.
We have the fast path.
Which is just the early return of the code.
And then we jump to the processing part.
So instead of having one symbol holding all the code.
We have two symbols.
Why does GCC dot dot do that?
It's just like it's better for range prediction.
It's smaller like instruction in the cache.
It's registered register pressure.
I mean it's doing optimization.
So let's have a look quickly.
Okay.
Let's move here.
And here.
Okay.
Let's do that quickly.
So I'm going to trace on no partial log.
That means that it's not optimized.
I can see if I'm running that.
That I have called here.
So it's probing when I'm calling with the values that are actually not early return.
Then I'm going to.
Okay.
Do.
BFF trace.
On the one that is optimized.
And I'm going to run it.
Here.
And nothing is probing.
Okay.
I'm actually doing it on the symbol allocate resource.
So let's see if I actually do it on the part.
So you can see that I have the stiffs here.
So it means that that's the code that is actually doing the location.
And I'm going to run it again.
Yay.
We have our props.
So.
That's just a summary of it for reference.
But basically.
You just have silence when you're like when you're probing to the allocate resource symbol.
Because it's just jumping to another part symbol.
And the second one is the prod that we just so when you allocate to the part.
So what are the alternatives again?
Generally.
You can.
Check for trace points.
I know that's the running guy of the talk.
But there is no one one translation.
Sometimes depending on the function in your use case.
So.
Yeah.
It depends.
But if you're using your prob.
You can actually as you mentioned before.
I use.
The non partially lining.
Otherwise for k prob.
You're not going to recommend the kind of right.
So you can check for syphix symbols.
There are different one of difference one.
And you can vary what we've dwarfed again.
Or see source mapping with object dump.
So that was the gotcha for in lining.
All right.
Now we're going to take a look at the next gotcha.
Missed executions.
So here we have this output from bpf tool.
Showing an f probe k probe in a trace point.
And there's something interesting there at the end.
So any ideas what recursion miss counter represents here.
A.
B.
C.
D.
All right.
Well, I got to move fast so running out of time.
It's B.
How many times recursion prevented the bpf program from running again while already executing.
So as we saw in the bpf tool output, you're tracing bpf programs can miss execution.
So in the case you're depending on these probes for like security or observability,
this can be like a challenge.
So for example, let's say you're on the security team.
You're interested in monitoring file operations in your infrastructure.
You're using bpf to monitor like file open events, for example.
And you might notice like you're expecting a thousand events from your metrics.
Or and your audit logs say like only 900 events or something like that.
You're missing like 100.
Or you're debugging a kernel function or kernel internals and you're tracing a hot function.
The bug that you're looking for may have occurred like when the probe missed an execution.
So you miss your like stack trace dump or whatever.
So yeah, might be surprising to learn that your tracing hook points are not 100% reliable.
So let's dig into the different ways that k probes, f probes, trace points can miss.
How they can and why and we'll discuss some more grounds.
So probes generally have two different places in the kernel where the where they can miss executing.
One is when the probe renters or recurses itself before completing the first.
The second way is when there's any concurrent execution of any other bpf program.
So crucially these misses only happen if execution occurs on the same CPU.
It's a different CPU.
It's a new context.
So it won't miss that's not a problem.
So let's look at what a recursion looks like.
So let's look at example one bear with the diagram here.
So let's say you have a k probe on open at two.
That calls print k which takes the print k lock.
At the same time you have a trace point on contention begin.
Let's say contention on the lock occurs like on the print k lock.
The trace point program fires which also might call print k.
That leads to a recursive execution and the kernel skips the execution of the second program.
And I don't oops.
I don't like scroll down.
I can scroll down.
Okay, that's fine.
That's fine.
Yeah.
Okay.
So in the second example we have something a little simpler.
We have a k probe on open at two.
And in the middle of executing the introp can fire.
That handler's code for the introp then calls open at two again.
And then boom you have another recursion.
And yeah.
So let's dig into k probe.
So k probes can miss at two different layers.
One is at the handler or the attach layer.
This is the layer where your k probe is actually like set up and called.
So and then at this layer there is like k probe specific recursion logic that's performed.
And it primarily checks if there are other k probes running on the same CPU.
And then the other layer is the actual bpf program execution layer.
This is like the actual jitted instructions.
And at this layer there is like the bpf program's mutual exclusion.
Which in other words means only one bpf program can be executing on the same CPU at a time.
So let's look at the handler and attach layer.
We have three different handlers depending on your kernel config and version.
First one is the standard like break point handler.
Then we have the f trace handler it's optimized.
Then we have another handler called opt which stands for optimized as well.
Yeah so anyway nowadays running optimized k probes can be seen here as like most common.
But in three hand there still exists for legacy purposes.
Okay so all three handlers have this same check here where they're checking if there are other k probes running on the same CPU.
And that's done by looking at the current k probe variable which points to the currently running k probe.
And that's very similar to like the current task variable that we all know from for processes in the kernel.
And so yeah if this condition is true we missed the execution of the k probe.
So why does this happen even anyway?
So from my understanding the kernel drops the execution to corrupting the state of the k probe that's currently running due to the k probe being like an interrupt based thing,
especially in the like the entry handler type.
And so if in the future that handlers deprecated I believe it's possible that the k probe running restriction could be lifted.
I would love to be corrected here but that could reduce the amount of missed executions for k probe specifically.
All right so let's look at the second layer here for k probes.
We've passed the attached layer meaning there was no k probe detected running on the same CPU.
Now we reach the program execution layer.
So at this layer we have a per CPU variable that's checked BPF prog active.
And this tells you if there's another BPF program already running on the same CPU.
So this check exists to protect the per CPU kernel state that BPF programs might access or modify.
So that's why the kernel drops the execution here.
And without that drop the kernel state could get corrupted from nested executions.
All right let's go to F probes now.
So F probes don't use the k probe running helper they use like its own dry lock function.
And this function actually supports one layer, one level of nested execution.
As long as that execution is from the same context.
So for example we have F probe with IRQ and NMI to F probes i mean and that's not allowed.
But if you have two F probes the same F probe on the same CPU but from the same context that's that nesting is allowed at the handler layer for F probes.
All right so F probes BPF program execution layer.
So F probes are BPF trampoline based not in truck based meaning the at the execution layer it checks if the same program is already executing on the same CPU.
Wrap up okay.
So.
Yeah so because F probes are based on trampolines each each F probe program has its own trampoline where it stores its state.
So if there's a nested execution the F probe the nested F probes could override the BPF trampoline memory.
Okay so here's summary of what we just discussed i'll skip this for time purposes.
So what are your choices so if you can try to use BPF LSM hooks.
For example if you have an open at two K probe F probe on open at two maybe try the file open BPF LSM hook.
Because LSM hooks are guaranteed to run under the NMI context with concurrent executions and that is unlike tracing BPF programs.
And if you can't use F probes sorry if you can't use BPF LSM hooks try trace points because they have the highest performance.
So if you have if you have spend the least amount of time in your hook then there's less chance of interrupts that can occur and therefore.
Recursion.
There's a summary move on.
All right if you take aways.
I think.
Got a wrap up.
Okay.
All right thanks everyone.
Thank you.