For the next presentation, we've got Sarmat talking about performance and reliability and
analysis.
I hope you feel sorry.
For BPM.
There you go.
Hello everyone.
Thank you so much for taking out the time to be present at this talk.
My name is Sarmat Socket and I'll be talking about the performance and reliability
pitfalls of EBBF.
So just a little bit about me.
I work as a software engineer at Data Dog.
I work at EBBF as part of the EBBF platform team.
I enjoy exploring and working with internal internals and thus the direction I'll be
approaching the stock from.
So the way I'll be covering this topic of performance and reliability will be through
three stories about some issues that we ran into.
And hopefully I'll be able to thread a common team through them and talk about what is shared
between all of these that could be considered like the general team of the problems that we're addressing.
So we're talking about some of the pains with scaling care at probes that we've experienced.
This deadlock in FNG that we came across once and recently a production incident that we were trying to resolve
a few probes were at the center of everything.
Well kind of next to the center.
There was something else in the center.
So starting with the scaling pains of care at probes some time ago,
we got this customer issue where they were complaining that when they install our agent,
suddenly CPU spikes and their network throughput is completely decimated.
The customer was running a pretty old kernel version so kind of 4.14 with Amazon and X2.
And they had a very large machine with 96 cores.
So we set up the similar environment,
ran our agent started looking at Perf to see if we could kind of just easily pinpoint where the problem is coming from.
And that turned out to be pretty helpful.
So before I go into what the actual problem was just to give you a brief overview of how care at probes function.
So as mentioned before the optimised care probes are what you commonly see so that's the schema I'll be talking about today.
So care at probes basically instrument a trampoline instruction in the beginning of a function.
This trampoline goes to a trampoline function which is responsible for overriding the return address on the stack.
So that when the function returns you go to your hook.
And it also saves some invocation specific state in this structure called the care at probe instance structure.
So some of the say that it saves is the original return address so you can eventually return back to it.
And the point thing to note over here is that these care at probe instance structs are invocation specific.
So each invocation of the function gets its own struct.
So these structs are present in the care at probe structure as a link list.
And in order to synchronize access to this link list,
the kernel sort of one for was using a spin lock.
So you can imagine that if you have this massive machine, let's say running a bunch of TCP received messages in parallel.
All of these invocations are serializing on this spin lock and that becomes a massive massive bottleneck.
And when you run ran per stop, this reflected this observation we saw large amount of CPU time being spent in this spin lock.
And that obviously explains the massive drop in throughput.
So unfortunately since this is a problem, as you see later on, all of these were problems with the surrounding infrastructure of the BPS,
the only recommendation that we can give is to upgrade your kernel version.
Unfortunately pretty soon you found that wasn't enough up until the certain point.
Like soon afterwards we received another customer impact.
And this time the customer was running kernel 6.6 again with a very large instance type and again with the same symptom.
Network gets destroyed.
Like the throughput gets destroyed.
So this time around Perth wasn't very helpful in pinpointing anything.
So we started looking into what changes have happened.
In the kernel especially in the scary probe infrastructure part since then.
And we found that in order to adjust that spin lock intention issue, the kernel developers had come up with a solution of putting the scary probe in
the infrastructure in this lock list field list.
So basically there's no lock so there should be no intention.
But if you look at the implementation of this field list, right at the top of the page in the header comment,
you see this very helpful hint as to what the problem could be.
It's not the fastest thing in the world under heavy contention.
And that's precisely this scenario that we're running into.
So what happens over here? You have this lock list field list, but you have this shared memory region that is being written to simultaneously by a large number of CPUs.
So memory itself at this point becomes a bottleneck because of excessive cache coherency traffic.
And obviously this gets worse as you add more and more CPUs.
And since we were running a thing this morning in the massive instance, the effect was particularly dramatic.
So I realized that I did not attach the first step output in the slide, but ultimately when we used birth to record some of them,
my rocket actual performance counters, we in fact saw that we were bottlenecking on memory,
like we were spending most of our time waiting for memory to come.
So again, recommendation is a great kernel, sorry.
And luckily enough after a kernel 6.6, I in fact I think in the later stages of the releases of kernel 6.6, this problem was ultimately addressed.
And now K-Red probe used this object pool mechanism, which uses this birth CPU rig, to store the K-Red probe instance structures.
And since it's birth CPU, very scalable, you don't run into this problem.
So while we were investigating these performance characteristics of K-Red probe, we were also realizing that, hey, we don't want to be running into these problems in our infrastructure.
And back then, unfortunately, we were also behind this optimization as well.
So we decided, let's do a way with K-Red probe, let's move into the modern world, let's use F entry.
We did that, we made the implementation, and then pretty soon we started getting reports of some failures on when the agent container was exiting.
So we set up an environment, launch the container, solve the container, and you see this error.
Try to kill container, but it don't receive an exit event.
Kind of gives you the hint that something stuck somewhere.
So we used this RQ to dump all tasks in uninterruptible sleep, to see who's the one causing a problem.
And lower than behold, you do have some tasks stuck in an uninterruptible sleep.
And in this case, you have to.
So on the left hand side, you see the star score system probe.
So system probe is a component of the agent, which is responsible for managing BPS and stuff.
And on the right hand side, you have this RCU task, K-Red, it's stuck somewhere as well.
So the system probe is trying to detach F entry.
And while it's attaching F entry, it calls this function, synchronize RCU tasks.
And once it calls that, it gets put to sleep.
And on the right hand side, you have this kernel thread, which is doing some kind of waiting.
And in the middle of this waiting, it calls this other synchronize RCU function and gets put to sleep.
At this point, you can already see that there's some kind of dependency over here, like system probe is obviously
depending on this kernel thread.
But what's preventing this kernel thread from making forward progress?
So whatever it was, wasn't showing up in the list of tasks that are stuck in uninterruptible sleep.
So we say, let's dump everything.
All right, pass equity to this RQ and the kernel just shows you all the tasks that are on all CPUs.
And scrolling to that pretty long list, we note, this catches our eye.
S6 SV scan.
So S6 are the set of process management utilities that Docker uses.
And so by the way, this was Docker container.
So process management utilities that Docker uses.
What, a quarter eye, because this is a Docker, and we see the issue happened inside Docker.
And also, this is the parent of the system probe.
Not only is the parent of the system probe, it's also the PID1 in the PID namespace of the container.
So if you look at what it's up to, it's trying to destroy the PID namespace and as part of it,
it's waiting on something.
So now you have these three entities that are in some way seen,
to be related to each other, but we don't really know what's precisely causing the problem.
In order to understand that, we need to take a step back and actually see how things work under the hood for each of the different entities that we're trying to understand.
So with system probes, it's working with FNG.
So let's take a look at what FNG is.
FNG, very similar to carrot probe instruments, a sampling instruction on top of the function.
But unlike carrot probe, where you jump into some shared infrastructure code and you have all of the state management stuff,
this does something very simple.
It just jumps to a trampoline image.
In the beginning of the trampoline image, you run the FNG program.
You call the function, and if you have an FNG program, you call that.
It's very simple.
The trampoline image is generated at low time.
So this literally a stub associated with each trampoline.
So this scales very well as well, because there's no shared state, no coordination problem.
They need to solve, just read only memory, amazing.
Now, on the other side, so that was system probe doing FNG stuff.
On the other side, we had this RCU tasks, K thread.
What is RCU tasks?
So this is a subsystem of the Linux kernel.
I don't know if we can call it a subsystem, but it's a different RCU flavor.
And the facility that it provides to the kernel is to basically allow it to figure out
when no process is holding a reference to some structure.
So the way normal RCU works is also similar conceptually,
but what the problem that's trying to solve is to provide the kernel with a mechanism
to figure out when no CPU is holding a reference to a structure.
So there are some invariants that the code has to observe and it's using normal RCU
which is that only one process on a CPU can hold a reference in contrast.
However, RCU tasks, multiple processes on a CPU can hold a reference to some structure.
And the way the waiting for the release of the structure happens is that before the start of the weight,
all of the tasks that were running need to have performed a voluntary context switch
before the weight ends.
So not preemption, but a voluntary context switch.
So how does this work with FN3?
Let's say you have this trampoline image.
Similar to that PCP received message,
example, bunch of parallel executions where a lot of tasks are executing in this code region.
Let's say something happens, like you need to update that trampoline image,
what the kernel will do is it will generate the new trampoline image,
point the trampoline to that.
So that all subsequent executions start executing in this new stop.
But now the problem that the kernel needs to solve is how does it know that the old
sampling image is safe to release.
So what could happen can actually is let's say some task is executing in this old
sampling image, it gets preempted.
And by the time it comes back online, the kernel may have deleted the
sampling image from underneath it.
And suddenly it's executing junk instructions and you get a crash.
That's not what we want, so this is where RCU tasks come in.
As I said, RCU tasks allow the kernel to figure out when no process is holding a reference to some structure.
And how does it do that?
So basically the algorithm broadly is comprised of three parts.
The first is that it scans the task list for all tasks on the run Q and has
adds them to this hold out list.
The hold out list is all the tasks which have not yet performed a voluntary context.
So the second part which is highlighted because this is involved in the bug is it
also has to synchronize with late stage do exit.
And the reason it needs to do that is in late stage do exit.
A task executing in late stage do exit is no longer on the task list.
So the scan of the task list doesn't make that late stage do exit task,
visible to RCU task subsystem.
So what can happen over here if there's no synchronization is that if you have a probe somewhere in this late stage do exit.
The same it gets be emptied out.
The RCU task system does not see it, so the kernel removes the trampoline image.
It executes the instructions and it does.
So in order to prevent that you need to synchronize with the stage and to do this the kernel brings in another RCU variant.
This is the SRC subsystem.
And basically the idea over here is that the RCU task subsystem weights until all tasks that we're performing and exit.
Bend the weights started, have done the exit by the time the weight ends.
And it uses this tasks RCU exit, SRC you log to do that.
Finally, the kernel loops over the whole list until all tasks have been seen from a voluntary context switch.
And then it's safe to delete what you were trying to delete.
Because at that point you know that nobody has a reference to it.
So one crucial thing is that this was in part of the original RCU tasks implementation as far as I can tell.
Maybe as far as I could see from the kernel history.
This was explicitly added to resolve this problem of coordination with late stage to exit.
So finally, we know what system provides up to.
It's doing some frances stuff.
We know how this RCU task K3rd is working.
And the final entity that we were dealing with was S6S scan.
So the zapper for the PID namespace.
What it does straightforward, it sends a kill signal to all the children.
Wates until they're dead.
But crucially, it does this waiting inside of the tasks RCU exit RCU SRCU that's a mouthful.
We decide critical section.
So now we kind of have a more precise picture forming of what the dependencies are.
So we have system probe, which calls synchronize RCU tasks.
And it's waiting on RCU task K3rd to make formal progress.
RCU task K3rd adds all runnable tasks to the whole list.
It calls synchronize RCU whatever whatever to wait for tasks in late stage to exit to finish.
And we have this S6S scan, which is also in late stage to exit.
This is waiting for all processes in its PID namespace to exit.
And it does this waiting inside of the SRCU recycle critical section.
So now you see you have the circular dependency.
One task is waiting to make forward progress.
One task is waiting for the other one to make forward progress.
And none of them can because this is a cycle.
And that was the bug.
So unfortunately, the fix for this was pretty late into the kernel versions from that time on.
So in kernel 6.9.
The author of this patch removed this SRC mechanism entirely for coordination and came with a different way.
So that this problem no longer exists.
And when we were trying to port over to FNT, at this patch had been back.
But at this point it's been back ported.
So it's safe to use FNT, I don't know how far it's been back ported.
It's generally safe to use FNT.
We also from this thread found another investigation into this issue.
In this case the author was trying to run into this one dealing with PID namespaces.
And when dealing with PID namespaces.
But as far as I know, I haven't seen any documentation of anybody running into this problem with FNT before.
So that was our foreign trying to scale here at probes.
Try FNT failed because of a bug.
And then now we can change the case a little bit and go to this other subsystem of the Linux kernel, which is the cube subsystem.
And we recently were trying to resolve this production incident where we were getting reports of service,
and unavailability on a few nodes in production.
And so eventually somebody figured out that K-Hantosti was reporting that a bunch of tasks were just stuck.
They were waiting for more than two minutes to proceed.
Some of these tasks were specific to the production application.
Some of them were like just random system tasks like bash, system D, run C, etc.
And we got pulled into this is because amongst the stack races that K-Hantosti was reporting,
there were stops, stack races pointing to your registration and your profiling.
So we started investigating, looking to whether the agent could somehow trick in an issue.
So we started the investigation again to each of the hypothesis, the deadlock again.
We saw like an almost circular lock depends on dependency, but not really.
So could not conclude the deadlock.
Is it a memory corruption? Because we found a math that patch that the kernel that we were seeing running was missing.
And it was talking about a memory corruption and an instruct.
So we built a custom kernel and with some modifications to trigger this corruption deterministically to see what happens.
But the symptoms that we saw in that case did not match with the symptoms that we were seeing in production.
So not that.
And then we had this while hypothesis.
Some kind of live lock due to memory pressure that the kernel is trying to allocate memory,
but due to memory pressure failing and it's looping somehow.
But we evaluated all page allocation paths in the UPP register and you're profiling and this did not seem possible.
So we had a lucky break when in order to establish this deadlock hypothesis.
We had built this very simple EBBF based utility that was leveraging these conditions begin and condition end trace points to basically track lock intention in the kernel.
And we stumbled across something that gave us the hint as to what could be going wrong.
So in order to kind of set the stage for that.
I was going to talk about a little bit about how UPP registration works, but I'll skip this because it's not really that important.
I'll go over to this one which is more relevant.
In that when the UPP registration code path is happening, you take a bunch of locks.
So you take a re-trike center for lock on the UPP that you're dealing with.
You take this interesting, dope, M map, same lock.
So this lock is very interesting because it synchronizes.
UPP registration with forks and the problem that is trying to solve is that.
While in underage trace happening and a fork happens simultaneously.
A child M instruction may inherit the debug trap that UPPs put in.
Without an associated UPP.
So this has to synchronize that the underage structure with the creation of the child M instruction.
And then finally what basically what this was showing is that using the iNode of the file.
The kernel discovers all of the MM strokes.
So all of the address spaces that are mapping the file that we're trying to probe.
It takes a right lock on it, does some work and unlocks it.
So the reason it takes an right lock on the MM stroke is to synchronize with UPP firing.
So the problem it's trying to solve over here is again with underage structure.
That you don't want the UPP firing path to discover a UPP that you have.
Underage district since then and it runs and it doesn't find.
So basically you probe down the line and returns an error.
So you need to have synchronization going on over there.
Essentially we try to graph what was going on with the UPP registration path.
And you see that in this system probe one execution is going down which you see a bunch of logs get taken.
And every single case we were seeing it get just get blocked on trying to acquire a right lock on the MM block.
So going over UPP firing the important thing over here is that it works with the address space of the process in which the UPP is firing.
And it takes a lead lock on it.
And with the case of UPP firing we were seeing a large number of tasks blocked on find active UPP probe.
And this is the function where you take the lead lock on the MM stroke.
And again it was blocked trying to acquire this lead lock.
And lots of application threads were piling out waiting for this lock to happen.
So what is this MM block?
Well it's this lead writes MM4 allows concurrent readers but exclusive rights.
So the EBU for utility that we have been talking about we modified it to try to get some questions answered about what's going on.
The important thing over here is what the depth of the weight queue is when these issues happen.
So the weight queue is like task waiting to acquire the semifur.
What's the contention duration? Like how long is the contention going on for?
And whether forward progress is always made.
So this is added to rule out dead locks. And we figured out that even though things were waiting for like two minutes plus they were making forward progress.
So eventually what we saw was that the weight queue became very very deep.
And the pattern was that there were lots of readers with a few writers interest rates everywhere.
And these writers were coming from MMAP and advice operations in the task and UPP registers.
So the way read writes MM4's work is that every time a writer gives up the lock, it obviously wakes up all of the readers in the weight queue.
But it only wakes up 250 readers at the same time to avoid like attending her situation.
So this means that if the weight queue is broken up then batches of reads become serialized.
And not only that, in order to prevent writer starvation, if there's ever a writer in the beginning of the weight queue, no concurrent readers are allowed.
So this big creates a situation where readers start to pile up.
And if they're writes interest rates here and there, you have these readers serialized in batches.
And then eventually the writer makes it to the head of the weight queue and suddenly more readers.
And if there's a very, if you have a UPP in the heart path, certainly a lot of readers are collecting.
So this diagram is supposed to explain why we were seeing weird task like batch and run C getting stuck.
So as I mentioned that in the UPP register part, let's imagine you're waiting for two minutes to take this MMAP lock because it's a very cute so deep.
So I thought that you've taken this dope MMAP Sam with synchronizes with fork.
So suddenly, all fork's in the systems are waiting on you to get this right MM lock.
So suddenly if you do SSH, a batch comes up, it can fork nothing happens.
System D can do anything, run C can do anything, your system is pretty much brick.
And in the case of UPP firing, it's blocked on getting a read lock, but it's still blocked.
In this case, application performance suffers because application threads just start to pile up in the kernel and they can make for progress.
So by things happen, the application gets owned, there's unavailability, etc.
So eventually there's a fix in the kernel and right now in the very latest ones, I think they have this speculative acquisition approach for getting the MMAP lock.
So they don't get it unnecessarily, they only get it if they see that something has changed underneath them and so they repeat the operation with the lock in place.
So this really improves the scalability of the operation.
So basically the common thread that I was trying to get to was that concurrency is very hard.
And sometimes even the BPS meant to be very usable and simple, the complexity of the kernel leaks into it, especially when you're trying to do things at scale.
And sometimes you have to go beyond the BPS program to figure out where some problem may be coming from.
So thank you, thank you for taking the time to listen.
If you have time for questions, I'll be back.
Yeah, we do have time for a couple of questions.
Thank you.
Just curious about the preempt.
If you could talk a little bit more on that, I'm not sure if I quite got the relevance of the preempt on the slide you were discussing.
So in this last part.
Over here.
So basically what you're trying to solve is that in certain configurations of the kernel, especially when you have when the kernel can be preempted, what may happen is that kernel code while at ask is executing kernel code, it can get scheduled out.
So imagine that a process gets scheduled out, wireless executing the F entry mechanism.
It technically still has a reference to the trampoline image, but the normal RCU cannot track that.
Well, in some cases, in some cases, the can.
So what you're trying to get a surety of is that the task that was executing F entry has performed a voluntary context switch, because at that point you know it has completed the F entry trampoline image.
And given up execution, because it's done with it.
Thank you.
One last question.
I guess the other conclusion is to always use the latest kernel.
One other question is also the F exit related.
Did it also expose the back or was it really only specific to F entry?
It's the whole F entry effect.
Okay.
Because if you look at the doctors over here, it's the trampoline.
Yeah, infrastructure that is.
I get, yeah.
I agree.
Thank you.