All right, well, welcome everybody, my name is Samuel, I'm a production engineer at Meta,
and today I'll present a BFF trace, um, profiler that I wrote. And by the way, good ordering
on the talks, because now I was not going to explain anything about BFF trace, so thank you,
so yeah, just a little agenda for today. First, we'll frame the problem, we were facing,
we'll walk through some implementation details, and look at results and
share some learnings. So the team I work on at Meta, it's called Scuba, and it's
basically a big log aggregation system. We have a real-time data ingestion pipeline,
and we have a query-side component. I'm sure, probably many of you guys have used similar systems.
You can think of elastic search, splong, I think it's Amazon CloudWatch. And visually,
it looks something like this. So if you trace a request through the system, it goes through the proxy,
and then it traverses the map, reduce architecture all the way to the leaves.
And this is the part of the system will be interested in today. The leaf you can think of them as
basically distributed database servers, and as with any good database servers, we have memory issues.
So we have too many ooms, and the goal of this project is so we already have protection in place.
In the Quarant Giant, we have a whole admission control layer. But really, even though we have protection,
we're still ooming. And so the goal was to can we improve the oomubs of ability.
And so we want to identify expensive use cases. And a use case, you can uniquely identify them with
this triplet that I put here, basically table source SQL. So, and then eventually,
if you're able to identify these expensive use cases as a first responder, when we're facing
ways of ooms, you can go and you can block them potentially.
So, let's try to see if we can do a proof of concept. So, the first version,
it's not going to be the final result, but naively, we're going to, our goal is,
we'll try to count these triplets, which I'll also refer to as aggregations.
We want a data set that is highly correlated with memory pressure ooms.
So we'll collect data only when we're facing memory pressure. We'll log only after oomm.
And then the final criteria is we want to be memory life weight. And that's because the system is already
running out of memory. So, our tracing should be not adding oil on the fire.
So, when I was discussing this project with teammates, they were like, well, why do you want to use
BPF trace for this? Why not just do it in user space? So, things out in our environment,
we have this thing called oomd. And this thing is responsible for killing processes that use
too much memory. So, it will act before the kernel oomkiller. And what it does, it sounds a sick kill.
Sick kill, you cannot intercept it, cannot ignore it in user space. And so, this is why it's useful to use BPF trace.
And what we do here, we have the BPF trace snippet. We intercept the trace point for the signals.
And then we just check our condition. Was it sent to our scuba process? Was it signal line? And did it come from oomd?
Yeah, but no oomd. It's open source. It's deployed on pet every machine as a system to service.
And any other reason for its existence was to have better control over what gets killed.
And mostly to avoid killing these golden Facebook processes that exist on machine that would render the
hosts basically unusable. So, we move on with our goals. Let's try to collect these aggregations.
So, the first thing we do is we go in our binary. And here we use the macro from folly.
folly is a C++ open source library. And we use this macro. We add two probes.
So, first the entry probe. And you see what we do is we add the query ID. And then our triplet.
Table source is equal. And then we also add the return probe, which is going to run at the end of the function.
And we add the query ID there. So, it's important to have a common piece of information between entry probe return probe so you can connect the dots.
And that will be important later.
So, now we've instrumented our binary. So, we can go in BPF trace and actually attach to this new usdt trace point.
So, here this is the entry probe with query SQL. And we just we have this map. And we just count our triplets.
And we also save the PID. Because if you're in this probe context, the PID will refer to our server.
And we needed that to do the filtering. Okay, another thing we add. And this is just for completion, but sometimes only is not perfect.
So, sometimes the kernel will actually kill our process. But we can detect this by tracing another trace point from the kernel.
And we can just filter on the PID. So, this is another way to detect if we've been killed.
And finally, yeah, our last criteria. Remember, we didn't want to use too much memory. So, there's few ways to do this. But the way I chose was to do backup duration over time.
So, what we do is we collect aggregations. And we always write in the current bucket. And then when every now and then we rotate the buckets.
And so, if it's elapsed, everything that was in current would put in previous. And we just trash everything from previous.
And so, you can see we rotate things, which means in the end the memory we use.
Our data retention period is twice the bucket duration. And in BPS trace, it looks like this.
So, you can do things with the interval here every 10 seconds. We look how long has it been elapsed since the bucket started.
Are we over the bucket duration? Is it time to rotate? If so, well, we have two four loops.
We delete everything in previous. And we put everything in current in previous, just like we showed it in the graph.
And then we reset the timer. So, that's not a perfect implementation, mostly because traffic pattern over time are very variable.
So, maybe during one bucket duration we got a ton of queries, another one we have almost nothing. So, it's not the best way to do things.
We'll be much better with the LAU Ashmap. And as Victor mentioned, there's active discussion and a thread for supporting this.
And the other thing, you cannot assign a Hashmap to another Hashmap. So, I wanted to just do, oh, pre-ev equals current, but it turns out you cannot do this.
And then the other issue was, you can clear a Hashmap, but it's only async supported. So, that created some data and consistencies.
But, okay. So, when we have achieved everything we wanted, let's ship this in production. So, the way this works, at meta we have the service, at the center, scroll, scroll,
blight, and scroll blight takes two things. It takes the system state and triggering rules. So, for us, the triggering rule is on memory pressure.
So, what we do is, we test scroll blight. Okay. If you see memory pressure, start our BPS tray sidecar, and then it will do a sting attached to all the probes.
And then if we oom, then we dump the data, and the data is sparse by a strobe blight, which sends it back to scuba. So, yes, we are monitoring scuba with scuba.
And, okay, there are results. So, yeah. Unfortunately, I do have to censor the prob data, despite what you've heard about our censorship policy.
But, but I still put the counts there. So, you see this, we have our triplets. We have the table source SQL, and we are able to order by the counts.
And it works. It's a proof of concept. It's oom, Taylor data, but it turns out it's not really useful. And, and the reason for that is a lot of this thing.
So, we have, on our platform, these type of, of use cases, which are basically help checks. And so, they're all going to be at the top of the list, which makes the data really noisy, and it's not really useful.
So, okay. So, we let this run for a month or two, we get some discussion, collect feedback, and we come up with new ideas.
So, let's try to one up our implementation. And what we're going to do is, well, first we'll do continuous lagging.
So, in the previous version, we had this bucket thing. We do current preview, and we will trash the data.
But, it's perfectly good data. Let's, we can still log it. And, and what we do though, because we want to retain the ability to, to say, was it oom on the data, like, after ooms.
So, we just had a filter. We do this. Then we'll track one query ID per aggregation.
That's mostly for us query ID. It's a gateway to many other data sets, get more information.
And then, instead of doing counts, we'll try to collect memory and CPU cost.
So, the way this looks is we go back in VPF trace, and we refactor the entry probe.
And now, so, we need to trace all this new information. So, that means we need to add a bunch of maps.
And we still want to be memory lightweight. So, we add this thing, the concept of an aggregation ID.
And really, it's just because we only want to store the triplet once. So, we store it once.
We create an ID, it's just a self increment and ID. And then, that's what we used to refer to the aggregation.
Number two here, we have a query ID's map. It's just a map from aggregation ID to query.
And finally, we add a temporary map, and this is just so we can connect the dots between entry probe and written probe.
So, this is important in the next slide. And it's mostly so we can track memory and CPU for the aggregation.
So, we use our temporary map, we retrieve the aggregation ID, and then we use the built-in from BPF trace,
which computes rolling average. So, we do that for both memory and CPU.
So, okay. Oh yeah, and finally, yes, sorry, this is a bit of a big slide.
But the last thing is continuous logging. So, we refactor, this is the code where we were previously rotating buckets.
So, instead of rotating buckets, now what we do is, we iterate through the aggregation, we get the count,
we get the total cost in terms of memory CPU, which is the average built-in applied by the counts,
and then we log it. But then we add our filter, our filter is important so we can detect was it after own data or not.
And then, we get more results and more censorship, but we have two new columns.
We have memory and CPU, and here you see it's ordered by memory, and it's arguably much better data,
because it's not such a naive way to estimate the cost. And, but we still have a bunch of open questions,
because actually, I ship this thing two days ago. So, I didn't really have time to look at the data.
So, it's like the follow-ups. How to best use this data set.
If you find like a top suspect, how do you confirm that it's actually a problematic use case?
Some people are so ingest and just replay it and see if you crash a prod, but not sure that's the best idea.
But yeah, so we have to play with this, find out how to best use the data set, and eventually maybe we can build some automation to do the policing efforts.
But yeah, there was my first BPF trace programming experience, and it's really interesting to work with the technologies like busy coding with hash maps, I would say.
And I just want to say a big thank you, also, to Daniel and Jordan. They really helped me answer it all my questions and do troubleshooting.
So, thank you a lot. And yeah, I mean, it's probably the best crowd to preach for BPF trace.
I'm sure you guys are all interested, but if you're a system engineer, I think you should really spend time learning this technology and add it to your toolbox.
So yeah, thank you.
We have time for questions.
Cool question. Oh, wait.
The rock for service?
Yeah, yeah.
Yeah.
Yeah, exactly.
It's just this one function. We added two probes to USDT.
Rockford is just a legacy name for scuba.
That's just the name of the binary.
No, okay.