All right, for the next presentation, we've got Vladimirov talking about lightweight XDP
profiting.
So take it off.
Hello, everyone.
Can you hear me?
Yes.
Perfect.
I'm Vladimirovaskali.
I'm a PhD student at Sabienza University.
And I'm actually here doing a listening video that you see
with Professor Tomberbetter and speaking of Professor Tomberbetter.
To my view, my interest that is actually open postdoc position.
So if you're interested, if you do networking smart nix and a speedpocket processing, please do contact him.
Otherwise, I'm here to talk about inspect, which is a lightweight profiling system for
ITP application, which outperforms both inefficiency and profiling accuracy existing
ABP profilers.
While ABPF is often used to monitor other programs and for kernel tracing, and bugging the suite of tools capable of
profiling the kernel part of the ABP application is limited with a pair of MPF tool
paying a key role among them.
So we evaluated five different XP application without profilers attached and measure their packet
processing rate.
And as we can see, most prominently in the drop application, which is a simple application
simply dropping every packet.
The drop goes from 15 million packets per second to less than four, which is a deep drop in performance.
And can also make it hard to profile fast network functions such as also not dummy once
like drop, but also CMS NASA and other can make them more challenging to profile.
But how do profilers work?
They rely on a set of specialized data to a register called performance monitoring counters,
to track data about the different hardware events happening in the system.
Most prominently, the extraction cycles, cache sheets, misses, and many others.
To profile the kernel part of the ABP application, Perth and BPF tool attached to programs,
the affinity and effects it around the target programs to read the PMC values before and after
then they compute the difference and store the results for the user space to be able to read it.
These affinity and effects it functions, although very fast, and probably the fastest one to
wrap around the program, are still very expensive, completionally expensive,
at the introduce as significant overhead as we saw before.
This profiling overhead comes mainly from the effects it, the effects it and the effects it
fentry and effects it functions, but as well comes from the Perth BPF Perth read value function,
which is the one used by Perth to gather data from the PMC system, which is pretty slow.
So these functions and these profiling functions get called for an HTTP application millions of
time per second, which ends up drastically distracting the throughput.
Not only the throughput is disrupted, but also the profiling accuracy, because as we see in the
drop application it should be only around two instructions setting the action to drop a
return in maybe a bit more, but Perth, the result of analyzing a drop program through Perth,
says that this program has executed 627 instructions. This is because the Perth profiler
considers also some of its own instructions used to profile the program before calling the
read value again the second time. So computing the difference adds up some some of its
own instructions instead. To solve this problem we develop the inspector, a lightweight HTTP
profiler, which has three main components, a user space components that is set up, that
tells the kernel part on which CPU events to record, two tracing macros to the limit
of the profiling section, the part that you are actually interested in, and a kernel module
to read the PMC's values efficiently called from the tracing macros instead of using the
Perth function. So these are the tracing macros, start trace and then trace, they can be
placed inside the any HTTP program, whenever you want it can be useful and can be used
to profile some blocks of code or even individual instructions. In the, both traces, both
macros, read the PMC values and store them, but the start trace is also managed to reach
an activation if you actually want to profile the set region, and the sampling rate that
will talk about in a few minutes. The entries instead, that's the difference between the
start trace function and stores the results in any BPF map for the user space. But how do we
access these values? So BPF has a little distraction set, and does not allow the use of native
Excel, it seeks instructions such as RDPMC, which are needed to read the PMC values very
fast and efficiently. This is why Perth has to use the BPF Perth or it value well per function
to access them. To overcome this limitation, we developed, we developed a Linux kernel module
that exposes a key function that invokes this RDPMC function, and retorts the value from
the, from the counter itself. So by calling the macro directly from the HTTP program, we
remove the need for effect, sit and defend, which saves us about 200 distraction for each
call. Then directly accessing the PMC through the key function saves us another 200
entity instruction, because we are not calling the BPF Perth or it value. As we can see in
this graph, on the left we have the regular Perth profiling, on the right one we have
where we still have around 40 instructions for each macro, which are mainly due to the
call to the key function. These are some of the programs we used to evaluate our
profiler. The drop is the dummy application that simply adopts everything. Accounting
sketch is used for monitoring flow traffic and stores data inside a pretty big map,
and not translate the BP, which is pretty similar to the tunnel, which does some
imp encapsulation and router, which is our obvious, that simply that looks up
in a pretty huge LPM3 to access the routing information.
So as I told you a little bit before, there is some profiling inaccuracy when the profiler
is actually pretty heavy, like Perth and BPF tool, because they add some of their own
instruction to the instruction count. Here in this graph we can see the retired
instructions that the application says that the profiler says the application is composed
of or is actually running. All of this should be the same, but they are clearly not,
because Perth and BPF tool are pretty heavy, and at more noise inside the profiling
result. For example, the case study that drop application on the simple one that we
showed you before, we expect around 2 per 600, which say about 40, so we are not
perfect ourselves. This inconsistency and accuracy also happens in other
while profiling other metrics, such as cash misses, eats, whatever, because it can happen
that these events are caused by the profiler itself, not by the application under
test. These are the results of the throughput while our application is the
very supplication, are attached to different profilers compared to the baseline.
Here we can see that our application, the inspector, is quite a bit better than the
other profilers, but still have the performance of most of most XP application.
The worst happens in drop, because since it is a very lightweight instruction that
the weight of the profiler is proportionally high. To mitigate this problem, we implemented
the sampling functionality that will increase performance by maintaining good
enough results, let's see. Our sampling mechanism is composed of a simple counter
that checks if the packet is in the actual sampling period. If it's in the sampling period,
we check the PMCs and computer results and store them and do all the regular stuff.
To somewhat fairly compare against the perfe, we also implemented the similar functionality
inside the perfe. This sampling functionality, which was not supported, is still not supported.
The time of doing this work for perfe starts.
However, the performance gains from perfe are limited, because it still has to
check to call the aventry to check if the packet is in the sampling period.
This call to the aventry is still pretty expensive, even if you are not doing the perfe
value event function to get the values. Instead, since we do this sampling inside the macro,
which is pretty lightweight and pretty fast, we can have some better results.
Almost reaching the baseline without no profiling attached. If we sample every 64 packets,
let's say we reach almost the performance of a non-profile application while maintaining good enough results
and accuracy. In this case, we are counting L1 cash misses. We get good results and expected results.
So, to recap a little bit during our work, we identified the main sources of order at the specific profiler,
which resulted to turn out to be the aventry, effects it, and perfe value.
Then we develop this, so this function are necessary if you actually want to provide the application as easy,
without modifying it, because you are looking around the application set.
If you want to modify it, or you prefer any better performance, you can keep most of this function by using,
calling the profiling itself inside from inside the XTP, and using K-function to access the PMC more efficiently,
then we also implemented the sampling functionality to further reduce overhead.
So, turns out that inspects, again, in this case, perfe, it's 71% faster without any sampling functionality,
and 122% faster with sampling, sampling against the sampling velocity.
But more importantly, we get 73% less instruction, no, it's at least while doing the test with instructions, which is pretty good.
So, thank you very much for the attention, I'd be brief about that.
Thank you.
Good time for questions.
Thank you for the talk, very interesting project.
So my question is, if you can do the profiling mechanism like this for XTP profilers,
can you adapt this to regular BPU programs?
So, is the overhead basically the same or very different?
Supposedly, we could, because the problem is being able to call data function,
we should be able to call from every BPU program.
The problem is that the performance gain are much higher in XTP,
because the application gets called millions of times per second.
So, a different BPU application might not get the same benefits,
because probably it's called less, let's say, or if it's called the same number of times, it could be useful.
Thank you.
By the way, I love that B flying from the behave to the flower.
Thank you.
I was wondering, like instead of the K-Fank,
did you also consider to implement that instruction natively,
as a one-to-one mapping in BPU by extending BPU instruction.
Because I mean, that would still give you even better performance,
I would expect, and are you planning to submit some of that upstream?
So, there is the idea of doing that.
Maybe doing it like the GET timer function that's in BPU helper,
so we can call this function and get the results.
Without all the infrastructure around it,
but of course the infrastructure is still a bit needed,
so it becomes harder to do something like that.
Have you been measuring the sampling overhead
over the testing mechanism?
What's the sampling mechanism itself?
What's the overhead of it?
It's pretty low.
The overhead, let's see, this one,
depending on the sampling rate you are doing.
The black line is the baseline,
so anything below it's our result,
and this is the inspect without sampling,
so we gain this much if we sample every.
This is to be found.
Something or it's with something one-to-one.
This is without sampling,
and this is by sampling every eight packets.
Yes, but I want to know the performance if I add sampling,
but I sample every trace,
so then I know what's overhead of the sampling mechanism itself.
Ah, okay, not so we didn't do this kind of test.
Thank you.
Hey, thank you for the talk.
I was wondering if you know what the extra work
that BPS per feed value is doing
beyond just calling the native instruction.
So it's mainly pretty heavy
because it uses file descriptors to access these values
as a reading of high let's say,
and that's the main overhead that happens in this function.
Thank you.
Any more questions?
I had a question.
Is there any of the changes that could be brought to PF
or to BPS to make them faster based on the work you've done?
Yes, I know.
The no-party is because since we are calling a key function,
it's a relative to our machine,
and it would be hard to ask someone else
to include a key function inside the kernel
and call a key function not knowing what it actually does.
So that would be the problem.
There could be a way of doing it without calling a key function
but by simply calling the per feed value function
inside the BPS program instead of calling the key function.
So you can remove the per feed, the fendry
and the effects it's overhead.
But it's still pretty heavy to call that function.
So the games would be marginal, let's say.
All right, thank you.
Have you run your tool in production?
Have you used it so far mostly for experimenting?
Or have you tried actually running that on production?
No, it was just experiments like this.
No actual production testing.
It would be interesting to get some benchmarks on that.
Okay, thank you.
Someone there has a question.
All right, thank you.
Thank you.