So if we get every attention, we have our next speaker.
This is everyone likes profiling.
So Brennan is going to tell us about profiling rest applications.
Thank you.
All right, so hi, I'm Brennan.
And when you're talking about profiling rest applications with Parko,
has anyone here who is not an employee of the same company as me ever heard of Parko?
One, two, three.
OK, perfect.
A few more of you, hopefully, will by the end of this talk.
Hello, my name is Brennan Vincent.
This is my email address.
It'll also be at the end if you want to contact me with questions.
I live in Tucson, so I had to come to Brussels to see sort of like clouds
and wind and that kind of stuff.
I'm a team member of Parko specifically,
I'm a maintainer of the Parko agent kind of sub-project.
And I'm employed by Polar Signals,
which is the main company that funds Parko development.
So first of all, what is profiling?
I can give a sort of abstract definition first, analyzing resource usage
by inspecting running programs.
So more concretely, what do we mean by resource usage?
Very things like CPU time, memory, while clock time, GPU utilization,
whatever's the resource on a computer that you care about being efficient with.
The goal is to usually attribute resource usage to a location in the code
or even better a stack trace.
So you might want to say something like,
I ran this program for X amount of time.
And during that X time for Y CPU seconds,
this stack trace was visible on the,
it was active on the CPU.
So how do you make that visible so that people can understand it?
There's a couple of kinds of visualizations.
Most popular probably annotated source for assembly listings,
so saying like this line of code to this many resources when we ran it
on flame graphs, which I'm going to go into more in a bit.
There's some famous profilers that probably more of you would have heard of
in parka.
There's Perf, probably the most famous one on Linux,
Firefox Profiler, which is mainly for client side JavaScript code.
It's bundled in Firefox.
And then there's Apple's Instruments.app for the sort of iOS, macOS, et cetera,
ecosystem.
So here's an example of Perf and a script called FlameGraph.pl,
which is what I used before I heard of Perka.
So this is a function that prints the 40th of a Nazi number over and over,
and using a naive recursive algorithm that is very slow,
but we're going to look at what it looks like in Perf.
Okay?
Hopefully this is visible.
So okay, I remember leaves this the same program.
I'm not going to go into like the meaning of these flags,
because the point of the stock is not to talk about Perf,
but we'll let it run for a few seconds, kill it,
and then do Perf report.
And you see this kind of confusing interface in my opinion,
but basically 98% was in this test rust fib function,
and we can annotate it and see, okay,
a lot of time was spent in the function prologue,
whatever stuff like that.
But like I said, I think it's a little bit of a,
and then it's a little bit of a clunky interface.
So this kind of brand-end rig,
made this tool called FlameGraph.pl,
which translates Perf output to FlameGraphs.
I have a screenshot of what that is.
Basically, if you haven't seen a FlameGraph before,
essentially, if one of these rectangles takes up,
let's say 60% of the width,
then it means, in 60% of the stock traces,
that frame and everything below was like the frames
that you can read off of the graphics.
So like for example, let's say 60% it's one of these like test rust fib,
that's like 15 layers up.
That means that in 60% which I'm making up on this eyeballing this,
but of stock traces while the profile is being taken,
we had these sort of startup frames, main,
pre-main, whatever, and then like 15 layers of fib.
So I'll show more realistic example,
especially since I'm here to talk about parka,
so let's talk about what parka is.
So how does parka differ from these kind of,
other pro-followers that exist like Perf? Why not just use Perf?
So the first and biggest difference is that it's intended to run
all the time, profiling everything on the host,
which is what we call continuous profiling.
Whereas Perf, it's more like, okay,
decide I want to profile something so I'm going to run it under Perf right now.
It has very low overhead.
Obviously it's going to kind of depend on what your specific workload is,
but the rule of thumb we try to use is it should take less than 1%
of CPU time overhead across the entire system running at all the time.
Obviously this second point is what enables the first point.
And if I have time at the end, I'll try to talk about how it's implemented
in such a way that the overhead can be solo.
You typically have end machines sending profiles to one backend,
so you run the parka server on one machine,
and then on every machine that you want to profile,
you run the parka agents pointing out the server,
and it sends profiles to the backend.
And the point of that is that you get a browser interface
that generates flame graphs on the fly based on queries
across everything that has been submitted to this backend.
So you could say, for example,
generate a flame graph of all the stack traces I were taken
from 10 a.m. to 11 a.m. yesterday on nodes that match this
projects with process scenes that have that match some string,
et cetera, and then it'll go back to the backend,
the backend will generate the data and then render a flame graph.
So let me show you a real world example.
I didn't want to use multiple machines because I thought if the
conference Wi-Fi goes out, it's going to make for a bad demo,
so just like believe that this could have been done on multiple machines.
So it's going to be done on one machine today.
So I'm going to run the parka back ends.
Don't worry too much about these flags.
They're all documented, but probably don't have time to go into
how to configure it.
But I'm going to also run parka agents,
which has to be run as route on every machine that you want to
profile.
If you use Kubernetes, it's a Damon side or whatever run it on every
machine, but in this case, I'm just running it on the command line
locally.
Okay, that's running.
And now for a workload, I'm going to run RG, which is sort
of like rip crap across my entire file system,
looking for some random string over and over.
So once everything's in the file system cast,
it should be taking up 10 cores or so.
And then for good measure, I'm also going to run this
Fibonacci program, why not?
So let's see what that looks like in parka.
This is Wikipedia.
I did not invent this, unfortunately.
So this is parka.
You get this sort of metric graph here by default,
broken down by comm, which is Linux is kind of word for
process name, roughly speaking.
So the one using the most is we expected is RG.
Then there's a test rest that's a Fibonacci program named kind of
uninspiredly, but then we have other stuff like
GNOME shell, just from Ubuntu, Firefox, parka,
agent itself, GNOME terminal, whatever.
So you can see it's profiling everything on the system.
There's a flame graph of everything going on on the system,
but it's a little bit more interesting if I show you the
filtering logic, so like let's do comm equals the
RG or test rest to sort of show you, you know,
the workflows that we actually care about.
And then you have it here, and then the flame graph itself,
you can also group by various things.
So I just grouped by comm.
So now we have on the left comm equals RG, which I'm going to
kill these by the way, because I don't want to run out of battery.
Yeah, so you see on the right is the Fib stuff and sort of
the form of this graph, other than being upside down,
like the root is at the top, but other than that, the form of this graph
sort of looks the same as the visualization I showed you before,
we can zoom in.
So that is a bug, sorry.
Of course, the bugs happen when I'm giving a demo of
costum, that's time for them, but anyway, okay, we're back.
And then on the left here, this is seemingly realistic
stat frames from RG, like whatever.
RG search path file open, et cetera.
This sounds like RG stuff.
So I understand what we did there.
So let's go back to my slides.
So it's reasonable for someone to ask, well, what is,
what is this guy's talking about?
Have anything to do with Russ?
Like, why are you talking about this in the restroom?
And this could have been, you know, park it,
can also profile C++ and go and whatever.
So for the second half of my talk, I'm going to talk about kind of
some rest specific features that we added.
Which do require instrumentation, like you have to decide
I want to use this feature from your code, but that's okay.
So the first rest specific feature is custom labels, which lets you
annotate regions of code with application specific labels,
dynamically at runtime.
For example of this, let's go back to my VM.
Okay, I don't know if people can see this, hopefully you can,
but this is essentially a web server that it takes in a user name
and length.
It generates many bytes and takes the shot to 56,
some of all of them.
And then it renders string that says,
thanks, username, the shot to 56 of your random data is whatever.
So you can, you can run this and then you can profile it in park
and you'll see about 70% of the time we've been spent generating
the shot sum and then 30% of the time generating the random bytes.
But what if you want to know how much resources was consumed
by a person with each username?
So we published a library called custom labels, so let's instrument that.
So let me edit the code to have that with label.
username and we'll make the value of the label be like the actual username
and then you pass it a function and then essentially it runs that function
and while that function is running, that label is applied.
So let's run this.
Let's hit this.
So I'm going to do username equals brand-in, length is a billion bytes.
Okay, thanks, brand-in.
Your shot is whatever could not have predicted that.
And then I'm going to do, okay, I'll do username petros
because then others are getting petros watching this online.
So hello petros.
Thank you.
Your random data is shot sum blah blah.
Now let's see again what that looks like in parka.
Okay, the comm is Russ Shaw server filtered down to that.
And now in labels, one will show up called username if it worked,
which I didn't, sorry.
Another bug perhaps.
Oh, it's because I didn't save main data as soon.
I actually added the custom label.
Not a bug in parka.
It's a bug in my ability to use VS code.
So let's run this again.
Let's do one with brand-in.
Reload it again.
Use some more time.
Now let's hopefully this will show up in parka.
Okay, now we can group by username.
And you see, okay, the one with no username on the right.
That's from before when I screwed it up.
But then you see on the left of username equals brand-in.
Was these stack frames, username equals petros.
Was these ones in the middle here.
So in real life, our users were using this.
Use it for stuff like trace IDs if they do distributed tracing.
Or if you run a SQL database, you can say, okay, set a label for the query ID.
And then later if some query was taking a ton of time,
you can try to kind of investigate why with the profiler.
But let's go back to the slides.
The second feature is memory profiling.
If you use J. E. Mallock, parka, can scrape profiles of memory.
That has been allocated but not freed.
And I consider this a rust specific feature because you can technically use this from C,
but we wrote the library that kind of translates this in rust.
So it's much easier to use from rust.
So let's see, let's say I had some code now that with a 10%
every buffer of a thousand bytes with a 10% chance leaks it.
So we'll get some, we'll get some weak memory, not free size.
Forget it.
Okay, you guys all know rust, so hopefully you believe this is leaking memory.
Going to save the file this time, I'll be great.
Let's load it a few times to waste some memory.
This should be leaking about 100 megabytes every time.
Okay, that's probably enough of that.
And go look at the memory profile.
Okay, you see now it started close to zero and jumped all the way up here.
If I select one of the points on this profile, this is okay.
We got to fail.
There's other people who work on parka watching this.
Let's figure out what's causing this bug.
So unfortunately I cannot show this, but if I clicked on this,
you'd see a breakdown similarly a flame graph of the bytes that had been allocated
and the stack trace is where those bytes were allocated.
Sorry?
Okay, I can try that, let's see.
This is, no, this is not.
Thank you, Frederick, that works.
But yeah, so you see, basically this shot ran function,
you know, it says it's in main.rs plus 34, so line 34.
That's where stuff is getting leaked.
And indeed, if I go back to my editor,
that's indeed line 34 where we're leaking the memory.
So that's the second rest specific feature.
I'm going to have to be brief for the rest of this,
but I just have one more slide.
How it works.
Keyword is EBPF.
Have you heard of that?
Essentially, I don't know, well, I'll time for you guys to digest this entire slide,
but basically the idea is that with EBPF,
you can write something that runs in kernel space,
but so in the, like, in the address space of your process,
but the kernel runs it.
And so you're not having to like,
pop you a bunch of memory back and forth to different,
to different address spaces in order to walk the stack.
And I should give a shout out that,
our on-winder and our agents is largely based on
although we have our own kind of features,
but it's largely based on this project open tomatry EBPF profile,
which several companies, including us,
collaborate on.
So thanks to all of those contributors.
And yeah, so if you want to learn more about parka,
you can join our discord, which is this QR code,
website is parka.dev,
or you're going to email me, this is my email address,
and that is it.
So I think we have like five minutes for questions,
if anyone has questions.
For the custom integrations, did you come?
Sorry, can you speak up again?
Sorry, for the custom integrations of labeling,
are you using like the tracing crate that's built in or?
Do we consider what?
The tracing crate, a lot of libraries use.
Yeah, custom integration with the tracing crate.
We have considered that, but we don't,
we are not implemented anything along those lines yet,
but like, it's definitely something we want to do
because it's sort of huge in the Rusty ecosystem.
With custom, custom user space decisions like building
your own stuff into the binary, like you did there with the user name.
What is the limit of using EBPF like that?
If I go ham and start adding every single thing,
like let's say I start putting in every time you read a single bite,
I will write down the memory address.
You're talking about custom labels features?
Yeah, custom labeling.
Yeah, so there are pretty tight limits because we do have to read those in EBPF.
So it's like around, at depends on version of perka agent you're running,
but it's around like 10 unique labels.
And the key in the value size are also constrained,
but we've made them long enough for like typical things like UIDs and stuff like that.
Do you have any experience on what is the practical performance limit of using EBPF
in that sort of like jumping into EBPF to do some kind of labeling or something?
I don't think it has a particular significant performance impact
because it runs by default 19 times a second,
and then we just got a copy out like a tiny bit of memory when it happens.
Like I said, the park agent itself has to use some CPU,
but we think it should typically be like again less than 1% of CPU.
Speaking of CPU overhead and such,
have you ever or has anyone that you've heard of ever tried running this on embedded Linux devices
to profile stuff remotely?
There are, I can't say the name in the company because it's confidential,
but there are people using this on things that are like out in the field
that don't have network connections.
So we do have like a, we do have a mode where you can say,
if you have an offline mode and then like later I'm going to upload all the traces.
When I said embedded, it depends on what you mean and I'm better
because I should mention this only right now works on X86 and ARM.
So I have the idea someday that we should also port it to risk 5,
but for now it's only X86 and ARM.
It only works on Linux.
Hey, just one question.
How does it work with ACingCrust?
So we don't directly attempt to like reify the ACing like logical stack trace.
So you are going to see a bunch of inner stack traces,
a bunch of sort of Tokyo, this that like stuff in the middle, whatever,
but it is, again, something we want to do someday.
Most users in practice about it, it's still useful even with ACingCrust,
but it could be integrated more tightly for sure.
I have a question.
What are some of the, how does it handle high card now?
So if you had like a hundred or a thousand users using your system
and the name tags start to explore.
The performance shouldn't depend on the card now.
Okay, that's of the labels.
Yeah.
That's nice for people using Prometheus and tools like this,
because the card now becomes a problem.
Anything else?
All right, thank you so much.
Thank you.