Welcome, now we are going to talk about some interesting stuff about the job
profilers, how we are going to advance them, what they are, what we are now and so on.
So, let me start with this.
So, you know the situation when you come to party or meet new people and somebody just goes
like, what do you do for living here or stuff?
Well, I do job profiling a data doc.
So, what does it mean?
Well, you're on us, it's a tooling to collect data, explain data points and provide with all the
insights and, well, if you talk to me, I don't know, I forgot what you talked about.
So, what are you really doing?
Well, I just, and kind of performance crime detective, you know, it's like,
are you shooting software developers?
I wish.
No.
I'm working on tools which provide answers to questions like, what is working
to CPU?
Why is your potgettic on-killed?
Why, why your service is time timing out?
Why the latest is packing in so on?
And for this?
Do you, well, I have tools to talk to.
Yeah, well, as I mentioned in the beginning, we're going to talk about profiles, right?
So, we have the toolbox for Java and there's something we just called Java file
recorder, right?
We're just coming Java and async profiler and something called OELTIL
EBBF profiler, maybe you know about it, but not we're going to mention today.
So, there's a bunch of tools.
So, um, and now you probably give us an opportunity?
Yeah, I can do that.
So, what's this JDK slide record?
Are you heard you've done something on this 30 years ago?
Well, I just helping out, but here we have Eric Galling from the JFR team.
You could talk much more about this.
So, he's in the right team.
So, I would just helping out with some stuff.
But it's a really neat tool inside JGK.
It's a lightweight, not only profiles, it's like big,
observability tool, which can give you insights into how your JVM is behaving
also your application.
It has very minimal overhead.
It's extremely stable, extensible, and okay, let me repeat it.
It's built in JDK.
So, it's always there.
It sounds very good.
Can you explain to me how it works?
Where we get execution samples from it?
Yeah, well, the big execution sample is kind of CPU profile.
Well, it's approximation of the CPU profiling.
And there is this algorithm behind that.
So, if you imagine we have on the right side, there is a thread list.
We have in Java and JDM.
So, JFR sample goes in, and each and milliseconds.
By default, we have like 10 milliseconds of profiling or more.
It scans the thread list, and picks next, non-blocked JavaScript.
That means the thread which is not worked.
You just kind of running.
Then it suspends the thread, and that takes the stack trace, and generates the sample.
Then it goes back, picks.
Next one, non-blocked, JavaScript does the same thing.
But sometimes it can happen that the stack, the acquiring the stack trace will result in error.
In which case, the JFR sample I currently would just skip it.
Okay, I don't know what to do with this.
So, it will just skip it.
And we'll repeat this until it fills up like five sample slots.
Right?
So, this is happening each 10 milliseconds.
I mean, that's fine.
For most of the users, it will give you pretty good view of what your CPU is doing.
But this seems to have a problem, for example, because we're using the sliding window.
We're getting only like five stack traces.
Every at least you know seconds.
So, that's a problem, because for me, it now blocks like the effective sample period of every set.
The process with the number of threads and you can clearly see that the effective sampling rates
scales linearly with the set count and the stack blocking error rate.
So, that's pretty bad, because now we don't have any clear relationship between CPUs and samples.
Between CPU time and samples, and that's not, not, not print, isn't it?
But I use the code of our profiles.
I'm not looking at you specific profile.
There were profiles that had some safe point bias.
So, essentially, that meant that, so we safe point essentially means that
their points in a program where you can reason about a program.
For example, that he has a specified state.
This is the point where we do, for example, garbage collection.
And do we have still safe point bias?
No, no, no.
No, we don't have.
No, no.
In JFR, when JFR is not stopping our JFR sample for the execution sample,
is not stopping at safe points.
So, it will just land at any place where it happens, right?
But this is the fortress?
Well, yeah.
But there is this thing called symbolication, where we actually,
we need to get the human readable form of the frames,
like in the stack trays, we have frames, the frames are basically addresses,
and we need to somehow figure out what the hell is that?
What is the name of the method and line number?
And this turns out to be slightly surprising,
if you are not very familiar with things.
So, how we do that?
We ask JVM to give us the information.
For the interpreted code, this is pretty easy.
Let's say easy.
The JVM does all the head lifting.
We will get the full information for the interpreted frame.
That's fine.
But for the jit compiled code, the jit compiler, it's a big beast, right?
So, it will just transform the code over the place,
and it might get difficult to actually get back to your source code.
So, what it does, it inserts something which is called the debugging form,
in places, and the symbolication part is using this debugging form.
So, that means like you see, we have a bunch of, this is, this is assembly, by the way,
of the jit assembly, right?
And we have a bunch of instructions, blah, blah, blah, blah, blah,
and we land the sampler lines at a particular instruction,
and then we need to figure out the debugging form.
So, we need to do this, we need to jump over all the instructions,
not that we hit the debugging form, then we will say, okay.
So, this texture is mistaken at this particular place.
Here, it's like, line number 17.
This is the default.
And if you, I'm pretty sure some people from the jit teams are familiar with the jit.
So, this is one of the places when you look at the code.
One of the places where jit actually is placing the safe points.
So, the one loop goes back, there is a safe point, usually.
Well, the jit people could talk more about this, this would be like another one,
another talk, at least.
So, we are getting debugging for only a set points.
So, that means that even though the sampler is not safe point biased,
the symbolication will actually go biased back to the safe points.
So, that's like, yeah?
But can we do something about it?
Yeah.
There are people at jvm.
They came up with the jvm argument, debugging on safe points,
which you can provide.
And when you do that, suddenly boom,
you have a bunch of debugging force, like,
sprinkled around the samblic code.
And this makes it much more easier for the profile
to reason about the frames were blended.
So, yep, just one jump, just one jump away from the happiness.
And you have the frame there.
And you see, instead of line 17,
we just do a loop package, we are in the line 16,
which is actually the code, which was executing at the time.
And if you are looking for hotspots,
like moving to the safe point can be quite misleading.
Like, you could be trying to optimize starting,
which is not really the hot code, right?
So, this are the jvm arguments,
which you should provide when you are trying to sample,
which are trying to profile the Java application to get,
like, the most precise results.
Good thing is, like, just to small note,
there are, there are profiling tools, like, yes,
as in profiling underpugging city here,
which are setting these flags by default.
So, when you are using as in profiling,
we will talk about setting profiling bit later,
but that will just send this flag by default,
and you will get really, like, fine detail data.
To summarize, jdk file recorder,
amazing tool is a big question,
everywhere in the jdk is from Java 8.
Up, it's stable, safe,
it's not going to crash your application.
It's fully supported.
Like, open jdk or a code, it's like, yeah,
it's something very standard.
Small downsides, what we find is that
there is no real CPU sampler,
that's the thing I was talking about.
This approximation might be sometimes not good enough
for the precision you want to get.
It fails silently for errors,
so that just jumps over.
So, the static statistics can be slightly skewed,
and currently there are no, like,
native frames in the spectracy.
So, we don't have mixed spectracy,
jdk is 9 plus native.
It's like, it's not big,
it's like something, yeah, it would be nice,
but.
So, second thing, right?
See, it's async profile,
looking back.
So, this is amazing tool,
based on something,
which is called asyncetcultures,
which can many, many years ago,
for prehistoric performance,
to some performance studio,
whatever.
And async profile actually made
really great use of it.
And unlike the jfr CPU sampler,
async profile is not using,
like, dedicated thread,
and going through the threads,
but it, it will use
kernels, scheduler support,
to actually blood the kernels,
send the signal every time,
a thread has consumed,
a certain number of milliseconds
of the CPU time.
It will send signal.
The signal we will process,
the signal handler,
the thread will be interrupted,
and then the adesync profile,
which is, well, the stack,
it will combine the asyncetcultures
and the native stack working,
to get you, like,
combined stack trace of the place.
That's easy.
Easier said than done.
But, yes, it's working perfectly.
The number of samples you are getting
is actually proportional to the CPU activity of the threads.
So, you can reason about the CPU activity of the threads,
based on the number of all,
on the proportions of the samples collected.
Right only works on the lyrics,
but, yeah.
Yeah.
I include, like, trick prints,
something for winners,
but it was told it was dark,
and I had to,
no.
You shouldn't do it at home, kids,
but it's usually only works on the links in that.
No.
Yeah.
So, yeah.
This is how you use,
if you want to go and get your hands dirty
with the asyncetcultures,
this is how you use asyncetcultures.
You need to, you need to resolve the symbol,
then you can use it.
Well, of course, it's unofficial.
So, whatever happens is like, yeah, hands off,
if you're a problem,
but I'm not saying, like, most of the time it works.
Sometimes it might crash.
100% is struggling, I'll need to pick
the most of the breaks, doesn't it?
They're my crushers, right?
There you are.
And the funny thing is,
there was,
it's a key, one test case in the open sheet,
and it's the only one in the end
for, like, many years,
but it's really an unofficial.
Yeah.
So, that's why, like,
asyncetcultures is trying really hard
to get to slightly different base.
So, there is this thing,
which was interesting, I don't know exactly.
It's like, half year or something.
It's being struck based stagwalker.
It's like handwritten stagwalker,
based on something called VMStruct.
And VMStruct,
which allows you, basically, to crack open JVM materials, if you know how.
Again, it's unofficial, not public.
So, you need to do all this dance with reading symbols,
and you actually need to go and read the,
the VMStruct CPP.
See, down there?
So, this is just an example.
Like, the file is huge.
You can go to OpenJDK source code.
You can read the VMStruct CPP.
And then, you can read the VMStruct CPP.
And then, you get, like, glimpse of what everything is there.
It gives you access to, like, almost everything.
It was edited for serviceability agent.
So, it's pretty powerful.
But, yeah, again, unofficial.
There is no API.
This is, this is on your own.
Um,
summarizing profile are really cool to, like,
get, it's doing the best what it can with the,
like, backing technology that means,
as in profile, I think it call trace, a VMStruct.
It provides the mixed tech traces.
And I must say, like, it's fairly stable.
Like, even with the, I think, it call trace, some issues.
With the VMStruct, it's, it's pretty, yeah.
I would say, it's not crashing.
Wow. Yes.
I think it's not crashing.
And it almost, and almost everybody,
for us in profile, and it's new functionality.
So, it's open source, it's fine.
And so, and it's the basis of many external levels,
many external profiling tools, or all your application,
performance, one just, probably use us in profile,
or photograph it, another one.
Yeah.
I'm so relies on chipmool intelligence, which is.
Yeah.
That's, that's the thing, like,
since all those things are unofficial.
Um, in fact, there are no guarantees, right?
Because, like, basically, you are relying on internal
implementation details.
So, they, they're, they're cannot be any guarantees.
So, that might be improved, I hope.
Somehow.
Oh.
And yeah.
So, this, this is our tutorials,
which I've been using almost daily for,
last, I don't know, how many years.
Uh, there's this, the, the E, E, E, B, B, F.
I think you, you did something with it, right?
So, if you see me, like, running out,
I'm running to the E, B, B, F, different, doing data.
But anyway, so, um, there's also, uh, the open technology
E, B, B, F profile.
It's essentially runtime, I'm not,
it's exempling, written in E, B, F.
But what is, uh, and it's, like,
from the CNC information.
So, it's also a stable open source project.
So, for it's many languages,
designed to travel, but also supported to travel with nature.
And, whole works, which is E, B, P, F.
E, B, F is, uh, a technology that makes the lives come
for comical, at native execution speed.
So, essentially, you can extend the literature
and the touch to hooks.
How this works?
Just half a minute on this.
Um, is that you compile your program.
You can write, for example, and see,
compile down to a bike code.
It's a fairly simple bike code.
And that just complex and intricate.
It's a shower once, but pretty simple bike code.
It's a pretty simple register machine.
It's like seven registers and so, um,
and no rules are recursion.
Take my not really looks, but, but, no recursion in here.
Um, what you don't do, you load just into the load scale,
because this program runs in the linear scale.
And we have a verified, like, the verified,
that you don't also have a bike code,
verify it in, like, the object,
or basically prevents you from having
sit in teaching files in the linear scale,
which is pretty bad.
Uh, typically, because of pressure system.
And then, you know, if the chip compiler,
uh, the JVM people in node and chip copars are really fun.
It's like, on all the major platforms, like,
you see, in the 390, um, and it really takes 862.
Um, do you know, the teacher's program
chooses some calls to Thomas and more.
And you can essentially do the same thing,
as you do with us in profile.
Do also have to secure Thomas and anything.
And events, but the cool thing is,
it runs in the linear scale.
So you have system-bindness ability.
Um, and also, it's far faster,
because you don't have to jump between
the profile and use the, like, all the time of the signal.
And in the end, it just uses VM strength,
which is essentially asking profile running in linear scale,
which is kind of cool.
So it has a few advantages,
essentially the same as us in profile is just a little bit more stable
it's backed by the scenes here,
which is a slow larger, um,
and it's a standard as product all in history cool,
but it's still relies on JVM and also
we have the same problem as we're passing profile.
And I talk with people that we're working on,
um, hotel or hotel travel profile.
And while, like, yeah,
it's interesting that we rely so much
on trying to travel.
Or, and they're slightly worried.
But anyways, um, so, um, yeah.
Yeah.
I think we just went through, like,
the most used profiling tools nowadays in JVM.
And I would like to talk,
or we would like to talk a bit more about how we would
prefer advancing this.
So I, I saw, like, you were working on some
jet thing, like doing, can you?
Can you tell us more?
Yeah, yeah.
And we're interesting, kind.
So, um, but first, we focus on JVM,
not on things like us in good cultures in profile,
so just because we want to have something
into OpenGK that's tested all those harmonium.
JK tested that people that work on the OpenGK
actually support, because we're in a changing
JK's make, they don't crash our tool,
because it's in the JVM, it's in the JVMK.
So, the test run all the time, um,
and that's really good to, we have for support solution.
Um, and one of the things that we, that we're currently working on,
um, this fingers trust we look at in my grindout,
comes into JK 25 as a property cutem sample.
That's what I'm working on, like, for the last few years.
So, if you're working on profing lovers, which you can tell us in a minute on,
we've won, have mixed settings, hopefully in the future,
like with nature frames, especially with panome project panome
coming in, where we have, um, much more,
where we have to ease the ability to call,
see functions all the time.
And, per small, we're thinking about this profiling
as slowly young, it's still young,
but it's growing exponentially.
So, uh, I'll, I'll start with CPUX.
Yeah, exactly.
Um, it's essentially, we're doing essentially the same way,
so simple for all those.
We use the timer frame functions that we use.
It's a CPU, we use CPU timers,
um, on Linux only, um, this gives, um,
um, and how this works, essentially, that every,
a few months, it's, of run time,
the fastest computer has consumed,
and CPU, um, we jump into a signal envelope,
we trigger by the kernel, and we want to stack trace,
but we only gain, like, the basic addresses of functions.
We have to some polycharicate, we have to some polycharicate,
later in a java effect,
but this hopefully changes in the future,
I think that, they're pretty cool people working on this,
um, let's see how it goes.
The causing is this, with this way,
we, um, losing tools, the main hindpoints that we see
and, uh, java are currently,
so, uh, that it felt soundly,
because we, we record when something fell,
so it's good to know, um,
and that it has known, real CPU time signboard.
Um, yeah, yeah, profile levels, very fast.
So this is something, this is not new concept.
Like, um, Marcus here,
that got somewhere in this email column of conversation,
they started, like, more than 15 years ago,
like, thread coloring,
but, what is it about?
We want to add more damages to profiling data,
than just, like, CPU time span,
the rocket memory located, right?
So you can think about, like, you,
you could attach something like,
just, uh, API calls or,
in generic operations to your profiling data,
or distributed tracing ID.
If you are using distributed traceers,
custom ID.
So then that, that will allow you slice and dice this data
in, kind of, analytical fashion.
So here, we have, like, example,
we have this bunch of samples here,
which are taken from the thread T1,
uh, and this stack, yeah.
So this, this is basically something we just called,
a flame graph.
It's a nightmare of its product.
I'm initiated essentially visualization
after runtime offer, of what a process does
during runtime, so essentially,
we throw all the time information away,
um, like, the specific time information away
and just plot the proportion of the runtime of each method
in the stack, relative to the other methods.
So, for example, we see here the top methods on the,
above, I like the top calling methods,
they call down, and the deeper we go,
the deeper we go into the stack,
and this critical visualization TTC all the time.
Yeah, yeah.
So, now we go back through the flame graph,
and you can see that we have operations.
We have rest of the UI, and on the left side,
we have colors, and we can slap the same colors
on top of the samples.
So, and we don't have to,
we don't have to stop at one dimension,
we can add more dimensions, whatever makes sense for us.
So, we can add, like, customer,
and then we can recolor the samples,
which are related to this particular big customer
to a nice orange.
And this is how the flame graph would look in real life.
Like, it's not going to be nicely chronological order,
it will be kind of mixed up,
and you can do, like, okay.
So, I want to see only the rest related samples.
Boom, you have it.
Like, you are focusing just on the rest.
And then, when you add the second dimension for the customer,
you can, like, laser focus just on the data,
which might be critical for you.
So, you can remove all the big data noise
in the current profiling tools
we are running on, like, huge services in the huge fleet.
And in data dog, this, this is proving very invalidable.
Like, we are having really good response from our users.
This is not just something I'm coming up,
or you're just going, really, really want to have labels.
Auto is also looking at the labels.
So, this, this, this, something which is getting traction.
And we've been talking with a gallon here for a while,
thinking about ways how this might get incorporating JFR.
So, keep on watching this space.
That might get interesting.
And what about the auto profiler?
It's, it's true in a good place.
We'll see how it goes.
It's probably, it might be the future we did,
then, maybe to integrate GDK, JFR data.
Maybe not, let's see.
Yep.
So, thank you for your sitting here.
I'll listen to this.
I'd most want to speak better.
If I didn't, most will be honest, I'll tell you my book.
And we're also back right here.
You can find me over here.
You have resources linked there with all the links I,
or we used here.
Thank you.
Thank you again.
Thank you.