So, on top, who knows who it is, and everybody dealt with Zoom killer errors before.
They're kind of a pain, right?
So, like, your programs running along, and somebody allocates memory that might not be your program,
and the kernel decides that your program is the fat program that's using too much memory.
It's killed, and then in your demessage log, you see some kernel stack trace and nothing about, you know,
what your program is doing at the time, so it's like, it just got nuked.
So, this can be thought if you're in, you know, production and trying to solve these issues, it can be a challenge.
So, what is Zoom Proph, so, Zoom Proph is a set of EDPF programs that knows how to read,
and it actually goes internal memory profile or data.
And at Zoom time, it actually goes and reads that, records it to some EDPF maps,
and then allows after your program is dead for your memory profile to be saved and uploaded or logged,
or whatever you want to do with it.
So, that's it, so we're done.
So, the motivation for our signals provides the profiling solution.
You know, CPU profiling is kind of our bread and butter, but memory profiling,
and other forms of profiling are things we do as well.
But, you know, if you're profiling memory, you may not get memory profiling typically works by scraping
and programs, maybe like once every minute, once every five minutes, you scrape your p-prof
endpoint and get a memory profile, but that might not be good enough, right?
So, like, sometimes you have some allocation that is too varied,
and you can just die right away with one allocation, or sometimes you have some, like, service failure
and you start getting to some failure, retire loop, and you kind of can die right away
and those allocations may not show up ever in a profile for using ad hoc profiling.
So, that's the motivation.
Not really.
Actually, the motivation is that I've been battling, um, killing problems for a longer than I
cared of in that.
I used to work on databases, and most working on garbage collectors, and, you know, memory
when these problems happen, it's like, it's got to be a better way, right?
It's got to be able to figure out why this item happened and understand what happened.
So, the real answer is that this was an issue I had while I was working on something else,
and because poor things are such a cool company, they allowed me to go take a little while and work on this.
So, anyway, so, for those who don't know, who's right, go programs.
Alright, so you guys probably don't know all of this.
Anyway, go has one of the cool features as a built-in memory profile,
and it's just a statistical memory profile, or the basically just has a little counter,
and every time it crosses a 512k, it's configurable.
But every time that happens, it takes a sample, and then it takes that sample,
it's just a call stack, and it sticks that to a hash table, and then that allows it to,
you know, track your memory usage with very low overhead.
So, you know, 512k is a pretty big number, so most allocations, zero overhead.
And then, if a, you know, allocation gets tracked by the profile, a bit gets flipped,
and so when that thing gets free, by the garbage collector,
some stats are updated, so it knows when those things are free,
and in addition to seeing allocation volume, you can also see in use statistics,
which is usually what we care about, right?
We don't care about all the garbage, all the things that were allocated free.
We want to know what the kind of outstanding and use allocations are.
But goes GC, so do we want to count garbage?
Like garbage, you know, as things that are going to go away,
but if you're dealing with loans, you do want to count garbage, right?
So until the garbage collector gets to the sweet point of its cycle,
that, you know, all your garbage is actually consuming memory.
So, the way this works in the profile is that, as it's profiling, and according to Alex,
and free, they get stored in this kind of like scratch,
temporary, you know, future bucket, and then at the end of the sweep,
all that, you know, future free, temp stuff gets swept into the active bucket,
and then that's what's reported to you from the go pre-process end points.
So the active bucket, the active, you know, profile is never going to have
any garbage in it, it's like the end of the sweep.
So what we do is we count all those things, so we roll it, not only just the active thing,
the P-profit can read those future buckets, and so if, you know,
the allocation to cause your problems was recently, the recent allocation,
which can happen, we'll see those, even though they may not have show up in a memory profile.
Does that make sense?
So we're in the garbage collection cycles, but anyway, so the first step
of doing improve is that, you know, we have a wash and program at scans,
all the pids in your system, it finds the go programs,
and then it tries to find this point or call the runtime and bucket.
So this is the address of the actual memory profile buckets.
So if it can find them, we'll register them, so we'll have a BBS map that says,
these are the go programs, we're care about, these are the ones we know,
we're the profiling data lives, so these are the ones we're going to watch for.
This stuff is all implemented in the parker agent, and the Oom Prof code itself
is a library that we just use.
So you can use the Oom Prof library separate from the parker agent,
but I'm not really going to talk about that because it's a little complicated,
but if you're interested in that, you can see me after class.
Anyway, so it works with a couple trace points,
what you do is a marked victim trace point that allows you to see
when the Oom killer is picked a victim,
and so you can get the pid from that data.
So we can do that to say, okay, here's the pid that's dying,
is this in our go prox maps of the go process as we care about,
and if it is, we know, oh, we care about this, so we can go read it,
go read it's memory and we're done, right?
Well, not really, it's a little more complicated,
because the Oom killer runs in the context of some programs
are allocating memory that may not be your program, right?
So the marked victim, even here, program,
can't just start reading data from your go program
because the context may be systemd or some other program.
So how do we solve that?
So the way we solve that after much spelunking
was to realize that the way that Oom works is it uses kill,
so it sends a kill signal to the program that's dying,
so we can hook on to the signal signal deliver trace point
to see when our program receives this,
and luckily for us, the signal deliver EVPF trace point
occurs in the context of your program.
So our EVPF program that's hooked up to the signal deliver trace point
can restart reading the memory from your process,
and it's still there and it can read it and it just kind of works,
which is nice.
So, you know, just to brief point about yes,
there is some overhead attaching a trace point,
just signal deliver, but, you know,
typically signal deliver is not like a super high throughput thing,
and typically it's not on the critical path
for key workloads, but who knows,
you know, which may vary, so.
When it's out, measure.
Let's see, let's set it.
All right.
So just to overview of, you know, how this all works,
Parker agent, Parker agents are, you know,
profiling agent that, that poor signals helps develop,
and then it, installs the trace points and sets up the maps.
And then it's gives the process,
it doesn't register the ones that can find the bucket address,
the one who happens, mark victim, signal deliver trace points,
exit, execute, sorry,
and then what we do in the EVPF single delivery trace point
is that we rip through the buckets of the GoPro file,
copy of those into an EVPF map,
and then we send a signal via a PVDF
purpose and output to the user agent.
And then, Parker agent gets that signal and says,
oh, and program died as an event.
Now, I can read that map, read those buckets,
which have been copied from your GoPro,
and your GoPro and going at this point is dead and gone,
but that EVPF map is still there.
And we can read all the entries from that,
and then create a people off profile,
and it can be uploaded to polar signals,
and then you see your profile,
and you can figure out what happened and save the day, right?
So, does it work?
Sometimes, you know?
Actually, it works pretty well, but like,
there are things that can go wrong, right?
So, the hash table that go uses distort,
things has a massive prime number of entries,
and then each entry has an infinite number of
possible entries because it's just used as linked list
chaining.
So, in theory, we could have an infinite number of records to read.
And obviously, that kind of doesn't fly with an EVPF program,
right? It's pretty strict.
And it's, you know, how much execution you can do.
So, our current implementation
can, in one EVPF program,
read 360,362 buckets, which seems like a small number,
but we haven't done a lot to optimize that
to probably get a little better, but anyway,
that's not enough for a kind of, you know,
resumong sized go program.
So, that's no good.
But then luckily, we can tap into tail calls
and get that up to 100,000 buckets.
And that turns out to, you know,
for all the things we've tried to throw at it,
there's plenty of buckets.
So, the map is a little hard, right?
So, what is a bucket of bucket is a unique allocation stack.
And so, if you look at a program,
like you can't just look at it and go,
how many unique allocation paths are in that program?
You know, it could be a small number,
it could be a huge number, you know,
basically, probably roughly correlates
with the size of the program.
So, if you have like a huge go program,
maybe that's not enough.
But in practice that we've seen,
that's plenty.
And then also, because that maps pre-allocated,
how many buckets you support
factors into the size of the map.
And the map roughly wakes that works out to
60,000 buckets takes 40 megabytes.
So, that's not a ton of memory,
but this is configurable, right?
So, you could jack it up,
and you can get up to, like I said,
100,000 before we start running into
our EVPF limits.
So, if you have more,
what happens?
Well, we just stop, right?
I mean, we only, you know,
the loop in our program is limited
to a certain number of iterations.
If we get to the end,
and we haven't reached the end of all
of the memory allocation,
buffets in the hash table,
we'll just record,
but now we didn't get to the end.
This is incomplete.
So, now that may or may not be useful,
like, you know, depends
to be hit the interesting allocations,
to the interesting allocations
come later in the hash map,
and we didn't get to them,
but I really still have something
so you can look at it.
So, it's a bit of a crapshoot in that situation,
but at least you know,
like, let me tell you,
if it's a complete reader not.
So, why do these limits exist?
Just a little more detail,
like,
the,
my slides suck by the way,
so I apologize for that.
Like, I should have like,
pictures and stuff.
Anyway,
the way these records are laid out,
you have like a header,
you have the stack,
and then you have like,
their actual mem stuff that tells you,
how many, you know,
Alex and Freeze and whatnot.
So, the header's a fixed size,
and then it's going to number at the end,
which is the number of stacks,
and the stack part is anywhere from zero to a thousand,
stack entries,
which is just the call stack up to that,
and then after that is the mem.
So, our ADPF program has to do three reads,
to figure this out right next to read the header,
read the stack,
and then read the mem.
So, that's,
it's pretty simple,
you know,
if you want to go look at the code,
I encourage you to do so,
but that,
you know,
doing those three reads,
the instructions to pull out the data
we're interested in,
copy them to the EDPF map,
is why,
you know,
we only get three thousand or so buckets per program.
The,
you know,
the other thing I was going to point out is that
this can be up to a thousand,
but our representation fixes it at 64,
so it's a fixed size allocation,
so that's why the three reads instead of two,
because we don't support the full thousand stack frames,
that go to us.
It's typically 64's enough.
This is all configurable,
like,
in your use case is different.
You can go in there and change it.
So,
walking go wrong,
you know,
BMPF reads can fail,
you know,
it's effective life,
you know,
if the, you know,
the programs under memory pressure,
which,
if it's boom killers happening,
you know,
machine is under memory pressure,
it could be possible that parts of the,
the bucket array,
massive array on any of those allocations,
what pages they live on,
could get put out to swap,
and if we try to read them,
we just get an error back.
So, that happens.
So, again,
that's why we have that incomplete flag.
If we can,
some reason,
we don't get to the end of all of your buckets,
we'll let you know that's an incomplete read.
And then,
something that can go wrong
is,
sometimes,
the Oomkill gets impatient,
and,
while it's trying to kill your program,
it will come along and kill another program,
in that case,
it can also be the park agents,
it gets killed,
and then,
you know,
what's the point of Oombrough at that point,
because that data's not going anywhere.
But,
it was surprisingly well in practice,
you know,
we have a bunch of different tests in our CI,
that tests different,
you know,
programs that crash in different ways,
and it's pretty reliable for the test.
That we have so far,
but,
you know,
probably lots of fillings,
I don't know where of,
but,
those are the ones we've seen.
So future directions for Oombrough.
We,
you know,
kind of,
I think the inspiration for it goes,
memory profile,
probably came from J Malik,
and it's profile,
so,
including J Malik's support would be pretty easy,
and be cool,
so it wouldn't be just go programs,
or TC Malik,
or Mew Malik,
which is commonly used in rust programs,
would be good to support.
In theory,
we could probably support J lived C,
but that would be a little bit more of a project,
because it doesn't have a profile built in.
And then right now,
we have that limitation where,
if we can't find the runtime,
and bucket symbol in your program,
then we're dead,
right,
we can't program a lot of go-binder,
as well,
you know,
strip out those symbols,
so,
not a lot,
I would say some.
But there are ways of finding that stuff,
if it's a strip binary,
but we just haven't gone to those links yet,
but,
you know,
in theory,
you can find a function that,
references that thing,
and then disassemble the function,
and find the address that way.
And the other thing we're looking at,
is,
you know,
like I said before,
a lot of profiles for go programs
and by scraping the people off end point,
it's a mineral.
But what if we could do push profile,
and what if your program could say,
oh,
something interesting happened,
or I'm in some interesting phase of my program.
We're doing some kind of memory,
intensive optimization thing.
Let's push,
you know,
so we have the ability,
because we can just read them at any time.
You can push those profiles.
You know,
having to get pulled.
So,
I encourage you to check it out.
Like I said,
the UMPROF project is,
under the parka umbrella,
that the pole signal is created.
That's the library,
and then it's included,
and built into the parka agent.
And then the parka website has a bunch of,
a link to start on it.
Is anyone ever used parka?
Anybody?
So, check it out.
You get parka agent.
There's a lot of useful things,
in addition to just doing Prophiling.
So,
and then I guess my last slide wasn't included.
That's all right.
It was just my name.
So,
well,
thank you.
Thank you.
So, we have lots of time for questions,
so I hope you have questions.
Yeah, thanks very much.
I'll definitely will try it out.
Especially I do a lot of direct,
we do a lot of direct profiling,
just like you said,
and I directly,
with running,
executable,
go.
I was just wondering,
does this offer any additional presentation?
Or is it literally I'm just going to get people
by the end?
Is there something extra I get,
additional to the people of itself?
So,
it's just the P Proph,
and then the information you get is
did a reader occur,
so,
along the way,
if we've got failed to read from the map,
we'll tell you what that happened,
and then is it complete?
It can be incomplete if we,
if there's too many documents for us to read,
so we didn't get to the end.
So,
for most cases,
you know,
complete will be true,
and read there will be false,
and that's a full profile,
and you should be able to use that,
to kind of figure out what happened.
Hi,
yeah,
I have a question myself,
over here.
Yeah.
I'm really useful to,
I'm super excited to try it out.
I'm sure it will help
to debug lots of these own issues.
While you're beginning the talk,
I had a thought,
and I'm wondering what your opinion is on it.
So, as the memory pressure builds up,
the kernel obviously tries to do a bunch of things,
like try to put pages as well,
and basically get them out of memory,
so it can free up space with the allocation,
and it's only after a lot of trying
when it can,
that it owns the process.
And I'm wondering if you can,
maybe use EBBF in this process
to kind of start to build a very compact model,
of which pages,
the kernel is having problems with,
and if you can communicate that to the user space,
so maybe the user space application can get,
ahead of the own,
like it has more context about
a which memory should be available,
and not be available.
Like early warning system.
Yeah, like an early warning system,
but like with a model of the memory that the kernel has,
so it knows that,
okay, the kernels having troubles with this section,
I was an honest memory,
and maybe I have some buckets over here
that I really don't need,
and I can shunt them out.
Yeah, I mean that would be nice, right?
If there was like a signal or something you could tap into,
I'm not aware of anything like that.
And I do know that like the go garbage collector
will, you know,
if it can't allocate,
will, you know, before,
you know, dying,
it will try to run the garbage collector
and try to free up space.
But I don't think there's any kind of like proactive signals
that Linux sends,
but that would probably the way to do it, right?
Like if basically garbage collector could know
that it's running out of memory
that maybe it can try to adjust its,
you know, policy,
so that maybe it doesn't overal get too much.
But yeah, it's good idea.
I mean, I also know that there's work
ongoing in the kernel
with improving some of this stuff
and making, you know,
kind of more information available,
like you're talking about.
So, I think the issue is also,
preemptively numbing one hour
and it's going to happen
because you could get a pressure.
But it's not necessarily,
but I'm just going to necessity.
That's the challenge I think.
Sorry?
Well, he said,
oh,
so what do you say?
The preemptively.
The preempt is hard.
It's hard to know preemptively
when you're going to get the RM, right?
So even if the pressure's high,
even if I set the signals out,
it could be 10,000 or four signals,
so like, you know,
which one do I get?
The current response to the RM I'm looking for.
So, I don't think that.
I think that.
That's okay.
Preemptively is hard.
Yeah, thanks again for your presentation.
I'm wondering a bit,
and that we kind of have the O,
M, not student to two use cases.
It's the machine runs out of memory
and you have a secret limited process,
which is like 99% of the O, M's
that process is usually C
because it's not the machine that runs out of memory.
It's the limit that's been given.
Yeah.
So the secret.
And as I was wondering,
there's no, there's work being done
by Roman Kuchen from Google belief
to add O, M management to the BFF capabilities.
So you, you don't get a warning,
but you get plenty more abilities to manage that.
And then the machine is not out of memory.
So you have plenty of time and plenty of memory
to keep done but do also the stuff,
which is more of a thing
than managing the out of memory of the machine,
because then that's a panic.
Yeah.
I mean, it will proff works whether
it's a C group or a machine.
So, you know, parker agent usually runs in its own,
you know, par and Kubernetes and when it
ooms, that can happen,
because people will typically tell parker agent
like don't use a lot of memory
because it's a, you know,
continuous production profile.
So it's not supposed to,
but yeah, it works in both cases.
And, you know, these things are constantly
going to get refined and it's good thing, right?
Like, I think, you know, for certain applications,
especially databases.
Like, you want to be able to use as much memory
as you possibly can and run tight
and, you know, cash as many things as you want
and have flexible manner, you know, flexible ways
to respond to a lot of memory on payment
and be clean up some stuff.
But this is kind of the something, like,
and develop before or most,
like, when things got too bad,
the process has had to die,
now you know why?
One question over here.
Yeah.
So you find that them correctly
in the end of the day.
It's a correlation of information
from user space in the kernel space
via regular BPF map.
So is there feasible or reasonable
to try maybe to use task local storage
for these purposes to store those buckets?
Use which storage?
Just local storage.
So like a different type of BPF maps
that are local to, to tasks.
Ah, maybe.
So what would be the range there?
Essentially it would be a advantage
would be that you will get
inside your BPF program,
right there, local to your threat
that's going to be a performance advantage
and probably a couple of other perks.
It is a different type of BPF maps for the purposes.
Interesting idea.
Yeah.
Because I'm really, I'm asking,
because it seems to be popular nowadays
for profiling of different type of thing,
mostly for CPU profiling, like,
if you get item of Python,
run time or something like that,
would still seem a similar idea,
run time profiling,
but in your case it's going to be memory.
Okay.
And what's curious if you're like,
we're investigating at the direction?
Not at the moment.
Yeah.
Excuse me, could you have this?
Yeah.
This project was used in the context of a project
that we had to do an unwinder for Luigi.
And in the course of doing that project,
we had some code that was written by your
colleague.
It wasn't so great that umed,
so that was kind of the,
that was the thing it was solving.
And it worked.
So, I'm sure there's a million ways
that could be improved,
but to open source and it's out there.
So, going through that.
So, I'm curious if you have plans to work
is going around time,
and you mentioned the legit,
to have some support on their site,
to make it more reliable.
Thank you.
So, making things more reliable
from a running out of memory perspective.
Well, for, as I go this right now,
it's super inconvenient,
because this Hashmap, for instance,
is scattered all over the memory space.
So, for instance, just, you know,
it's done by the,
if they did something that it wasn't,
that much scattered,
but state,
I don't know,
three megabytes of continuous memory space,
and this could be some special making
that lived after a process has been killed.
But I don't want to suggest this.
I'm just thinking, general,
that maybe programming languages
at some support,
the whole infrastructure could work better.
And since you're like,
doing the peer-nearing work in this area,
maybe you started doing something like that already.
I haven't,
but I mean,
I know there are solutions,
especially in database software that do stuff like this,
that kind of monitor,
you know,
how much memory is free,
and we'll kind of,
react early
to low memory situations
to avoid this kind of stuff.
But things happen,
and it's good to have a escape,
you know,
patch for when it dies.
All right.
Thanks for coming out.
Thank you, Tommy.
Thank you.