I'll summarize what I've said, you're said.
Our goal is to have a always-on verbus tracing system in such a way that your performance
won't hurt too much, right?
Zero is impossible, contrary?
I think.
Yeah.
But that's the goal, because sometimes you get those issues that happen somewhere sometimes
in a narrow and impossible to deal with, right?
So that's the motivation.
So it's technically, it's somewhat similar to F-trace, right?
I hope you're familiar, it's tracing, but it's used in, usually, in development or in
the debug, when you control the system, and it's a single term, right?
So here we're talking about by repression, per your module, and where you control the, what is,
you know, what, and how, etc.
So the problem, yeah, as I just mentioned, fully formatting the loading is of a slow, even if
you don't dump it into demensage, it will still hit over from it, right?
So this is going to be a technical detail in what we did, and how we did it, yeah.
So the problem, the problem is memory, memory, memory, that's the problem, right?
So if you want to have good performance, you have to be hitting a close to your memory,
and doing, doing as little as possible.
So that means that the idea of increasing, meant that while, well, we may be reaching out
of your, except the working space, right, because you're loading somewhere else, right?
You have your memory that you're working memory, and you have your loading memory, going
outside of your working memory, means you'll get your cache misses, and you will hurt
performance.
That's the main problem of your, of any logging or any files.
So try to minimize the size of the trace, right?
And the number of times you're accessing a remote memory, remote, and it's outside of
a one two, right?
So it's not really remote, but still something we try to minimize your performance
hit.
The goal is to make that as small as possible, right?
So why I'm saying it?
Because you will see some civilizations, and that's the idea of trying to make everything
as compact as possible.
That's why, also, if trace is a bit flat in my mind, for at least for this case, right?
So why binary trace is, by interesting is kind of the fastest way to do it, and many
of the things that you do, you can do it compile time, right?
So we keep an in memory only the parameters that you save it, right?
So you have some kernel letters, or some ID, some token.
You save only, you don't need to save the whole string, because the string is somewhere
else, all you save, so we save it just once, at compile time, right?
So, we have, for each logging instance, we have this data structure, right?
So we have, at the as delta, what at the as delta means that we don't keep the whole use
64, for your time frame, we keep only a data from a baseline, right?
We keep the baseline, actually, in the next slide, so, but we don't keep the whole time
frame, only the data from a baseline where you allocate it a page buffer, instead of, for
a shot in something that I'm going to talk about, so each entry has the data, so it's
more compact, source ID is where your line was, right?
So you save somewhere, each time you log in interest, right?
On the first time you hit this line, you save the location, right?
The file, the file name, the the line number, and gets an ID.
So next time when you are logging, you only save the ID, right?
And so when you are disserializing this log, you have the exact time, you don't need the
string, you just have a U6 and a, why does it say, hey, fine, okay?
And so you have this, and you have a user context, for example, in, in safe, we have FSI
globally, but we also can save some client specific, something that specific to you, in
your logging, so you can get it fast, right?
And the left, so it's basically a TLV structure, and that's kind of the idea, using
as compact as possible, a, how are we, you know, how we're doing, especially, so you have
a serialization, with the LV, it compiles time, as much as you can do it, compiles time,
you save it, and that's kind of the, the, the magic, right?
And then you, you see always on demand, right?
So, what's, what's the magic, the magic is that we, I'm not using actual circle buffers,
but it's a circle memory allocation system, right?
I should have invested in a slide of this kind of thing, but basically, we have a magazine
allocator, some, the memcash work, so magazine allocator, you get a magazine buffers, right?
An allocator per core, right, you can have two of those, so there's a paper on it, not, unfortunately
not mine, and it's been long in the can offer for, but we're using this reason to make
it, so the allocation is very fast, for each frug, right?
But we are allocating very, very small segments, right?
As I mentioned, we have this on your parameters, and only the, the TLV header, right?
So each, in our use case, what I've saw, with this, in this way, we could save two
logs per cache line, right?
And it's improvement on, whatever, whatever it was before, basically means that when you
have one log, it's, the cache line is only in your L1, the next log will be essentially
free, right?
So how do you allocate fast?
So first, you have the magazines, allocator, so you have per core allocations, which
you go to the global context once, once in a blue moon, again, it's a talk about magazines,
but you know, if you're not familiar with pattern, my advice, you use it, no, it
because it's very powerful.
And the next thing we do is take each one of those buffers, and actually use it for a,
what, I'm forgetting the name, we slice it up, right?
It's again borrowed from networking, right?
Neplications, what you have usually, you have 30 k, a buffer allocated, and then each time
you have a network pocket, incoming just slice it up, so your allocation is just adding
whatever the size of the buffer is, and that's the same mechanism we're using here.
And so for my experience having this pair of magazines with, a, page frauds, with a fragment
into this page, it's very, very fast, right?
It's, in this, you, you want to see the performance, I use it with 100 gigabit nix, pair
pocket, a memory operations, so this is not what core superfones, what core superfones
is the actual axis, right?
That's the first slide I saw, I, I, I, I showed, right, so memory apps, and so we have
this magazine allocation on, on the one side, we have your page fraud allocation, when
you're using it, and then you have a, an allocator that, also an magazine allocator that
puts it in and exists in logs cache basically, right, and all magazines go into a global
list eventually, and then recycled, right?
So you have full control, and this cyclical, a, allocation scheme provides you two things,
hey, you control them out of memory, you're willing to provide for your login, and then
make sure that each log keeps its life as long as possible, right?
So it's only when it goes through the whole, whole cycle, will your logs be recycled,
right, in globally, right?
So performance and, and, and life, so additionally we want to have process context,
but, you know, not everyone happens in process, or that's something happened, something's
happened in, I agree, or not, because that's inappropriate, right?
So for this specific change, you're trying, and that's the one big change I need to
kind of get in, we are adding a point in the structure, right?
So when you have a process that is logging, like, we're talking about file system, right?
So when it does the right read some operation, you want to see, you want to be able, when
you get in the logs, back out, you want to see which process did what, right?
You want to, you want to, be able to take your A Python script, and organize the logs in
ever, which way you like, because you won't get them in a, a, a, a, a, a, a property
serial assign, you'll get them each pagefrog, which will be properly sealed as, you know,
by time, but they will be out of order, right?
Anyone will organize it by your process, or whatever context that you have, right?
And second thing you have the dynamic context, you have packets coming and we're seeing,
so the process context is relevant is wrong, right?
So that's why we, we separate between the two, and we have pair process logging, and pair
CPU logging, and that separate, they all belong to your module context, right, and you'll
get them, but you don't, but you will see something that happened when you actually did
something, when you made the C school, and when something happened with your module, but
you don't really know kind of, you don't know who, who paid the price, because, you know,
whatever was running on the CPU, when they, you happen, so we tell, there was a guy,
if you're not, it, something came from the wire, something came in your, I think, this
is called the C school, and, you know, make sure you have the proper logging.
A memory layout, let's try to make it, let's get the slide, so this is kind of the
explanation of what I will, if you session with our spleen, so we have the empty mug,
it goes to global least, that's kind of the blue, and it goes into a full mug, at least.
So we allocate a page for context, as I said, so we have, we have a smelly, magazines,
we allocate an entry, but it belongs to a page for our context, which has at least had,
because as I mentioned, when we allocate one, it goes into this global least, and a page
had, it remains alive and usable throughout the life, no matter where in the life cycle,
each page for a is, and that's so that when we dump your information, when you want
to actually want to see to be serial as a logging, you can get all the logs, right?
So I have the list log, and we have the distribution, like the PID or the CPU, wherever,
and the base G-fix, so when this becomes active, you record the base G-fix, and so
the next time on each log, comments don't need the delta, right?
That's how we make it small.
Yeah, so that's kind of the description, so we have the buffer, the metadata, and then you
have the logs, and when we deploy this page drop, instead of trying to do a cyclical
location, we just move it aside into an existing logs, cache, and get a new one, again,
because we don't, we want to keep the logs alive as long as possible, right?
Global content, that's kind of what they start, so we have the location, but this is
the kind of the live log batch, and I think it's a typo, and so you have the source
map and the climb up, so the climb up, as I said, is the identifiers, they're specific
to your module, whatever you want it to, to save, and the source map is the location,
so you have the lines, and the, yeah, so it was a big dive, the original one talk was kind
of longer, I hope it wasn't too dense, so thank you for questions.
How do you read the trace, then once you put it up, how do you read the trace, then
is there integration with any tracing GUI, or do you use some custom scripting to then
reason about it?
So, what you get, you have the current solution, is we have a debug a fast file for your
module, and you just cut and read it, right?
So I'm not sure it will be the main, so I sent back in October, the batch set, it's about
long-dish, and Steven wrote that who wrote F-trace, long-dish, why would I use F-trace
so the current implementation is debug a fast, and we'll see how it goes, they might
go into it.
Are you tracing only kernel space, only user space, both?
It's only kernel space.
Okay, thanks.
It's not in the big dev solution, it's just inside your kind of module and compile.
You started your talk with, with comments about memory latency, yeah, and does this really
play such a big role, because as a tracing provider or creator, you're just writing,
that means you never have to wait on memory, you just stored out into write buffers and
it eventually, the system will flush them out, what are you saying stalls?
It's not about the stalls, it's about the memory footprint, so whenever you're going
outside of your memory, you're built in your L1, your L2, etc, so whenever you have
a working context at the CPU, it's just a long-dish thing, I need to remember, the Intel
has passed the, what do we call the non-curve or are you using them?
Yeah, that would buy fast all these times.
He actually tried, for my memory, the hustle is not very, so we used to, made some experiments
work like a decade ago, in different contexts, not this one, it's actually an interesting
idea, I would need to do it, so my memory of it was that it's not very good, it's not
very useful, but maybe out of date and the specific contexts may have changed enough to
make it very open, so it's actually good at it.
Is this specific to kernel modules, you need to do something in your kernel module, is this
specific to kernel modules, do you need to modify a kernel module, or can we use it in the core
kernel?
It's a subsystem that you can, it's a library, you can use and just change your logging
like, the P trays to, you know, be locked trays and you get it, all right, and if you
am, okay folks, any more questions, you want to see the, you know, double the badges,
my email is right there, thank you for listening, thank you.
Thank you, thank you, thank you, thank you, thank you, thank you, thank you, thank you, thank you