So everybody knows Fred, nobody's fine.
So if you don't know me, ask Fred, he'll tell you both.
So while I work in front of my school team, from 2011 officially, okay,
so before I just, it was for fun.
And I was working in Santa Cruz System benchmark center.
So it was pretty interesting working with any kind of benchmark,
it's a long investigation on performance in my school,
but tried to scale it and it was so easy, of course.
But at some point, so my school acquired my son acquired my school, sorry.
And so in the previous talk, there is one my school version, which is misset,
and I don't know if you recall this.
There was person my school 5.4.
And in fact, after 2008, some work at closely with my school team to improve performance.
And I know from my school 5.4 for in 2009.
For the first time, we published a benchmark result.
So normally, we would not allow it to change into the record,
but they were strong balls, they changed it.
And we showed to Oracle, then we have better performance than Oracle database.
Unfortunately, so it was presented during my school conference,
and Oracle made a note and they acquired him son.
So, and then all teams worked together, and then it was, well, now you see what we have.
Well, there are many other also details, I will skip it,
but if you want to know, we have a long history here.
So today's talk will be about practical stuff.
So this practical stuff came, if you want to hate somebody, you need to hate Amazon,
because it's all about business, so Oracle should do cloud as well,
because otherwise, you can see all the money going to Amazon,
and my school team delivering product, and here's nothing just use everybody,
and thank you, my school, for Amazon.
So we started to do also real stuff, and, well, all this problem,
well, not problem of my school team.
Who cares about memory location, and now when I discuss it,
live example, booking guys.
So when they discuss it, then do we have other out of memory killer?
Oh, yes, do you know why? Oh, no.
How you manage this?
We just have many replicas.
So it died, it restarted, in other replicas.
People did not care.
In our case, you know, when we believe or something,
it's our product, so we need to understand what's going on.
In many cases, the problem was not ours.
For example, you cannot imagine, but anti wires on your system,
can just eat 16 gigs, we'll kill every single one.
And disappear.
And just see it on your medical service, not more here.
So memory profiling was one of the main problems,
and then CPU profiling came into our.
You, we'll look back in history.
So today was 30 years from my school,
we can speak a lot about history.
You see, the first maloclibrary was just not empty safe.
Not much is how that saves.
One of the, if we can empty safe,
it doesn't mean it will scale.
So just global lock.
So every time spread allocations,
we'll write in for the same matrix, and it's not scaling.
So the first, the first one, which I recall,
was whole library, which was scaling,
and how people discover this, you know,
well, you develop code.
Most of code was implemented in the past and C,
and when you manage your maloclocation,
okay, you can't live with this.
Each thread will allocate an unknown,
big enough memory in your fine,
and then the pay attention to not get too much contents on this.
Once people started to move to C++,
every time you create new variable,
you have malocl,
it's a malocl, a malocl,
and every single one you have a fine,
and you don't have control.
So all this new library's came in game.
G malocl was most popular on Linux.
You have other for another system.
I today prefer to see malocl,
and I will explain new why.
So when we deployed the first system,
and started analyzing why,
it cannot survive during one week.
It was just because of this.
So you have a loop on simple test work load,
so we know there is no memory on my scale,
but you have memory fragmentation coming from a bit similar.
So your memory just growing growing growing,
and it just will be finished in swap,
and then you die.
In worst case, there is no more memory,
and every single frozen,
you cannot boot the system,
if you reset, and you don't know what to do.
So in fact, this was first to discover,
even it works well today.
So it's more designed at now with multi-shrad,
so there is no performance computation,
the problem is memory fragmentation.
The parent to see, to see malocl, you see.
So it's, I tested it explicitly,
two different versions of,
to see malocl, just understand.
With all versions, everything is fine,
like with new one, it's fine,
but ellipse malocl is like this.
Before performance reason,
and I'm mainly working with performance,
you know, so,
jim malocl is more efficient for performance,
but for memory education,
so it will free,
you don't have this memory fragmentation,
like with ellipse malocl,
it will free or memory, okay,
but you may have the spikes,
because for performance,
you would not pay attention,
let's allocate memory as much as we want,
and you have the spike,
and you just kill your instance.
So if you do test,
and don't care,
fine,
in production,
it's not possible.
So the good point is,
and in my scale today,
so we have the performance schema,
which was pushed at the end of my scale,
five,
it was horrible,
with horrible overhead on the past,
and then,
big work,
big optimization,
work was done around,
and we introduced
memory allocation,
so we have memory instrumentation,
which will give you exactly what happens inside.
It's working fine,
so overhead was very big,
on the beginning,
but then not,
but again,
you cannot go until every malocl, right?
So it can be too costly,
most of the goal today is a C++ in my scale,
so you cannot trace any allocation.
So today,
it's really great to see,
and we have memory,
it can be higher probability to have it in plugin,
on something like this,
but still it may happen.
So when we started to deploy first cloud instance,
it was surprised after surprise,
because you may have some specific plugin,
which will never use it,
and will be used only when given customer,
and this given customer has a problem.
So the good point was,
and TSMoc has memory profiler itself,
and you can use it,
and you can profile our memory to very digit details.
How it works?
All you need is just to set
a heap profile in the Romance variable,
so you start my scale,
and I'll set it up with LDP load for TSMoc,
and then as soon you start,
it starts in bombarding with dumps.
So the dumps are generated automatically,
so we have few options I will show you,
and you can manage which frequency one to have,
and you can see it directly with P-prof,
so it's dedicated tool made by Google.
So it's always coming for Google DevTools,
and you can get it in text,
and you can get it in PDF,
and PDF it's amazing because you really can follow
your memory education from every function,
and in fact you can understand what you're doing with your code,
because people sometimes have different impressions,
I should not do it,
but they do,
because another function will come in this one,
and another here,
and then at the end you can go to another place.
So the only problem is,
then if you did not put it on the start,
you cannot get anything.
So you need to restart my scale,
and you need to have a pretty small case,
because you know what happens in production.
Oh, my memory is growing, why?
Let's restart the most,
if you restart that problem is gone,
you cannot reproduce.
So if only if you have reproducible case,
you can debug it so well,
that's the totalization for this.
And again,
the main point,
and well,
and jump a little bit ahead,
of a hot overhead is very huge.
So you cannot run production like this.
So in fact,
you can start it,
and call API just to start
and stop profiling on demand.
So it's much more simple,
but in theory,
so in fact,
just call in API,
so we've got many discussions
to be edited in my scale code,
or probably not,
or whatever,
but on pile with TCMLog or not.
And so on.
And well,
this discussion was going on
and back and so,
so in fact,
every time we need it to use
TCMLog profile,
they all let's do it.
And then when it's fine,
so people just busy with something else.
And just discuss it,
oh,
it was a few months before for them,
discuss it with Fred,
you know,
I don't know why people,
anytime we have a new problem,
we will be in,
in big trouble,
and we still don't have the solution.
They told me we need to use component,
and Fred has,
I don't know how many slides about
component called developer.
Well,
if you do it all the time,
it's easy.
When you look at your time,
if what?
You have inflation,
rich Chinese,
I don't know.
So,
but it's market easy.
So Fred told me,
you really need it,
you do you have some specification,
I already have a page with the PI,
I don't explain any of them.
In the evening,
you can try it,
I can pilot,
I cannot deliver.
So,
in fact,
you don't need to compile with this envelope,
so your component can be loaded,
install it at any time,
but you won't.
So,
all you need to just to deploy
a GPR2,
or a magical host,
and this variable,
you need to set up.
So, first,
a preload,
a similar library from where you want,
and then you have two variables,
allocated interval.
So, by default,
it's one gig,
I put it to zero,
and use,
and use interval also.
It's one hundred next,
I also put it to zero.
Why?
In fact,
allocation.
So,
you may,
in your code,
you may allocate,
and three memory,
all the time, right?
I trace it,
my school code with dethrase.
You cannot imagine how many millions per second
we do in the lock,
and this is keeping in mind,
and we all have a memory library,
which will all cache some data.
So,
it's not every time,
when we need 10 bytes of memory,
we will call them a lock.
It's already reuse it.
And any,
in this case,
it's many,
many millions per second,
okay?
So,
one gig,
you allocate and three,
it will be probably every two seconds.
So, it just won't bother with these dumps,
which for you probably not make no interest,
because there is no leak.
And the same,
if you increase in memory,
one hundred,
one hundred megs,
so you will have a new dump.
Well,
it may increase decrease,
and so on,
well,
I don't care.
I want to see,
if there's already some big increase,
then I'm interested.
The good point here,
is,
you cannot make this stuff.
You can monitor from other tools,
system tools,
for example,
if your memory is growing in a nice scale,
you just invoke this,
your component may be already installed,
you just invoke the dump,
so,
you get your information.
So,
Fred also add another one from Gmail look.
If you want,
well,
you see,
I prefer to see my look,
because on Gmail look,
so if you have this dump,
so memory location,
so go.
It's interesting for,
well,
if you have free time to check,
but we don't have too much now.
So,
just a few lines,
how it works.
So,
if you have links,
Fred already wrote,
detailed documentation about every steps,
I just put practical steps here.
So, install component,
as you want,
so install,
install,
or install it in.
So, we have,
several components,
so you have generate component profile,
which managing all the staff,
global variables and so on,
and then you have dedicated profile,
or memory.
So, from select,
from my scale component,
you can see,
everything related to your component,
if you have global variables,
and status variables,
related to profile.
So, when you do,
profile are clear enough,
it will clear all your dumps.
So, you have clear on state.
You can start,
and stop,
memory,
profile are,
call it down,
if you want.
So,
I discuss it with Fred from,
you know,
when you test it in practically,
so you see,
and it's better than you have already done,
on start,
and always,
a dump on stop,
in case you forgot about.
Because,
other had maybe,
maybe pretty high.
So,
in the start,
we added all the additional options.
You can start with parameter,
stop it in 20 seconds,
for example,
and give it dump.
Because,
probably when you will start,
you don't have handy anymore.
So, your might scale,
client will not work,
and you will have no way to stop it.
Anything may happen.
Right?
So, we are about security here.
And here's the test case.
So, we,
preload the similar cleverer,
set in heap allocation variables,
start in server,
so both this as script,
based on benchmark kit.
So, you will have a link,
then.
So, we just pre-form,
we use,
we use both our pool,
sides of sort of two gigs.
So, it will be pre-formed,
and then we play in with,
with data,
which are small,
it will be only 20 gigs.
But, it will allocate memory,
because we use order at range,
so we have,
group by inside,
order and so on.
And,
and,
at then,
we want to have a report.
So,
the test load is coming from sort of two,
whatever uses,
finishes by 4,000.
Just for several,
to see them,
to allocate a lot of memory,
and then it will go down.
Okay, and as you can see,
we start from,
just for 60 seconds,
second,
a profiler,
to be sure,
and we,
it will always stop it,
and we don't need to have interest.
What happens next?
So, the first rough is just to show,
you,
okay,
it's three loops of the test.
Connections going to 4,000.
Fine.
Now,
what happens to memory?
So, you say,
memory,
allocation,
and Linux is something.
So, it's, I don't know,
it's can be one day dedicated to talk,
and it's then,
it will not know the truth,
because as many experts you have on Linux
kernel, as many opinion we will have.
So,
you have virtual size,
which is,
virtual expectation,
if you have real size,
or a test.
So,
in virtual,
swap,
allocation is based,
it also on this.
So, you can see,
we allocated,
how much eight gigs
from the start of the test.
And then we don't free,
well,
free,
some part,
but we'll go up and down and so on.
So,
in real allocation,
we'll allocate them less,
and then it will vary it
about one,
dot five gigs,
some say like this.
So,
well,
because some memory will be used,
or they'll look at it
and never free,
one,
some will be free,
some use it by
some cash in and so on.
Now,
from performance schema,
we can see what is allocated.
So, you can see the green line,
this is performance schema allocation.
So, in fact, you need to keep in mind,
performance schema,
has some memory dedicated
for each connection.
So, we arrive at the four south of the case,
one gig,
250 megs,
will be always allocated till the restart.
The main point,
I want to mention here,
you can see them before
to go into 4,000 from 2,000.
It's going from 750 to 1,
so it's 500 megs here.
So, just one jump,
I can renext.
We will see how it's reported
from TCMA log.
Just to mention them from performance schema,
you can see in details,
which,
which part of the performance schema,
which part of whatever else,
allocated the memory and how much.
So, the performance schema
allocated and keep it,
and you know exactly where.
And here we have,
you see memory allocated in 3,
allocated in 3.
It was from SQL bar,
on the file sort,
and also on SQL,
it's a main root sort,
self-cash and a memory location in my SQL.
So, okay,
the performance schema tells this.
Now, let's see what they say in TCMA log,
profile.
So, you can see,
you can see,
it's 3 iteration,
from the first report,
it's exactly reporting 500 megs
for performance schema location.
Good,
we are all agree.
And then, if we look more in details,
so we will say that it also reported
for 5 sort,
the same and the same for memory.
So, exactly the same stuff,
which we have in performance schema,
and so, just the colorate,
and it's all coming to the same stuff.
So, the good point is,
very easy to do it from my SQL connects directly.
So, once you know it's in place,
you can install the uninstall as you want.
You start the term demand.
The dumps are really small,
so you don't use this place and so on.
So, if it's disabled,
there is no impact.
The main problem is,
when you enable,
it's mostly stopped the world.
So, that's fine.
So, it's,
so what happens inside probably can be improved,
I don't know,
but,
well, it's not to be running production.
It's when you want to have a snapshot of your memory,
where it's going for a short period that will be enough.
So,
I enjoy even more now,
performance schema,
because this one is always enabled by default,
and you have the same information with minimal cost.
You know,
the first person of performance schema,
memory instrumentation,
was having something like
such a person who never had a new enable.
And we were,
we were resitating,
and we were during my school conference,
and okay,
which will present this stuff,
but people complain,
as there was many users coming in the complainet,
can you reduce,
and there was a guy arrive at,
guys,
we solved the production problem,
even it will be hard,
one had,
a person who never had,
doesn't matter,
because it pointed at the problem of solving it.
So, okay,
then we'll continue to now,
it's really fine.
Here we have example of PDF,
you can get from,
a similar profile of,
so you see,
you can really see all the story,
if you're,
it's pretty lonely,
you can put it on slide,
if I put it on slide,
you will see nothing,
but you see,
all allocation going,
how many,
megabyte,
going to each function,
and you can follow exactly the chain of allocation.
And then,
when I discuss it to Fred about the component,
you know,
I said,
I discuss it with developers,
inside of my school team,
well,
we can have memory allocation,
and in my mind,
what's okay,
if one day is agreed to have something
which can enable,
a similar profile of the API,
then I will speak them next step about CPU,
because you also have CPU profile.
Fred,
oh really,
in the morning,
next day in the morning,
you have CPU profile.
Easy,
the component,
I will call him component month,
so if you need anything about component,
one point about components,
you know,
you always need to keep in mind
and it's part of,
once you started,
it's part of my school process.
So if you,
it's easy to write,
but if you do it,
you do a mistake inside,
just crush your server directly.
There is,
there is no way.
If you part of the code,
it's like you,
you added,
I don't know,
I can say it.
So with CPU,
profiler,
the biggest surprise,
zero overhead.
So if for memory,
or you have,
you say,
the problem with,
profiling of the code,
you see if you use P stack,
for example.
Any,
interested in about the stack,
may give you a problem.
So for example,
you cannot find production,
and check your stack all the time.
At some point,
you will kill your instance.
And it doesn't matter.
My scale,
or I'll go to the database, whatever,
whatever process you scan you with this,
at some point,
you will kill it.
The same,
with GDB will,
GDB probably,
but we stop it anyway.
So it's pretty dangerous,
and generally you have overhead.
Here it's not overhead at all.
But,
the only interesting stuff,
that something which won't
profile with Perf,
and something which you won't
profile with Google Perf,
CPU profiler,
they don't tell you the same story.
So who is right?
Who is not?
It's hard to say.
So my,
you know,
just use both,
because we,
we working with different hardware
vendor, for example,
when you run Perf on virtual machine,
it will just slow down already
everything you're doing.
But we have also vendors,
when it's impossible to run Perf.
You start Perf,
and this just stops you work.
So everything,
even in your cell,
it's,
it's not,
it's not.
So this is probably a solution,
at least on the sandwich zone.
And well,
some part of,
through is here anyway,
because I,
is,
I can believe,
and again,
lip system,
and receive,
we spend a lot of time,
because there is communication.
Why it's not reported in Perf,
I don't know.
So probably combining both,
you can get a full picture about everything.
And so,
you have reference,
but,
blog,
if you don't know,
blog from Perf,
so you need to,
to put it on the big lines.
So you have all the details about
the components inside.
It's very well written document.
It's a technical writer,
which,
developer,
component money,
whatever else.
And so,
well,
and I have,
tools,
blog,
and benchmark,
it's interesting.
So the last one is just to celebrate
30 hours with the cake and,
and firework.
Thank you.
We have time for more questions.
Yeah,
I got to leave.
What I'm talking about,
about the digital money,
and the memory profiling,
so it's a really consider,
already system profiling,
and I'm not saying that,
just by the point,
you are one of the balance profiling.
So if I change the digital
of the digital money,
where does the digital money
then we have that,
you know,
where it doesn't know that.
And right now,
you know,
the digital money is going to come from
the digital money.
You,
you speak about memory profiling, right?
I've had a lot of digital profiling.
But,
it doesn't matter what you do,
right?
It will show you what happens.
So,
we know that,
so it means only for,
so,
you need to select,
for example,
to sign up,
we have one library
that allows you to provide memory
once it provides you,
and one,
provide memory,
and it's easy.
And,
we do a master deal with that,
so we need all the rest of your
operations,
that doesn't care.
Okay,
so,
it's a point,
about,
that,
about,
doubt,
about,
about, about, about, about, about, about, about COPI started,
about invisible real economy working,
about,
about deposits
side. So that your CPU is faster or slower. So it depends on frequency inside. So in
GPRF tools, you don't have option to say at which frequency you want to get a snapshot
about on this term. In Perf, you can say my frequency is 99 or 9 million. So you will get
a bit higher overhead all over. So in this one is overhead is lower because probably
frequency is not a big. And that's why you don't have much overhead. But if you
use CPU a lot, so you have a spike in usage, you can be able to put down the stand
where is it. Because whatever frequency you use, it's something big which will be always
on top.
And then, can you use this number of files on that, that you've got all this should be
brought to the source on the network over the frequency.
No, it's whatever you allocate memory, you can use it. It's just about my skill
instance, right. So it doesn't matter if it's replica of source.
You cannot on the same time. Because you replace memory locator. So I was surprised
that with Lipsy Moloc, you don't have these Lipsy Molocs integrated directly in the
kernel. And if you want to test the different changes on the Lipsy Moloc, it's
silly story, it's time. So if one day it will be externalized, it will make life more
simple, but not for the. So Oracle Linux team addresses this. And they fight and
also they won't, they made some improvement. So then they can stable memory
location. So what, but you don't have profiler inside.
Yeah.
You see, because performance team has so low overhead, because of design. So you say initially,
it was like Lipsy and so on. And with scratch it's the head, with Markov, especially
pasted days in office. So it was, at the time it was some of his head in Paris. And we made
it step by step. So my fight was, and you know, if you're too, which you want to use, give
you bigger overheads on your application, it's useless. And the resistance fight inside
of my skill, okay, we want to, we will do correctness first and performance rest. But when
you enable the performance team, I have blockbuster about this and the past, you just
kill performance. And so you can't show anything because your problem is gone, because
your instrumentation is so heavy, then you remove the problem by yourself. It's not for.
So in performance team, I have to show you all the case. And it's, it's memory. It's
going so fast. And whatever you add some counter, which will have contention, you deal
with everybody. So memory instrumentation is local to each thread, okay. And then when you
run a query, it will just run through all's heads and bring you a summary. But it cannot
be one place or storage or whatever else. Everything will be in memory until your thread
is here. Once your connection is going down, the case is going down. But information about
each thread, we cannot do also, you know, in performance team, and every time you have a
new connection, we'll allocate memory in the free again, allocate and free again.
You know, do you say that it works in the grace of the rest hour from the new thread to
release performance team on the real world?
No, performance team, what's your reach for 1000 connections? It will keep memory for
1000 connections. So it's more safe to use proxy as hell, for example, which will keep
force of non-connections. But in reality, you will have to hunt it on my scale server.
See, because once this memory is allocated, it will be allocated. Well, otherwise, this memory
fragmentation story, because you know, when we deploy my scale, most people don't think
to install gmail, or ccmlog. So they will have memory for fragmentation all the time, because
it's huge amount of memory. And you will die much more faster.
Yeah, so you have, but you can see that some memory is freed. But, now, in performance
team, I guess, by design like this. So it was explicitly designed like this to not
free-wall. I test 4,000 or 32,000 connections just because I want to see if it will
pass or not. But when we look in production, so some not so many people have so many.
You can modify the impact of my schema by set mix. Yeah, sure. And you can do it very
though. Not here, you want to mix a feature request on it. But when we do it, you can't
separate. Could we also do a schema? Why not? But it's possible, but we're very very
go for the VCS. It's possible. Thank you.
Thank you.