Hello everyone.
Thank you all for coming.
My name is Augusto.
I am a software engineer at DataDog.
I work at a performance engineering team.
And my work mostly focuses around benchmarking tooling.
Yeah, hello.
My name is Kemal.
And I also work for DataDog.
I am part of Language Platform team,
which maintains client libraries for instrumentation
with all the languages.
I am, like, accidentally here,
because I was using seeing all these PR commands on our PR
says that, oh, there's a performance regression whatnot.
And I, like, dive deep.
And so that it's, like, something cool.
And I ask Augusto, like,
Why aren't we giving a talk about this?
Now that there's this total engineering team room.
And he said, like, why don't you do it?
So now, that's how we are here.
So let's start with the basics.
So performance matters.
Period.
So it matters because low latency systems,
we will see some numbers on these,
but it creates, if you have more performance system,
if you pull down your latency,
it will affect the user experience.
And then you will have high throughput on your online systems,
or like the queueing systems.
And in the end,
it's the better user experience.
This could be rather a U-A, U-I-U-X app,
or it could be any C-I-A,
or, like, anything you would feel these latency.
And on that, we, like, get a couple of studies.
You can see that on our references.
So according to, it's some sort of point of time,
Google measured that 200% like the 100,
500 millisecond delay,
cost 20% of traffic drop.
Same thing happened with the Yahoo,
if you just make things like 400 millisecond faster,
you got like 5 to 9% more traffic.
And these numbers are all probably,
these are, like, more sensitive right now.
And when it comes to the cloud cost,
because of all these performance things happening,
this is like, this is 600 and 75 billion dollars.
It's hard to imagine, apparently,
there's a market for that.
So if you're in this business, go for it.
So, yeah,
take talk without a quote,
wouldn't count.
So this is my first quote.
This is from,
from the CEO of Shopify.
Like, this is one thing that we need to understand, right?
It's maybe not the first class feature of our software system,
but performance is always counts.
And like, the all-vert class photo is there fast.
So when it comes,
how the user perceive this differences and how,
like, they are noticeable,
if it's like order of hundreds of milliseconds,
it's just minimally noticeable.
But if you go up,
then you can see that, like,
users are actually switching away.
I'm sure you feel this, right?
If you're a developer,
like, even I feel that every day,
every day, because of the CI runs,
I hate CI.
And this is, like, my pain, right?
It says 5 to 10 seconds.
Yeah, some CI systems, 30 minutes.
It's killing me.
So, if you want to get just one thing from this talk,
this is that, like,
write benchmarks, but run them continuously.
Probably,
we're going to talk about that,
but probably the benchmarks you have,
also wrong.
So, you also talk about that,
how to make them right.
So, a quick poll,
who has, like, written, has written a benchmark here.
All right?
Yeah, this is what they are trying to form.
I should have, this is coming.
So, who has been surprised by the result?
All right.
Okay, we have our audience.
So, about first, like,
why is what they are slow?
Why these things are happening?
First of all, like, CPUs,
don't recognize the better algorithms,
if you are, like, if you have put something wrong in there,
like, CPUs won't make it faster.
Same goes for the compiler.
You cannot, like, rely on your compilers.
If you are using a compiler right away,
I hope you are not writing Python.
Ruby?
Yeah.
Okay.
So, compilers won't solve all their problems, right?
And then, we always have this notion of, like,
we have a synthetic notation,
so we, like, write our algorithms,
or it's, like, oh, and whatnot,
but then you don't think about, like,
the cache misses, branch misses predictions,
so this is called, like,
mechanical sympathy, so you need to understand
underlying system and optimize your workflows for that.
So, according to another, like, recent study,
by, like, giving attention to these details,
they actually sped up, like,
60K times faster by just, like, tuning these things, right?
This numbers, you know, you can check the reference.
It's cool.
So, how to design benchmarks?
I think there's two major things that you need to focus on.
First, like, how you represent,
representative of your workload,
and then, yeah, it's your, like, results,
are your results are repeatable.
And repeatability requires a lot of signs
that I'm having a hard time to understand.
I will hand it over to August for when it comes,
and then the time comes.
So, Arturo, like, before we segue to that,
we need to classify what are, like,
the benchmarks, and for that, like,
I call this part as the Arturo programming benchmarks,
because we need to decide whether you would like to have
a micro benchmark that you want to write,
or a micro one.
So, when we say the micro benchmark,
it is, like, isolated functions,
operations, like, not a second precision,
the thing that you put on your, like, unit test,
if you are using code, you have an advantage.
There's, like, first class facilities to do that,
but they are, like, not representative.
They are micro benchmarks.
I should have put a code here, like,
the premature optimization is the root of all evil.
It was mandatory, but I forgot about it.
So, this, that falls to the same category.
Okay.
Do you really missed all the things that I said?
Sorry about that.
Okay.
And then the micro benchmarks is about,
they are, like, enter and test flows,
like, really realistic workloads.
They have higher variance, because you are doing a lot of stuff.
Maybe you are, like, talking to other systems,
but not, but they are actually closer to
what you are doing in the production.
And, like, it's hard to isolate the causes.
Like, you can maybe run a continuous profile,
or take a profiting snapshot,
while you are doing this, and try to optimize,
but still, these are the benchmarks.
Like, what you can additionally do is,
like, use a production observability system,
platform, shameless flag,
and then you can, like, discover your production things,
what's going on, and you,
basically know more about these things.
So, choose the right tool for the right job,
but it's in the end, it's a mashup,
like, basically use both of those strategies,
and use your judgment and find the bottlenecks,
and optimize.
So, when we talk about, like, representative workloads,
what are they?
They could be your applications,
could be, like, CPU ones, maybe IO bands,
maybe are, like, in a mixed position,
maybe you are saturating your memory bandwidth,
like, you need to understand,
and try to find the correct representative workload
for your benchmarks, right?
For that, like,
and what we try to come up with,
is some sort of archetypes, right?
While we are designing our macro benchmark tools,
like, it could be an ideal app,
mostly a thing there, doing nothing.
It could be, like, low latency or latency sensitive application,
or maybe it's, like, a throughput sensitive application,
or it's a mixed back, right?
We call it enterprise, like, you have a bit of everything.
And then we run all these things,
and try to get all the numbers,
and try to react to that.
Like, we are obsessed with this,
because, like, all the libraries that we write,
we put them in the production services
over, like, users,
and then if something happens,
like, they actually play us.
Like, I included that,
and, okay, like, this break my system,
because, yeah, we just make this observable.
So, yeah, we are trying to be super sensitive about it,
and reduce all our workloads for that.
So, how to design benchmarks,
this is where the science starts.
So, I'm leaving that to Augusto.
Okay.
So, we have seen that benchmarks have to be repeatable
and representative.
We are going to focus on the repeatable part now.
And the idea is to take a benchmark
that wasn't repeatable in the beginning,
and try to make it better with some few changes that we're going to do.
And often, in my job,
people come to me saying that, oh, I got this benchmark,
and it's not repeatable.
And what can I do?
And this is usually my first reaction to this.
And the benchmark itself has the goal of
measuring the database Java,
which is one of our libraries.
Instrumentation overhead on a spring app.
And it's a really simple app,
just made for this benchmark.
The system on the test was the app
instrumented or not with the database Java,
so we could measure the overhead.
The workload was as many requests as possible
by five concurrent users.
We wanted to measure the system at a really high load.
And we separated it into two stages,
a 22nd warm-up, and 15 seconds of actual measurements.
Cool.
We have these initial results,
and they look right.
We have the warm-up,
and it seems like we have cut the data points
at a place that makes sense.
But actually, we had many false positives,
so the benchmarks were telling us that we had some improvements.
When in fact there wasn't really an improvement,
and we had a really high coefficient of variation,
and this is just the way to measure the variation of a sample.
It's the standard deviation divided by the mean.
And it was 11.80% which was unacceptable for this use case.
So the first question that I asked myself was,
okay, are we running the benchmark long enough?
Is this really the warm-up?
Turns out that no.
It actually took way over 15 seconds.
It took a little bit above 150 seconds to warm-up.
And so tip number one is to run your benchmarks for long enough,
so that you can uncover perturbations,
and one of such perturbations have to do with warm-up, okay?
Cool.
Ideally, we could be able to run benchmarks for a really long time,
and get consistent results, but we don't have infinite money to do that.
So we have to choose when to stop,
and for how long should we run the benchmarks?
To know that, you have to experiment.
And here's an example of experiments that we do.
We take the measurements and cut them at different steps
to understand how the variation behaves.
So for example, at 30 measurements, we have a 7% coefficient of variation,
and when we go to 90, we have a way lower coefficient of variation of 4.6%.
Okay?
But it will all depends on the benchmark that you want to do,
and what you want to measure.
Tip number two is therefore to connect collect enough samples,
to reduce intro-run variation.
And a good rule of thumb is to have at least 30 samples for that.
Okay, but if you run this benchmark several times,
you might even still get different results.
There is something we call inter-run variation,
when you run your benchmark multiple times,
and you see different results like this.
This is a screenshot from a paper that we reference internally,
and it's measuring the initial state,
the impact of the initial state on FFT benchmarks,
and you can see that on every run,
separated by the vertical lines, we have a different latency measurement.
We can do the same for our experiment,
I ran it five times,
and by plotting the graph in a similar way,
we see that the same effect happens.
On every run, we have different measurements,
so we have to take that into account.
When we see the results,
and you don't really need to read all of the table,
we just focus on the Ming values,
which concentrate around the same value of 20 milliseconds,
so we know the benchmark is consistent,
and if we aggregate everything,
so in the last row,
we see that the coefficient of variation is way lower,
at 3%, which is already a good spot.
Tip number three is therefore to rerun benchmarks,
so that you reduce this inter-run variation.
A good rule of thumb is to run it five times,
but again, this is something that we use internally,
depends on your problem.
By applying these three tips,
the coefficient of variation down from 12% to 3%,
which is already great,
but there are more tips that I would like to briefly talk about.
I don't know if we will have enough time.
The first one is to use deterministic inputs.
If you have random inputs,
you will have random outputs,
so that is not good for repeatable benchmarks,
and the next one is to use low generators
that avoid coordinated omission,
and coordinated omission is when your low generator synchronizes
with the system of the test,
in a way that you get artificial latency results.
So if you have a low system,
the low generators slows down,
and then you have artificially better latencies.
And if you want to measure your system at a high load,
as it is in our case, this won't write it.
That is a really good talk by Jutany about
how not to measure latency,
where he goes really deep into the subject,
so I highly recommend it.
All right, so we have a well-designed benchmark.
You are getting results,
but you cannot rely on simple aggregated data points
to tell the whole story.
You've got to look at the entire data,
and aggregate values won't tell you that.
To show you an example,
suppose you're working on a project,
and you make several commits,
and at some point you decide to make an improvement,
so you make an improvement,
and you see some improvement on the throughput.
But is it really an improvement?
By looking at the main throughput,
you can't really know.
Zoom in in on these two last commits
and there are benchmarking results.
We can see that there was maybe an improvement,
but if we plot all of the data points on a histogram like this,
there is quite a lot of overlap.
So maybe if we have ran this experiment another time,
we wouldn't have this same conclusion.
And we have to somehow evaluate
how big the difference between the first and the second one is
to make an adequate conclusion.
So the question is,
how can we tell if the difference is big enough?
First of all,
we see that there is some spread in the data,
so we have to compare the difference with the noise.
So you can set up,
wait up.
I forgot about this,
but yeah,
I won't get too much into the map.
We can just focus on the intuition for now,
because otherwise,
we would be here for a long time.
So we can set up a simple ratio like this
to tell if the difference is big enough.
So we compare the difference with the noise.
We can call this a guy T,
just to simplify things.
If T is bigger than some critical value,
then we can say that yeah,
there was an improvement.
How to define this critical value?
Mathematicians,
I've thought about this,
and it turns out that you can plug in a false positive rate,
which we call alpha into a formula,
and you get a critical value.
And this false positive rate is something
that you can control
to,
as a tradeoff,
to detect more things,
or to detect less things.
But you are also going to detect more false positives,
or less.
It depends on what you want to do.
So for example,
if we have a high false positive rate,
we are going to detect more things,
but also more false positives.
Cool.
And we can write the critical value
as a function of alpha.
And mathematicians,
as I said,
they worked really hard on this.
There are formulas that you can look up,
and the critical value is noted like this.
And in fact,
this is a hypothesis test,
and it's T-tats specifically,
and this is why the name of that ratio is T.
But that's all for this talk.
I won't get too much deeper into it.
There are other approaches for that.
One example is change point detection.
And for that,
you can check out Enrico's in-go-stock
at 150 p.m. on this room.
So be sure to check that out.
Tip number six is therefore to use statistics.
For example,
hypothesis testing,
to see if your performance improvements
or regressions were statistically significant.
Cool.
But what if you run your experiments
in the morning and then in the afternoon,
especially if you're using cloud environments,
you're likely going to get different results.
The mitigation for that
is to control your benchmarking environment.
And this is, in fact,
so important that it's actually tip number zero.
Otherwise, all of the other ones are not going to work.
And we are going to try to see how we can do.
Okay?
Are you still with us?
Number part is kind of over.
Yeah.
So you can be relaxed.
Now we have a story.
So imagine your researcher
and you're trying to measure the speed of
subatomic particle called the neutrino.
And the neutrino is a really small thing.
You can't really measure it on your laptop on a desk.
So you've got to build a huge ton of it.
And we had people here,
given the previous talk,
Max, which was an amazing talk.
That talked about certain.
So you build a ton of from certain to
Gran Sasu.
A 730 kilometer ton of.
So you can measure the speed of the neutrino.
And it takes a long time.
And a lot of money.
But when you start getting results and
people are publishing papers,
everyone is happy.
But at a certain point,
you start breaking the laws of physics, right?
And this shouldn't really happen.
So you double check everything.
You check the math, the sensors.
You check and double check everything.
Until you come across the root cause,
which is in a revolutionary physics.
But it's a loose cable.
So this goes to show that really small things
can impact highly controlled environments in a big way.
Okay?
Most of us aren't viewing 730 kilometer tonals.
Some of us are.
But we have to deal with this.
So to speak loose cables,
every day when we are working with softer performance.
This is a no extensive list of sources of noise and
mitigations that we can do.
We are not going to talk about all of them.
Only this one.
Okay?
This is already quite a lot.
So I have to go a little bit fast.
All right.
First one, virtualization.
The problem with virtualization on cloud environments
have to do with the noisy neighbor problem,
which means that on a single host,
you will have multiple tenants competing for resources.
And this will give you no repeatable benchmarks.
The mitigation for that is to use bar matter cloud distances,
and besides getting rid of the noisy neighbor problem,
the kernel and CPU layer mitigations that we are going to see
on the next slide requires bar matter access.
This is basically you have access to the hardware of a dedicated machine.
Okay?
These are the kernel level mitigations.
They have to do mainly with scheduling and caching.
They are studying CPU affinity, process priority,
and warming up or dropping caches.
Here we are dropping caches.
And I won't get into the comments themselves,
but this is something that you can copy and paste,
and immediately have a more repeatable setup.
With regards to the CPU layer,
we have simultaneous multi-threading contention,
and dynamic frequency scaling.
They are a little bit tricky, so we are going to see each one separately.
So simultaneous multi-threading,
SMT is when you have multiple hardware threads running on the same car.
So you are going to have some resource contention
if you have CPU bound processes.
To disable SMT, you can simply run this command on unix machines,
and the impact of disabling it was measured by a really simple experiment.
So we had a bare metal instance,
and with dynamic frequency scaling disabled,
which is the other CPU layer mitigation.
We are using two CPU bound tasks on the same car versus separate cars.
And we can see clearly that when we are running things on the same car,
there is contention, so the latency is higher.
And if we look at the numbers,
the coefficient of variation goes down significantly
from 24% to 0.24%.
So this is 100 times less variation,
which is huge for a really simple trick that you can do.
Dynamic frequency scaling is when your CPU works harder for hardware tasks,
and works less hard for easier ones,
so that you save energy.
But this is terrible for repeatability, right?
So it matches the workload,
and you don't really want that for repeatable experiments.
To disable it, it is a little bit more complicated.
You have to pin the clock rate to a frequency that you want,
then you have to control the scaling governor,
which is a kernel subsystem,
to make it use this clock rate,
and then you have to consider things like frequency boosting,
which are features of modern CPUs as well,
and you have to disable them.
The impact of disabling DFS was also measured by an experiment,
and here we're measuring the latency on a varying number of CPU
bound tasks on the same core with DFS on versus DFS off.
And here, of course, when we have DFS,
we have a smaller latency because the CPU works harder,
but we can see that the variation is quite bigger when we have DFS,
which we don't want.
Again, you don't really need to look at all of the table,
but you can see that the coefficient of variation drops significantly,
when we are looking at DFS for one task,
in comparison with no DFS for one task.
In this case, it's 10 times less variation.
And of course, there is a caveat that these experiments were made,
specifically to test this,
but it was a way for us to test the boundaries of these mitigations.
Those experiments were meant by my manager, in fact,
as you meet with your chanko, so close to him.
And these C-pups run button.
The CPU level tweaks were mostly sourced at Danish baccalaves book,
about performance analysis and tuning on modern CPUs,
which is a really good reference on this.
There is another source of noise,
which has to do with vibration.
And the mitigation product is to not shout at the data center.
And in fact, it actually happens.
So this is Brendan Gregg, performance engineering legend,
shouting at the data center,
and seeing some discounted latency spikes.
So don't shout at the data center.
So after all these cool stuff built,
and they are doing the maths, what not,
but as a developer, what we see.
So first of all, let's check out the architecture.
This is a simple one.
What we do is, like, GitLab is an implementation detail,
but that's where our C-IPLs work.
We run these on Kubernetes runners,
and collect the data, store them for our in-and-art effects,
storage, S3 whatnot, for like,
analysis later analysis.
We have a specific, the purpose build UI,
where we analyze them.
We also get their metrics out of them,
and push that to our dashboards.
And then the action part comes.
They're like, all these statistical analysis tools run
on the data that we aggregated,
and they send comments to the GitHub,
PRs, or they block our releases.
So you can think the whole system,
like the action part is like that is the alerting part
on the monitoring system,
and the rest of the things are just like collecting data,
having dashboards whatnot.
So that feedback loop, how does it work?
As we talk at the beginning,
we segregate them into two classes,
is the microbenchmarks, and the microbenchmarks.
We run microbenchmarks for each PR.
These are benchmarks that are written for the person,
think of people that actually write these client libraries,
and they know what to measure,
and they write these benchmarks for themselves,
and it gets run.
We do the static analysis.
And if something is not looking right,
we send these comment messages to the PRs.
And these are actually like,
basically merge gates,
so you cannot merge the PR without like addressing these concerns.
This could be anything,
like maybe you're increasing the memory,
you are like introducing higher CPU usage whatnot.
In theory, yes, sometimes you are adding a feature,
and then you need to say that, okay,
like this is somewhat something I was expecting,
but most of the times,
or like what did I mess up,
and you go back, run the benchmarks locally,
collect some profiles,
and try to optimize these things.
And the second floor,
follow is about running the macro benchmarks.
These are the representative loads that I mentioned previously,
with the archetypes and everything,
and then we run them.
If this is a virtual machine environment,
we even like warm the machines,
as like Augusta mentioned,
we have different type of archetypes,
plus like low,
like lower higher throughput,
like V-run punch-off scenarios.
And then depending on that,
if we also set SLOs in our benchmark environment,
and if those variations are like not breaching our SLOs,
then we allow releasing these libraries.
But if they do,
and then we go back,
and focus on how we optimize those things.
This is,
how you see,
like we get notified by blocking releases,
just like message.
Everybody uses like nowadays,
so.
And then we have dashboards on the SLOs that we put,
and we also can go and analyze what is going on,
depending on the scenarios,
and we have like different server to levels,
and then we actually,
even though we don't breach anything,
we also check out the trends and where they go,
and we have the thing called operation excellence,
meetings,
and we check these dashboards,
and try to understand what is the trend,
why this is happening,
and we try to be proactive
and fix this issues before they hit the production.
So.
This is what they built.
It's not open source,
unfortunately, not yet.
This is not something we're going to say,
so we plan to make it open source.
But right now,
these are the tools that they are actually,
you can use.
Like high-profile is a CLI benchmarking tool.
I need to already handle,
running your benchmark a couple of times,
and collect data analytics,
and it is better than most of the tools.
Or if you're working with go,
go benchmarking facilities already,
that I was this for you.
And I think,
like Henry going to mention about the GitHub action benchmark,
in more detail,
but it's something that you can immediately use right now,
if you're running your benchmark on GitHub actions.
And like,
what is this key takeaway in the end,
or all these things in mind?
So control your benchmarking environment,
so this is the most important part,
I guess, like try run your benchmarking on your benchmark,
on bare-metting machine, in isolation,
disabling SMT,
and disabling the FS.
Unfortunately,
you can do that on a virtual machine environment,
so like out of the box,
GitHub action runners won't help you,
so you need a solution for that.
And design your benchmarks to be representative,
and like repeatable,
that's the other most important thing
that Augusto already mentioned,
like there are a lot of variations going on.
And then like interpreting the benchmark result,
is the most important part in the end,
maybe there could be false positive,
it's like there's no way to automate these things,
and you just need to use your better judgments,
or maybe just re-run your benchmarks again.
And then for us,
like from a user's perspective,
integrating benchmarks into your work-it-flow,
and continuously optimizing your workloads,
and be aware of like basically having
performance engineer mindset,
it pays dividends in the long run.
But maybe the most important part,
don't shout to your data centers.
Maybe we don't use the 8HDD anymore,
but now we have AI alerts in there,
so make sure you don't shout at them.
So, for that, thanks for.
I fully understand wanting to reduce the noise and variance,
but if I'm targeting desktop and laptop machines for my software,
I feel like for example,
disabling multi-threading is kind of lying to myself,
because I'm not representing the machine as it is.
So, what do you do with that?
I couldn't hear entirely the question.
So, if I'm targeting desktop and laptop machines,
I feel like if I disable multi-threading to reduce the noise,
it does reduce the noise,
but then I'm not being true to the actual machine that I would run.
Yeah. In this case,
there is a clear tradeoff.
There is a clear tradeoff between representatives and repeatability, right?
So, in our specific case,
where we need to have repeatability,
as we are running benchmarks on every PR and every release,
we need it to disable them.
And if you need more representatives,
as we do for some tests, we keep them on,
and we see what happens if we have those things on.
So, I think the better answer is to run everything,
that is probably not possible.
So, it depends on what you want to really focus on.
One of the things that I do, like locally,
use Dr. Compos, and try to isolate these resources
against the Dr. Containers,
and there is much as possible tracking it
to get that content.
Yeah.
I touched this thing,
and you said,
constantly, that has another rise.
There are,
I think, running it now on the team
and dealing with partners.
Yeah, thank you for the question.
This is a really important point.
Sorry.
I had one more question.
So, during your talk,
you mentioned that, like,
so an important aspect of benchmarking is to test.
So, we have to assess the environment,
making sure it is well.
And, as you said,
we don't know how, for how long we won a test,
and it cost money to test.
So, the question I had in mind was that
was there any perspective of,
instead of testing,
expecting the results.
So, for example, using machine learning models,
or deploying models to expect the result,
which would maybe cost less,
would be accurate.
So, I want to ask if you have any information,
or pretty about that.
My team has already researched
a little bit about performance models.
We had, have not,
gotten too much deep into it,
because I think it was less costly
to simply get the benchmarks running.
It would take too much developer time
for us to focus on this problem itself.
So, we had this in mind.
We're probably going to look into it,
probably running more in more benchmarks.
But for now,
it is not something that we have looked into,
but that is a really good thing to look into.
If you can be done well,
you can save a lot of money,
and really predict,
like,
maybe not at the really accurate level,
but predicts in a general sense,
the performance of your system.
Yeah.
Thank you.
Are there any other questions?
Okay, then.
Thank you for your presentation.
Thank you.