So, thanks for inviting me and for organizing
track on our favorite topic.
I don't think I have ever spoken at a conference where the previous talk among other
things was a project that was one of the Nobel Peace Prize.
No, no, Nobel Prize in physics, in certain.
So, it's like a high bar to follow, but the other talk has been good.
I will try to do my best.
I have to talk about what I have been calling continuous performance engineering, something
I have been doing for the past 10 years, in various database companies and now with my own
company trying to productize and evangelize it.
Doing this presentation, I realized that everybody else is talking about continuous benchmarking.
So this is where we are, we haven't even agreed on what we should call this thing that
we know we need to be doing, but I will after this talk maybe adopt this as well because
I realized that majority is gravitating there.
There was a poll earlier, but people have changed a little bit, so this is of course, first
them and out of first them, this is the few hundred people who actually care about performance,
but it's like somebody in the room who not just you, but like in the project, there isn't
really any benchmarks and nobody is doing benchmarking.
Yes, and for some projects, this is actually valid, but then maybe the second option, yes,
we do benchmarks, but not running continuously every day, but once a year, with the release
candidate, I'm told to past it, 10 or so people, maybe.
So I think things are not as bad, we are making progress, so I think the number 3 is probably
going to be popular, so we do have benchmarks running every day in continuous integration,
but we don't actually look at the results or even if you try to look at the results
there, not very useful, who would identify as this one?
Seriously, if, okay, so number 4, we have benchmarks in continuous integration, we run them
every day, at least, or maybe every commit in the best case, and we get actionable results,
we assign tickets to the engineer who did the progression, and they are fixed within weeks,
rather than in December before Christmas.
Now, okay, and most people are just listening, that's fine.
So, fairly even distribution, this is still better, I did this some years ago, and there was like
one who is in number 4, but like I said, of course, we are, this is like maybe
out of all of Europe, the engineers who care about the finance might have been in this room, so
not so bad, we are making progress, but still, you know, if you ask the same question
like about unit testing or testing, like who, who has automated testing,
that, and not even like flakey, the tests, we have good results, and then we, if something fails,
you know, we probably catch it in the poll request, don't even merge it to me,
this is basically everybody doing this for 10, 15 years at least.
I had in 2006 a project where we had Nokia, so one guy actually had the phone and with his own
fingers, he had a hundred tests that he would like physically press on the different buttons
and user interface. He was finished like at 15, 16, in the afternoon, went home and in the next
morning he took another phone and did the same program. This of course never happens anymore,
but with performance, we are a little bit behind, I would say, it's not like a given that
all the projects have some performance tests and we know which tools to use and we like just,
you know, put some Yamal lines in into GitHub and it just works.
So we are, I would say, 10, 20 years behind, so we are like the Velociraptor, which is the fastest
dinosaur, so it's fast, but it's not there. And where is that Augustan Kamala right here?
Yes. So I had to steal your slide because it actually is the problem. This is exactly the problem.
I think there are some other problems. One is that performance and benchmarking is kind of
difficult. Deploying, I automatically deployment, I think it's also difficult until we had
tools like that of form or something which you can just, you know, let's take care of the hard
problem. But the thing with the automated deployment is that when we do software, we do need the
deployment. I've got to watch there is no point in what we are doing. You can kind of postpone
the performance stuff until customers really complained. So this might be one reason
that you really had to solve the deployment and also to benchmarking, you also need the deployment.
So there is a dependence, you need to solve the deployment problem first anyway.
Now when I've had two years start up when I'm trying to sell performance and benchmarking related
services to companies, one thing I find that the performance engineers are often very busy.
Might take like weeks or months to talk with them because they are at some important customer fixing
performance issues in production. And because we are fixing performance issues in production,
we don't have the capacity to do to kind of evolve that tooling that you could fix them before
they get into production. The other thing is, and I used to do this as well, sometimes it's kind
of nice to be the hero that you get to work with the most important customers and you go there
and you know spend some days doing something and then it's fixed and everybody's really grateful
the account manager takes you to a nice restaurant and then you fly to the next place and do the
same thing again. So maybe we don't want to fix it. I don't know. Certainly,
and this was like genuinely my slide before I saw the data dog talk, but the part of the problem
is that math is kind of difficult but unfortunately we are going to need it. So that is probably
one problem as well. And later when we talk about the tuning which also has been covered a
little bit here in previous talks, I find it most performance engineers actually unintuitive
how to tune a server to get repeatable results. So what I'm trying to say is that maybe we
need to do what was done for QA testing and deployment. Some years they had like a DevOps movement.
I actually first time I heard about DevOps was in Boston probably like 15 years ago or so.
And if there was to be like continuous benchmarking movement, these are some
some people and conferences and open source projects and companies that you might
want to follow. Scott Moore is very active on YouTube and LinkedIn. So he's a good
good place to start. And if it wasn't obvious, New York here is the company that
I have been pushing for two years and I'm trying to, the goal is to mainstream some
open source tools that in the past 10 years have become developed and open source and so on.
So I'm here to tell you what I've learned and made available in the past 10 years and I hope that
you will also tell others about these solutions that continuous benchmarking actually is possible
and achievable and getting the repeatable, stable results is something we can do if we share
the shared knowledge how to do it. And from here on the talk has three points.
If you ever see in Finnish President's tube, there is not one interview where the answer
wouldn't be structured into three points. So I believe this is a good way to go. Now for the first
one we and we had kind of synchronized before that the benchmark design in itself is already hard
even like let alone doing it like repeatably or continuously.
So I'm not going to repeat a lot what was already in the previous talk and part of this is kind
of well-established already. So the nice thing is we have these frameworks typically each
language has like one or two. So if you are writing in Java you do JMAG and so on.
So that's kind of well understood and then of course if you use these frameworks you can
relatively easily put them into GitHub workflow or Jenkins and so on. So here we have a good starting
point. I have also in my career developed frameworks to run like distributed benchmarks
so like deploy an entire MongoDB cluster and then deploy some application that creates some
work load and then run the benchmark and then of course because it's in the cloud all these
servers and the spins so you need to collect all the log files and results and so on. And it's quite
complicated to do that. I believe this is an area where solutions are not very established yet but
I see a lot of opportunity for innovation here. So I'm not going to talk more about that but it's
like to highlight an area where I think this isn't very established yet but even if if you start
with single node benchmarks with the well-established tools that's a good starting point I want to
mention one open source project that is like relatively popular and active and I have also like
forked and a new conversion of this one but this has been this is existed for I think six years
at least and has several contributors and so what this project does is you run your benchmark with
the frameworks from the previous slide and this GitHub action benchmark has parses that understand
the output of all of them and you can of course contribute more. I saw recently some activity
I have contributed also a parser for if you want to just use like the time command line utility
to like execute something from the command line which of course isn't very very like a high
fidelity result but it can be like a simple way to do at least something. So hopefully that will
get merged upstream and what this then does is it collects your results and there's some
interesting way to basically maintain a database or like a JSON file in your GitHub repository
which I wouldn't do because your analytics gets kind of embedded in the test history itself
but it's kind of a simple way it doesn't cost anything when it's in your GitHub repository
and then it has threshold based alerts and default threshold and and this highlights the
problem I'm talking about in this talk default threshold in this tool is 100% so so if your performance
describes 2x then it will create a ticket or depending on configuration failure or pull request
and block it and but 2x is quite large is the point so we would want to maybe catch
regression as much smaller than that. And this leads to topic number 2 how do we do change point
detection what is change point detection so topic number 1 we have design benchmarks
we try to design them well so that we get like useful results but they are still noisy
this is from an old publication we did at MongoDB excuse me I think I need to take medicine before
got to take it before it or so so this was a MongoDB test I think at least the first one is
one of the worst case scenarios that we had in 2015 and in this picture there is one regression
on one of the graphs but everything else is noise so how do you how how do you work with this even
like looking at it as a human it's not easy to spot where it's the regression let alone having
something automated where you you have no human looking at it and it should it should find the regression
so but this is normal I don't know those who had like in in the data go dog case for example
20 30% up and down and this is not I know this you for example had variants in your slide
we decided in MongoDB to look at the range from minimum to maximum
result which is stricter but we felt it was kind of like meaningful that we want to have
all all test runs behaving well and so then you get get up to 70% outliers sometimes but
but there is an actual change that is persistent so this was caused by something in the actual
git commit at that point right so this means this means when I was on the team there were
four of us so in a month each of us had one week where we were we were attached to the triage results
coming out of the out of the CI system and you can imagine that mostly they were false alarms
when input data is like this this is something more recent is from the Nürkia
side and some of our users choose to publish their results especially for open such projects it's
convenient because the contributors don't need to create an account and and here also this is
torso database finish American project very popular they get like 20 pull requests per day
many of them from external contributors and simple two simple test select one select count star
this is with the git how default runner very so once in a while you get this kind of hiccups
up and down 40 to 50% compared to the here you can of course with the human are you can see
where is the kind of normal result that that's that's the one that is kind of without these effects
from the infrastructure so now if we do threshold based alerting on this kind of data in
MongoDB or third database it's like first idea that comes to mind here is a generated data set
and and on these two graphs the green line is like an actual change which could be like regression
and everything else is false alarms and in 2015 this was my job so one week every month
I would come to work on Monday and I would have to look through they were like issues created out of
all of those and you know already before you look at them that maybe today 100% of them will be false
positives or like once a week there would be like an actual regression that you then could
file and send to the developer to fix and we were lucky so the first thing we tried
something else than threshold based alerting was this kind of algorithm from Mattis on
and James and that had just been published a year before and you need to understand some
math that we were also lucky that we had an intern that actually studied math and went back to
the master and PhD later who then did the first version of what is today the Otava
incubating project in Apache and this is so so this is the same graph so this is the same same
times is now in both of these graphs the top one is the change direction algorithm and the bottom one
is this like just comparing like some percentage compared to previous point gets an alert so
the key and the key of how do you get this kind of better performance is that it looks at more
data of course than just just the previous point or some some small window and the key word here
is many monitoring tools that have these kind of functions have been designed for monitoring
production and that is a different problem than finding regressions in like in in poll
request before you commit why so in production so the two phenomena you have are outliers which
can be like a spike or single event and then change points which means there is a persistent
change like in the middle here in this demo data set and and we are now only interested in
in in change point the pection so we want to if there is a regression in the actual code it
means like when we rerun the benchmark tomorrow the regression will still be there and then everything
else that's from like network flakiness or noisy neighbors or so on we want to ignore but a lot
of the a lot of the existing tooling actually is designed to also alert you on outliers and
and we want we consider for us that's a distraction okay so that is change point detection
you can check it out on a patch or but this was transfer formative for me and the team
and and it's a sense we published it has been used by many other companies and and that was
the first time that we started believing that this continuous benchmarking is possible in
the sense not just that we can execute the benchmarks but also that we can automatically
analyze results in an actionable way and also actually back then we did another project where
where we looked at the other problem is that why is there so much noise in the benchmark
that results in the first place and now I can a little bit there's a little bit overlap between
the previous talks but let's say if you were here for the data talk this is like wrong answers
only slide so don't like we already know the answer but but if I ask some people who went
sitting in this room today then the typical answers are well you shouldn't use cloud infrastructure
for benchmarking probably we had noisy neighbors or there is some other kind of bad cloud
instances actually this previous talk from cover you and it's true that sometimes you observe
this correctly and there's reason why this kind of change in performance happens but it's not
across the population it's not well understood what what are the reasons why we why we can't
get like easy to analyze reliable performance results and again we had to go back to our core principles
let's let's apply scientific methods or not just math but also like okay let's let's stop
you know quoting things that you read on somebody's blog and let's get to work those on
experiment and see what the results are and then you repeat this we we actually spent two engineers
spent the three months where the objective was to find out that what would be a good
configuration in Amazon in our case to minimize this range from minimum and maximum result
of benchmarks where we knew that there was no regression in fact the benchmark like the same
for me all the time 25 times and here is a fun exercise you can do I'm not going to read all of
them what we will come back so so we we kind of retroactively listed the assumptions because this is how
science work you have hybrid thesis then you test and then you test either confirms or or
invalidates the assumption and then you go to the next one and these were assumptions that were built
into our benchmark in infrastructure but they had never been validated they were just accepted as
fact so for example at the end if you if you read the third one there was there was this kind
of belief that when you launch cloud instances you can kind of sometimes get bad instances
for some reason like maybe noisy neighbor or maybe they're just bad without neighbors and
because of this we had a system where we run some kind of test benchmark first and if the result
wasn't above some threshold we would shut down everything and try again three times and then the
third time we just kind of gave up and run the benchmark anyway and yeah so now to what we did was
instead we took like three months that we didn't really test MongoDB that much and we instead
we tested like our own infrastructure that can we get better results out of it so we stopped on a
on a on a specific git commit that we knew were good and so it was always the same binary
and because you you don't want to have too many moving parts right so you you you only want to
change one thing in the test and keep everything else constant and then we launched five
different servers and repeated each test five times and here it's like from an old publication
again copied I copied what I did and and this is what it looks like so now immediately when you see
this result what can we see from it that validates or invalidates the assumption on previous slide
this does not show so like the five five first dots on each line it's like from the same
server and then like five to ten is the next server and this does not show that suddenly there
would be a server where the five dots are significantly different you see maybe like the first test
there's a bit like the cold cache effect and then like test four one to four a lot of five are
faster but generally there's a lot of variability here but but the variability does not
correlate with like a 0.5 10 15 where we changed server so this was false and imagine this was also
at the time when we paid for Amazon instance is per hour so imagine we would start 16 node
shorted cluster in the worst case they would run this test and say oh no this is like some of
the null server let's shut down everything we had already paid for an hour let's start 16 new ones
this was all based on the false assumption but this does happen the reason this didn't happen for us
is we used C3 family instances which are kind of like a bit more expensive and better quality
if you use the M instances it's true that you can get different CPU generations
even if you you are using like the same same instance type from Amazon they might I think they
kind of give you like more better CPU than you are paying for or something but in the C
with the C family still today this does not happen so so this is also a good lesson now when you
go home you could do the same mistakes and I listen to Hendrick at first them and he said that
this does not happen but actually it can happen if you don't have the exact same instance actually
that I had so you need to do the testing and kind of rigorous scientific attitude so this was
covered so once we iterated over this and kind of changed various configurations we found that
for example the CPUs are not designed so infrastructure that we use isn't even trying to produce
repeatable results in fact the CPUs have a lot of features that explicitly change performance
all the time for environmental reasons for example to save energy so so there is now what
is it's called CPU power client the common line utility which you can then use to turn off
most of these features so that your CPUs have and you can use this in the cloud some
things you can not do on virtualized infrastructure but but actually it turns out that mostly
the cloud mostly it's not the problem to do benchmarking in the cloud like all of these things
if you had your own server in your own laboratory this would still be a problem and you would
have to turn this off by configuration the other big source so we had assumed that
that if like SSDs of course are fast so that has to be good for benchmarking we had assumed that
if you use the so called local SSDs in Amazon that's probably good but that was the biggest source
variation and the explanation is if you read Amazon documentation know what nowhere does it say
that these are local to the server instance so they go somewhere top of rack or somewhere where and
this is where you have the noise neighbor problem so actually the CPU virtualization sorry
CPU virtualization is quite good quality and in my experience CPU doesn't have a lot of noise
neighbor issues this is the reason we have and this could like even on the same instance during
the same hour this could suddenly change a lot because of the SSD and if you use EBS at the time
we use like provision IOPS this is very stable performance because Amazon's are cheap so
if you pay for like 5000 IOPS you would get exactly that not more and this is what what it
looked like when we then deployed the improvements in production so we were able to get more stable
lines in fact in the MongoDB case and there is a blog post still public and there is a link at
the end of this presentation we got all the tests within a 5% range which is a significant improvement
compared to something like 40%. One thing we did in a part of this project we also did tests
where we omitted MongoDB completely and just like tested CPU and this can network performance and
then we thought hey why don't we put this in the CIS as well so every day we kind of
verify that it is the infrastructure still behaving the same as before and generally the answer is yes
but one day we came to work notice this like January
for in 2018 what happened all cloud vendors released fixed firmware fixes for hard bleed
or what was the other one I think this is hard bleed so this is a curative fix and all our performance tests were red
and we thought okay so who screwed up but because we had the canneries running to say oh no actually
this is now a problem in the infrastructure so but by that time we had already been going a year or so
so this is this is like exceptionally event that the regression was explained by the cloud
environment okay so I've been working on this now last half year again 10 years later what has
changed oh so yeah okay so if on on Nürkir.com I've done the same kind of tuning so so you can have a
GitHub runner and there are there are many companies that offer these third party runners and often
they are like faster than the default runner and or cheaper or often bought and Nürkir is the only
one who is offering runners that are not faster and not cheaper but they are better quality
because they are configured for repeatable results and okay I'm apparently out of time but
but using this is quite simple you install the app and it's like one line where you can change the
runner and I just want to say I was personally not expecting this but in some of the tests this
is the same that was in the beginning of the presentation select count star when used which to this
configuration in the torso project you actually get like for a time period when there is no regression
in the code itself it stays within one nanosecond and I was quite impressed by this so you can
use cloud for continuous benchmarking and if you have your own hardware you will have the same
problems and this same configuration is needed so I think this is a good slide to end
do we take any questions nope I will hang around here for the rest of the day so happy to talk
you
sorry small announcement could you please help us and leave through that door