That if you would like to benchmark your BPM programs, you most likely need to get creative.
And sometimes even a little bit weird in a sense of what kind of work loads you're using
to benchmark your applications.
Workload plus configuration.
And the reason being as we have learned in a hard way is that if you've got your BPM programs
for security, like we do for example, or for monitoring or for something else, all your
programs are exposed within the kernel to all sorts of weird and strange phenomenon
unfortunately.
And this leads to a situation when sometimes you get a very counterintuitive performance
results when you test in your application.
So that's what we're going to talk about.
And to poor point, I've got one particular example.
It's some sort of a deep dive in the next couple slides about the BPM framework.
Sort of a case, study to show what kind of unexpected things you might actually expect.
Well, unexpected things that you're going to expect, it's a strange thing but nevertheless.
So it's a benchmark from the kernel self test.
So you know that there's some amount of those that are quite decently written, very interesting
benchmarks.
They are very interesting.
And this is roughly the diagram how the Ring Buffer benchmark looks like.
It's a very simple one.
So we've got a small BPM program.
This program is getting attached to a simple CIS call, this one, I believe, like, get
process a group ID or something like that.
Every time when the program is getting triggered, it tries to create a bunch of freckards
and tries to write this bunch of freckards into the Ring Buffer.
So this essentially very simple kernel part.
And now there's also user space part obviously.
So you see there are some amount of producers marked with P. They are duty essentially
to call those like to trigger those CIS calls.
So they are essentially producing an input into our system.
They are creating the workload here.
And on the other side of the thing, we've got some amount of consumers that are reading
from the Ring Buffer.
And by this, we're essentially getting some sort of a loop of events that are flowing
through our system.
So that's how it looks like.
And obviously, it's a benchmark.
We have to measure something, right?
So there are two things that we measure here.
One is a successful event, writing activities.
So every time when we were able, actually, to reserve space and submit something to the
Ring Buffer, it counts as a hit.
And every time we were forward, whatever reason, we were not able to do this, whether
it's a lock-in issue or Ring Buffer is full of something like that.
It's called as an event drop.
And essentially, based on those two metrics, we could get some rough understanding,
okay, how quickly our system is performing within this configuration, right?
That's pretty much it.
Very simple configuration.
And I said, I really like this benchmark because it's very, sort of, like, an
good example of how could you test something?
You could, you know, you can provide concurrency, for example.
You could create different amount of producers and consumers.
You could pin them to a different CPU course to see what is going to be like.
Intercore interaction.
You could also figure out, for example, how the batch size is going to affect the overall
performance and all those, you know, pipeline and effects.
Oh, it's very interesting.
But there's one interesting thing that got my attention.
The work of generation itself is a tight loop, meaning that bearing the back-to-back mode,
essentially what happens is that those producers are just like creating their firing
the Cisco and then Cisco is finished and that's when it's possible they're just firing
another one within the wild loop.
It sounds very straightforward and reasonable, absolutely reasonable.
I'm not saying that it's better anything.
It's a way to drive your system essentially to the edge and that's a little reasonable first
thing to do.
But every time when I see such a situation, I have to think about this article.
I have mentioned here, open versus closed.
Now going into the deep details, it's essentially a very interesting result from the
Q and theory.
As you may know, everything in computer science is a Q that's like Q and theory is sort
of important here and this result in particular shows us a very interesting situation.
It compares closed system that are somewhat similar to this tight loop where essentially
what happens is that we've got a system, we send an event into the system and then we wait
until the event is getting processed to send another one.
So there was a fixed amount of event in the system where the open system is something
where we just push an event into the system based on some distribution and we don't care
how fast or quickly are they getting a no-process, we just push them in.
And the point being that the last one, the open system, they're actually more realistic
and they are those that actually not see in quite frequently in the benchmarking exactly
because of this tight loop situation.
And at the same time, in terms of metrics, those open systems tend to be, they don't have
a higher latency unfortunately.
And in that for the realistic workload, maybe you will get actually quite different
tail latencies.
So, I haven't said that, I always think okay, what will happen if we will replace this
workload generation with something different?
And on this premises, I have performed a couple of experiments.
This is just a description of what the environment looks like.
So I was trying to reduce the concurrency as much as possible to make it stable, all the things
were pinned to single core, disabled, hyper-threading, scaling error, like a turbo was disabled
of course.
And the only thing that was special in this case, it's that we were using a different
workload generation tool.
So this is, I mentioned this is a berserkered tool, we have created like a toy tool, I'm
about this a little bit later.
But essentially what happens is that the tool is generating a random process, which is
scientifically called a Poisson process, where essentially what happens is that we are firing
into C-scores a little bit randomly.
So not so regularly about a tight loop, but just a little bit with a delay in between.
To make it clear, there is a difference in the sense, so it's not obviously we do not
do like a synchronous C-scores or anything.
There is still one C-score and after another one.
But there were three workers, they were producing some amount of concurrency, although they
were still never going over the one tight loop in terms of performance.
But like essentially three workers, they are still producing Poisson distribution because
of activity.
Now on this diagram, what we are doing here is we are replacing this part essentially,
right?
So we are just literally doing a different workload type, where before we were doing this
tight loop, right now a worker, what it's doing is essentially just a firing C-score,
then weights a little bit around the random amount of time and then proceed forward.
Very silly idea why would you do this, right?
So now let's take a look at the results of this interesting benchmark.
So first of all, we need to establish a baseline.
For this test, the baseline was the amount of C-scores, this whole setup was produced.
Because obviously, the expectation would be that since we are doing the lace, literally
the lace between the C-scores for the worker, we are just doing less load and doing eventually
less C-scores.
And yeah, here we are, pretty much as expected, x-axis is time, y-axis is a C-score,
1,000 per second and yeah, clearly this randomized version I am going to call it later
on a berserker one and the tight loop I am going to call it the kernel like a vanilla
implementation.
And the berserker is producing less C-scores, I expect it all fine, so far so good.
Now let's take a look at our metrics from the benchmark.
Again, no surprises here so far.
First of all, we are not getting any drops yet, so events are not getting drops at all,
here and okay, we are producing less C-scores and yeah, obviously we are getting a little
bit more heat for the tight loop, so far so good as I said.
But now since I was mentioning this open versus close, let's try to take a look at the
latencies.
Now unfortunately it is a little bit hard to get, the point is that essentially a lot of
tooling that we have got so far is not allowing us to collect no histograms and all that stuff
so I had to patch with bf2 actually and what happens here is that we just profile the
bf program under the test and we on top of this with thanks to this patch, we just also
collecting not only cycles or how long how many cycles it will take to execute this
bf program but also histograms like those buckets within which those cycles are fallen
into and let's take a look at the results.
So this is a row data, y-axis is an amount of cycles that was spent within the program
and the y-axis is a frequency, so how often this result has appeared.
So you call or you see from the row data that suddenly there was a shift right, so this
run the mice data, do a producing less load into our system, strangely enough the latency
that latency is actually a little bit higher.
So on other edge our queries are actually a little bit slower than that and to make it
a little bit even more clear than before I have created a some sort of a probability
density where it's the same as you can see the berserker one is an origin fortunately
here whatever for what I originally know why, but you see it's shifted right now to the
right side, meaning that on average it takes a little bit longer time to execute one
operation like one run of this bf program within this context.
If you're a statistics freak I have to tell you it's not exactly probability density,
it's something like a derivative from the histograms just to be completely mathematically
correct, yes we care about it, okay so the summarized, we do less work load but we get
in slower execution, that's strange right, weird, the next example is going to be even
more weirder, so I was showing the environment before, and let's try to change one thing,
let's try to enable turbo back, so essentially what will happen is that we're just going
to get a little bit of support from the CPU, purely on the hardware basis for our execution,
things go completely, completely like mad, so first of all, send it to check, everything
so far so good, you see that we're executing now much more C-scals because we're getting
support from the hardware, but still the randomized version is performing poorly, we're
producing less C-scals in this case even more significantly less than before, but then
benchmark results look completely different, so here is it, the legend is the same, X axis is
a time, Y axis is a amount of records, a million per seconds here, and you can see first
and you can see is that randomized version for whatever reason produces more hits than the
title for the reason, it's a very strange situation where doing less work and yet a little
bit faster than before, even more, you can see that randomized version does not have any
event drops whatsoever where the title, producing a little bit less amount of hits, it also produces
the same level of event drops, so you probably can combine it in your head, so the title
version is actually producing more activity indeed, but how of this activity is a drop event,
so the effective work load here is like essentially housed unfortunately, yeah, so that's
how it looks like you might notice also this drop in the middle, it's a very interesting one,
I haven't really investigated it deeply unfortunately, but the results were actually not done in
parallels, so it was they were actually done asynchronously, like at different moment in time,
meaning that this drop happens still at the very same moment on the graph like close to the
middle, so probably it has something to do with the boost effects, but yeah, as I said,
I did not really investigated it deeper, what I did though was I've tried to investigate what the
hell is going on here, because it's a very interesting question, right? We just changed the workload a
little bit and then changed one configuration in our system, admittedly not necessarily production
way but still, it's a very reasonable change that you can apply and you could get a very strange result,
so what happens? We could try to figure out what happens using top-down approach. So fortunately,
nowadays, this just works out of the box, it's really magic, literally two years ago I had
to struggle with this but then there were a couple of fixes in Perf and since then everything works,
just as magic. I'm not going to explain what top-down approach is, you can just search for
this very powerful tool but essentially what it gives us, it tells us that most likely what we're
dealing with is a core-bound workload. In particular, we've got some amount of stole memory activity
and much more of that for the kernel side of things for the tight loop. Accidentally enough,
those are also the geographical coordinates of some point in the Polish forest, I don't know,
some are in the middle. So yeah, some arise again, we're doing less workload but in a different pattern
and we're getting completely wild results. So now a little bit about the tool I was talking about
and I was using for actually driving this benchmark, it's a bit surfer, it's a tool that we're using
for our project about like a security projects, tech rocks. Originally, it was pretty much a toy
project now, we're using it a little bit more actively but overall it's a very beginning,
so to manage your expectations, it's not something mind-blowing and what it does, it essentially
does what I was talking about. It creates very strange extreme workload that might potentially break
your program. In fact, not in terms of performance but we have caught a couple of buggers thanks to
this tool in our project. And yeah, so it creates essentially some amount of workloads like
process-based workloads, endpoint-based workloads, some unreliven c-scores, for example, unreliven
for your application, with a different distributions, with a Poisson, random uniform distribution,
and we're also doing a very interesting thing, user-space networking, it's a clever idea of one of
my colleagues, where essentially the point is that thanks to this user-space thing, we could also
first of all, we could save lots of performance and at the same time we could also seem late
sort of a real connectivity where connections are coming from all over the world and not from
the cluster where we test in it. And yeah, at least recently, landed a program, BPA programs
contention, I don't have any results from this but it sounds like a really cool idea, what will
happen if you have a program that's been attached somewhere, and then there are also like
hundreds of other BPA programs also attached to the very same trace point. So whether it's going to
affect your performance or not. And some other things that are like right now working progress.
So yeah, that's pretty much it, but I already see you asking about that. So why do you want to
come up with some custom tool for these type of things, right? Why not just use stress and g?
That's a robust question. And in my defense, I have to answer, probably the most straightforward
answer is just purely my ignorance. I did not know back in the days how awesome stress injury is,
it's really great. You can do a lot of stuff. So for example, the first benchmark I was showing,
you can really do this with stress injury. But at the same time, you have to keep in mind,
the stress injury was and is designed to exercise kernel interface, to stress test kernel interfaces.
Where what we would like to do is we would like to stress test our BPA programs. And sometimes
their goals, those goals are a little bit different and fortunately. And in this particular case,
it was actually a good idea that we have started with something custom, because then otherwise we
would need to go probably with iPad or a netbench or something like that to get like networking
performance. And even those two are doing this point to point performance. I haven't found any
single one that does this pretending to be from all over the world. So yeah, this is a
essentially an explanation. It's just a good idea to have something custom. And I think that's
pretty much it. So I hope you've got some amount of questions for me.
Yes, please?
Does the training install that you see? Does the BPA programs have like garbage collection?
Well, not the time of the world. I mean, it's a very actively developed area. Maybe it was
developed like why I was speaking, but not the time of the world. Why? Why are you resting?
We see that thing in the Java world all the time. Yeah. Yeah, okay. Yeah, obviously, but now I don't
really think that I just have something to do. You're talking about this one, right? This in the
middle, right? It's probably most likely have something you need to do with this like time frame.
You can get your boost over from the hardware because it looks like exactly this amount of
interval. So that's what I think it's going on here. There was some, yeah, please?
Yeah, it's not a public patch. It's something that I have hacked on my laptop. But I'm thinking
actually, there was some ideas about actually sort of extending BPA to with various cool metrics.
I was actually trying to do the same thing with BPF trace. And unfortunately, the overhead was so high
that you could see like we were doing 10,000 of hits with BPF trace trace and it was 2,000. So it was
just incomparable. And the only concern of mine is that actually you could do a lot of this stuff
in Perfor example, and I'm not sure what should be the distribution between those tools. And I don't
want to start any holy war between those. So that's why I'm trying to be careful about it.
Okay, great. Yeah, please.
So in this graph, you show brackets, units per second. Yeah, brackets to do a slow drop.
From the previous one, you were showing a number of system calls, right? Yep. You compare the number of
brackets between those two, like before you and I made the tools on all of them after.
Do you mean do you mean between those two configurations? Yeah.
Yeah. But before you and I go to a boy and after? Well, you could get this, those numbers from this
graph, right? So you're, okay. Yeah. So this is it. Yeah, there is always difference. You see?
Exactly. Yeah. That's why you get drawn. Well, essentially, yeah, essentially what happens is that,
yeah, I think I did not actually explain this, but yeah, the tight loop is overwhelming,
the green buffers infrastructure, of course. Yeah, then we're falling back on their work
correction, and this whole thing starts to be probably less efficient in terms of memory, of course.
And the first factor, I assume, like, when you, when you wait, right, you can be wrong,
you probably will do some sort of sleep. Yeah, it's a thread sleep. So, like, you don't
burn sleeping, you have to be wrong. Yeah. Yeah. Yeah. So you get, like, sort of, this
batching and relaxation, just because of the context, which is a two-car model.
Oh, I mean, yes, it may be the case, it may be the case. To be honest, now, I haven't thought
from this perspective, I need to take a look maybe, from this context as well. I see what you mean,
yeah. General speaking, the batch market we have is self-cast, right?
It was man to, like, just pass, kind of look this artificial. Mm-hmm.
Yeah, yeah, of course, of course. Yeah. I mean, and it's not clear, it's reasonable way to do this,
of course. And all the rest of the things are just, like, a silly, silly things to, you know,
to try to play around with this whole thing. Those, by the way, that if you're asking yourself,
five-sealer things, those are just the reference to what the tool is doing, five different
workloads, so nothing more. So, there were more questions I've seen, yeah, please?
Hey, so, this is, this is the smallest piece, and I think,
I just want to understand this kind of piece, so it's, why are you so defining,
kind of, does it also replicate where this people are doing this?
Well, I mean, it depends on what you mean by realistic, because again,
the distribution, this portion distribution, I was saying, it claimed to be a little bit more
realistic than anything else, because think about this. Imagine you've got to be a program
attached, for example, to a web server. And this web server is, in turn,
copyright-based on some customers triggering an API. And those customers arrive,
well, it's usually modeled by plus on distribution, so at the end of the day, just by pure
transitivity of those activities, it's quite easy to imagine that your BPS program is going to
be called in a similar fashion.