Hi, can you hear me?
Okay, cool.
Hi, I'm Anna.
I'm going to be your MC for the next couple of talks.
Next up, we have James Bellchamber, and the performance impact of auto instrumentation.
You can kick it up now.
Right, hey, everybody.
Yeah, I finished this about 10 minutes ago, so stick with me.
I didn't last year.
Last year, I did this thing where it was my first talk.
So I did, hey, for STEM.
And I'm going to do it again because it was really fun.
So hello, for STEM.
Woo!
And that's the sort of energy that you've got after a whole day,
and how many people were at Brightnight last night?
It was a couple.
Okay, so it wasn't just me.
Awesome.
And if I've got this much energy, and you guys weren't,
mostly weren't at Brightnight, then let's keep that up.
Right, so everyone.
My name's James.
I've spent the last year or so in the trenches of an observability
transformation.
So I thought I'd come and tell you about it a little bit.
So I'm an ops guy.
I've spent about 20 years now working up from fixing computers
door to door.
I was a Linux sys admin, then that became uncalled,
so I became a DevOps engineer.
Now I'm a platform engineer, I guess.
So everything's coming from this sort of perspective.
I'm now working with a great software developer who will be referenced in this talk a little bit
as well, so we've got that as well on that side as well.
You might know me from my talk last year, introducing observability to an airline.
So I'm still there.
Still plugging away.
Look at this guy.
It was so young and so happy.
So the stack that we built to support our auto instrumentation needs
has kind of evolved into a platform now.
And we've some pretty cool things since there.
So we have collectors in our base images that are ready to export
forward telemetry from them to our back end systems.
We've got pair team customization on that as well.
So we've got nice suite of metrics that are coming from no exposure
and windows exporter.
We do have some windows instances that we can use for systems monitoring.
Notably, by the way, just as an aside, no exposure if you're on AWS,
far better information than you're getting from AWS itself.
So use that if you can.
And we have everything as code as well, which is lovely, including our dashboards,
which have key metrics that have then did an updated for all services based on what they're using
and what we should be keeping an eye on.
Following sensible use, utilization, saturation,
saturation, saturation.
And errors and red requests, errors and duration.
Guidance for the infrastructure and application layers.
Again, this is customizable on a pertain basis.
And of course, a lot of this is driven by auto instrumentation.
Autom, am I allowed to swear?
That's auto instrumentation is fudging brilliant.
And who here is contributed to the auto instrumentation libraries?
I was hoping, as I said, that's fair enough.
These are cool, and wherever they are, I love them for doing this.
It's a made-up job so much easier.
They're basically there.
You hate this one.
Some people hate this one, I say this, but it's basically like pattern matching.
It reaches into applications.
It recognises common patterns, mostly functions in frameworks and like this standard libraries.
And they write sensible spans to go with that code.
So in your auto instrument, a web app, for example, you're not going to get everything.
You're not going to get like your custom logic.
But you're going to get the connection coming in.
You're going to get the response going out.
You're going to get like spring framework function calls.
You're going to get database connections and stuff.
You're going to see that all in your spans.
It's a ridiculous uplift, and it's basically free.
Basically free.
Now, some people that I worked with have pointed out that code needs to be executed.
And that uses CPU cycles.
These applications, which are being meticulously designed to be as performant as possible,
mostly written in Java.
So yeah, have you thought about the performance James?
And you know, they're right.
James, performance is impacted by auto instrumentation.
Luckily, this is something that I did some testing on.
And so for this talk, I fleshed out my knowledge.
I did some demos and stuff like this, and this is what I'm going to be sharing with you.
So before we go deep, let's do some basic testing on a couple of languages that we can currently auto instrument.
I would like to say that this is not me showing you internal testing with the client.
This is me using demonstrative testing that I can use to show what I've experienced.
So the stuff that you're going to see up here is not benchmarking.
You shouldn't be looking at what I'm showing you today as a benchmark of how these things perform.
But it does help us tell the story of what I've experienced over the last two years.
Let's start with Java.
So Java auto instrumentation is incredibly easy.
If you've got Java in your stack at the moment, auto instrumentate, it's so easy.
You download the agent and attach the agent, dash agent, and then you attach two environment variables and restart the application.
That's basically it.
And like most of these solutions, what it's going to do is start going to start pouring out open telemetry protocol on 4318.
I think by default, to local host.
And then if you want it to go somewhere else, then you can add another environment variable and stuff like that.
And we, in this instance, we did want it to go somewhere else.
So for all these testers, we're hosting the collector externally.
That just means that you'll see a little bit more network traffic, but you shouldn't see any other performance impact from the collector.
So test auto instrumentation.
I wrote a very simple spring boot application.
This is possibly my first Java application.
And it just responds to any request on slash hello with hello.
Obviously, this isn't quite it.
There is some other boilerplate.
It is Java.
I like Java, but you get the juice.
So let's start with the most basic test.
We're going to do a get a sleep, a get a sleep.
So we're running 10 virtual users.
We're running over 10 minutes.
And we're just going to get my IP address on slash hello and then sleep for a second.
We won't do this often, but this is just like the demo.
And this is what we're getting.
So although it looks significant here, like this kind of stuff here,
I want you to pay attention, I like load these as well.
I'm probably running straight out of the camera, but it's another.
You can see here like five milliseconds to 20 milliseconds on this.
You can see these bars over here.
This is all relative.
But this is all within margin of errors.
And so I'm just kind of showing this to you as an example of what we're doing.
We can see the manual instrumentation at the top.
This is average request for second.
This is the average P99.
And then you've got your P99 in your requests.
Errors don't come into it actually all that much.
So I didn't really need that.
That was probably a waste of space.
And as you can see, I'm not really seeing any difference between auto instrumentation.
Oh, manual instrumentation.
I asked Marnia who's the software person I'm working with.
It's a manual instrument this Java app that I wrote as well.
So we've got no instrumentation, auto instrumentation,
a manual instrumentation.
I also installed no new tool from the back end.
So we can see one second sleeps.
If you can't see these properly, I will upload them as well, don't worry.
So everything hovers around the same place though,
except for memory.
This is what you're seeing here is this is the weight of auto instrumentation on the RAM.
Of course, we're loading this agent in.
But these are, these are on AWS.
These are T3A micro.
So these are two cores with one gigabyte of RAM.
So, you know, there's not, again,
relatively not that much going on here.
And of course, you can see over in the network transmit there.
We're seeing more obviously because we're sending out the spans.
But pay attention, these are killer bits.
So these are these are tiny changes.
So essentially this is boring and this was just me using it to show you
like what we're doing today.
So we're going to remove the sweet now and we're just going to see what happens.
So this is just your simple spring boot.
Hello world application.
And as you can see, we've got something really weird going on.
So manual instrumentation is, this is not margin of error.
A manual instrumentation is performing.
We'd be a lot worse than auto instrumentation and none.
So auto instrumentation and no instrumentation seem to be performing that same.
But we seem to be getting this ridiculous performance impact
from manual instrumentation, manually instrumenting my little application.
Let's pin that before you all crucify me.
We will talk about that in a second.
And again, looking at the instance stats, we're not seeing all that much going on.
Again, weird manual instrumentation is using a whole lot more memory.
Otherwise, this is just a story of a busy server.
There's pin that.
Let's go and have a look at another language and see if we can get our bearings on this.
Go auto instrumentation is also pretty easy.
Technically, of course, it shouldn't be possible.
The way that auto instrumentation works is it tends to attach to a virtual machine
or an interpreter and it adds stuff.
And how do you add stuff to a completely compiled application?
Well, you use EBPF.
EBPF is really cool.
I'm not really going to talk about that too much here.
But it's very much working progress this at the moment.
I think they introduced a beta release.
Is that true?
Yeah, a couple of days ago.
So do look at go auto instrumentation.
Again, you compile it.
You do have to compile it currently.
Upload it to your server.
The way that it works is that it runs as a separate application.
And it looks at the application utility.
Which is different.
So again, gin application.
I asked my new to write this again, because I don't know go.
We'll see how this performs.
So again, sleep one seconds.
We're not really seeing much.
These are the reasonable gaps.
This is just margin of error.
And with sleep one second on use.
Again, we're not seeing crazy amounts of differences between it.
So let's go straight to no sleep.
That's the interesting bit.
And this is more like what I would expect to see than when we saw
which are the manual instrumentation.
Is that we would see actual differences.
Sometimes significant, sometimes not.
Again, this isn't crazy significant.
In fact, it's the auto instrumentation that's performing the worst.
The worst here.
We're looking at five milliseconds versus ten milliseconds.
That's not crazy big gaps.
So yeah, we're seeing a significant drop off in a request per second as well.
And again, we're seeing server busy.
We are again seeing auto instrumentation using more memory than manual instrumentation.
But these are for relatively small differences.
Okay, that was a lot of taking screenshots of Grafana and I'm passing them off as slides.
So let's go and look at some lessons that we can learn so far from the talk.
So we definitely, 100% know that they were right to suspect that code,
upon being executed, does use resources.
We also know that manual instrumentation uses resources.
Okay, I'll get to the serious insights.
One thing I didn't tell you is that Mario being a great developer is not a Java developer.
Another thing I didn't tell you is that he did this incredibly quickly.
So we basically just got this application working and we just left it there and we started doing testing.
Money is a go developer.
So the manual instrumentation of our little go application,
although implemented just as quickly, we're dripping with her previous experience.
Like, you know, she knew how to write go applications.
And so this is the old adage, right?
It's like how assembly can be more performant than Python.
And that's true, but can your assembly be more performant than your Python?
We did actually go back and rewrite the Java code.
And I did this mostly because I didn't want anyone that was working on that project to think that I'd miss represented them.
And as you can see, once we worked out what was going wrong, that we rerun the tests,
and now auto instrumentation is performing about the same as manual instrumentation.
But it was a great example of how manually instrumenting an application can catch you out,
just like with any software development.
This is the instance that's for posterity.
Again, I'll put this in the slides that I'll upload in a bit.
And you can see how now manual instrumentation is slotting in right between auto and none,
which is the kind of thing we'd expect.
So auto instrumentation by comparison is much more consistent.
You definitely can squeeze more performance out of manually instrumenting an application.
But honestly, even if you can, at the expense of maybe getting caught out and bringing out bugs and stuff like that, it might not matter.
What we might have built there was possibly one of the most simple applications that ever served a HTTP request.
I'd wager that more resources were spent on creating and shipping a span for each request than actually serving the request itself.
So what we've explored here is certainly interesting.
It's given us some basic orientation on the performance of auto instrumentation.
But I think it's time for us to start looking at the performance of real applications.
I did say real applications, but I'm going to start with Peclinic.
Peclinic is a, you've heard of Peclinic.
I heard a couple of giggles, so giggles, laughs, manly laughs.
Peclinic is an example application written in spring.
It's a bit clunky, a bit slow, it has cute pets.
I mean, it's basically a sample application that allows you to learn the tooling, right?
And it's kind of a pretend CRM system for that.
So we're going to instrument this.
We're going to script some interactions this time.
I'm not just going to do the one thing.
I think I did like 17 different requests across the system.
Searching, browsing, and form submissions.
We'll see how we do.
Now, the first thing I know is that I thought it was a fairly lightweight test.
But again, we've got this very simple server,
and we've got this Java application that can't really handle that much.
It's not designed to be performing.
And 20 virtual users working through 17 different interactions.
It spruggled, as you can see.
Errors, bars started actually coming up here.
So yeah, auto instrumentation.
Is also instrumentation making a difference to this application?
No.
324 versus 326 requests.
This is, I mean, the application that's under a huge amount of load.
But we're not seeing the kind of gaps that we were seeing before.
But maybe that was just because I put it under huge amount of load.
The reality, and this is, by the way, the instance side.
The only difference here really is in memory.
It's worth saying that pet clinic uses an in-memory database,
and I only gave a gigabyte of RAM.
So it didn't quite run out, but it was struggling.
There's no other pressure on this, because let's be fair,
even if this was a real CRM for that,
you'd have maybe three users making five requests a day.
So this is one virtual user, like, and this is our red testing.
And even then, like, when we're taking off the pressure,
what you can see here is, and this is what I've seen in reality.
You know, the measurable differences from our basic testing,
from our performance testing, from our, you know, all of that stuff.
They've gone here.
We're actually seeing the auto instrumentation has very negligible impact.
And again, on the back end, again, RAM seems to be the only thing I ever see
is that there's also an instrumentation, a little bit more RAM.
Again, we've got, that's like 60 megabytes,
as I could get to play with.
And so this brings us to lesson six.
An application that is actually doing something,
more significant than just serving a simple response on an end point,
is probably only going to be negligibly impacted by auto instrumentation.
Or, really, indeed, manual instrumentation that's been done well.
Going to go through a couple more, and then we'll move on.
Let's pick something that's real that's doing less.
Do you know that there's a module you can add to HDTPD that starts pouring out spans?
It's good.
And, of course, it should be these, like a C++ application.
So that's really cool.
I don't know if it's auto instrumentation.
I guess it's just instrumentation.
But it one pass at this quickly turns out that I had to turn the errors.
It's going to hit the just it works page.
You know, you load it up to HDTPD and does it works?
That's a 403.
Who knew?
And yeah.
Oh.
So this is HDTPD just responding with the simple, you know, HDML page, it works.
And as you can see that we're adding an order of magnitude more on top of that in terms of response times.
And we can do three times the three put with that auto instrumentation versus with auto instrumentation.
And at the back end, you know, you know, the disk was much more expressive.
Interestingly, I think that I could do a lot of tuning here because I think that it was writing a bunch of logs that it didn't need to write.
They weren't errors.
That's just by default what it was putting there.
And this only had eight gigs of disk space.
So I suspect there's tuning as possible.
But this kind of represents the fact that you should really be thinking about what your applications are doing.
Because if you've got HDTPD instance there, that's just meant to be very quickly responding to static requests.
You know, that's an example of something where you might see an impact.
But this is not how people use HDTPD, is it?
People use HDTPD to host WordPress.
And then I'll give you with each other.
Let's put WordPress behind this and try again.
You can see here that I ran WordPress with and without.
I could have chosen been in labels for this with and without HDTPD instruments.
And then I instrumented WordPress itself with PHP based auto instrumentation.
The rare double auto.
And as you can see here, the big gap that we saw between HDTPD being auto instrumented and not auto instrumented.
That basically disappears here.
And actually it's the PHP based auto instrumentation that you start seeing having a problem.
Not significant.
I'd say that I'd still be happy with that personally because I think that it's worth it.
We'll talk about that in a second.
I'm not sure any sure what's happening here.
But I won't note that for each request to load the sample page on WordPress, it created 35 spans.
Twenty-two of those spans were database queries.
This is for this single sample page.
I don't understand WordPress.
But yeah, otherwise everything looks pretty much within margin of error.
And honestly, that's not, I've done WordPress hosting.
And that's not what people worry about, right?
When you're doing WordPress hosting, you're thinking about your caching efficiency.
So again, you see that performance impact and you go, oh no, big performance impact.
And then you think about your application.
Well, actually most of your WordPress applications, yep, ten minutes, oh, I'll speed up.
Most of your WordPress applications come from caching, you're looking at caching efficiency for your performance impact.
But it does underline the need for good performance testing, which you should be doing anyway.
But when someone asks me, what will the performance impact of auto instrumentation be?
My answer is, my answer is well tested.
Because it does depend on your application.
And most of these auto instrumentation stuff, it's been pretty easy to ensure.
So the harder ones were the HTTP module, had to spend a bit at the time on that.
And WordPress, because WordPress doesn't install with composer.
And the auto instrumentation installs the composer, but you'll work it out.
But yeah, the little bits do add up.
So yeah, get into performance testing of applications.
And now is a great time to start.
Okay, so far I've been relying on you, dear audience, to believe me when I say, hey, all of this is worth it.
I would like to leave a little bit on why it's worth it.
Firstly, looking at applications in terms of traces.
Even if you have the most punctory span, and most of your application is just custom code.
You're still going to get the request span.
You're still going to get things like the database course.
This is actually an example that we had from real application with our client.
This isn't a real trace.
We're going to throw on a tempo doesn't output this.
And this is a representation of the problem that we had.
We had an application that had a huge amount of custom logic.
And we were basically not seeing anything, even with auto instrumentation.
But we did know that one of the things that we were being told is that the application that we were running that they had developed was slow
because our database wasn't performing as well.
We were hosting the database and our database wasn't performing well enough.
And of course databases never perform perfectly.
So of course the database engineers went off and they tried improving the response times in the database.
And they could see that there were some slow queries and things like that.
But from the traces we could see that when we got very long requests that were causing us problems,
we were getting very short database queries.
So we were able to flip the scripts on that problem and to be able to say,
Hey, yeah, we know that there are slow queries, but those slow queries are not connected to the slow requests in your application.
And this was because it was a Java application so we could just auto instrumentate.
It was incredibly easy.
The other thing is automated metrics.
A lot of, especially older applications aren't outputting the metrics that you need to be able to do proper red dashboards.
And even if they do, they might be wrong.
With auto instrumentation, you can pull all of this out and we could create dashboards that upper management had been asking maybe for years.
And they were not being put ahead of, you know, feature requests and stuff.
It was very easy with tracing to go and generate metrics from that.
There's also things like the service graph, I'll get through this very quickly.
You've seen this in other things, but I never really understood the value of this because I was just like,
Well, it shows you how your application works, but you know that, right?
But of course, with a lot of enterprises going back to maybe 60, 70s, 80s, 90s, you know,
and they've got stuff that was developed that's out developed maybe 10, 20 years ago.
And only very few people actually understand what's going on.
And to stay understanding what's going on, they have to work very broadly instead of being to get deep on stuff.
Traces just give you this map of everything and correct everyone's understanding really, really quickly.
And even when you've got applications that are not instrumented,
where you can't auto instrument and you're not able to go and manual instrument,
there was somebody in the last year that was doing cobalt.
Does anyone here do cobalt this year?
Yeah.
What's the point, yes, someone else?
You're grossing on them, you're grossing on them for using cobalt.
And so yeah, like, you know, there's no instrumentation library of cobalt.
So far as I know, but you can inject the context for the trace into the application,
which means that you can go, oh, it went off to this application.
I can't see a traces for that.
But I can click a button and see all the log lines that match it.
God, that's useful.
And as we've seen from our experiments, the performance here is not really apparent at the scale of whatever most applications are doing.
You should be doing performance testing.
But if you don't have any of the things that I just listed across your whole estate,
you should be leaning towards auto instrumentation.
You should be leaning towards picking up these libraries,
testing them on your systems, implementing them and getting them out there,
because there's not much of a reason not to, it's free of observability.
And it's so useful.
I do want to talk about manual instrumentation for a moment,
because you might be thinking that I'm telling here,
don't do manual instrumentation, only do auto instrumentation.
It is true that manual instrumentation code is a ton of work.
This, for example, is a simple application, and all the bits in yellow,
a bits that you need to do to instrument your application manually.
Of course, this is unfair, as an example,
because of the more complexity you have to application,
the more code versus other stuff that you need.
So the balance would change.
But this is still a ton of work,
especially on instrumenting existing applications.
Doing it for new code is really, really easy.
You should just do it.
Just start doing it.
The other problem is that humans, of course, create bugs.
They also fix bugs.
They also create bugs.
Auto instrumentation has already been written.
It's been tested in a lot of places already,
and the bugs have been worked out.
To varying degrees, some are more further along than others.
And of course, because it's open source,
you might find it easier to fix the few problems that you're having
in your auto instrumentation than you would
in debugging your application and getting it working
with manual instrumentation, which, of course, is the beauty of open source.
And the other thing is that open source,
auto instrumentation, you hate this slide, don't you?
I'm not big, I'm found a better way of explaining it than this, frankly.
The reality is also instrumentation is awesome.
It's fudging awesome.
But it's also a reasonably dumb.
It's essentially doing pattern matching under the hood.
It's finding things and decorating them and stuff.
Those are people who are going to come up to me afterwards,
and be like, it's not that incorrect to me, please do.
But it's guessing what attributes should be and shouldn't be added,
and of course, that's not going to match your application.
And so how well it works is going to depend on whether you're mostly
writing custom logic and custom code or whether you're mostly
averaging applications that have already been also instrumented.
So it's kind of like the bus that you're going to get almost to your destination
while you're still working on that custom rocket engine that's in your garage.
The rocket car isn't ready yet,
and the walk from the bus stop isn't that far.
That sounded so much better when I wrote it down.
And with auto instrumentation, you can start your observability journey now,
instead of waiting.
Remember, observability isn't tracing.
Observability is a different way of working.
Instead of your telemetry being something that you design,
and then to answer no question.
Instead of telemetry being something that you design into answer
known questions, and then implement as part of your software development,
telemetry becomes something that you implement to expose the internal state of the system.
And then you can start asking questions of it outside of that loop.
You really want to get to this for two reasons.
One, it's going to be a bionic arm for your troubleshooting.
But secondly, it means that everyone else has got this capability as well.
They don't have to ask you what's going on in your application.
They don't have to ask you to implement that new counter.
You can just say, no, no, no, no.
The traces are there.
We've made sure that the attributes are all there,
and you know, just go and pick it out.
Right, have I got time for miscellaneous?
Okay, really, really quick.
Okay, firstly, AWS Lander.
AWS Lambda works.
So, if you're using that, the only thing is that by default
the AWS version distribution of open telemetry
includes the collector in the Lambda layer.
If anyone works on landers, you'll know that one of the things you want to do
is you want them to start up and complete very, very quickly
and having a whole collector start and then stop again
as a huge amount of weight to that.
I would recommend adding an external collector
and having it send to an external collector again.
Your mileage may vary, but this is something that's really important
when it comes to performance of auto instrumentation.
The other thing is graphite of bailer.
Well, I'd say generally EBPF.
EBPF is awesome, as we could see with go,
it was your only option.
We'll see, see, plus, plus with rust.
It's going to be your only option for getting traces out.
But it gets you less unless you configure it.
It might change as we go on, but at the moment, if you use EBPF based
also instrumentation, it will give you less in the way of spans.
It will give you less context.
Someone will come up again afterwards and correct me on that
and I'd love to see it.
But my experience is that it's giving you less,
and also it's about as performance, maybe a little bit less
sometimes, maybe a little bit more.
So my approach is still to use traditional auto instrumentation
where you can use it and to use EBPF where you can't.
Again, I've got the, this is just for posterity.
You'll see this afterwards.
So yeah, these are our lessons.
You should probably be using auto instrumentation.
Time is up.
You should probably be using auto instrumentation.
So just say at the end there, we are looking for people
who community led projects that want to leverage this kind of stuff.
We benefited a huge amount from this, so we want to give back.
So if you're running a community led project, open source projects,
come talk to us, we'd be really happy to help out because it helps us out
with some experiences as well.
So maybe we can give one minute if somebody has a question.
Is there any questions that I perfectly answer?
There you go.
Okay. Thank you for the presentation.
So in my experience, auto instrumentation tends to create a huge volume
of traces, matrix, logs, which in the end cause costs for the storage of those
that are sometimes one order of magnitude higher than the cost of running the application itself.
So how do you handle, like you should, what are the strategies to reduce the costs?
Okay, strategies for reducing cost.
How much do I want people to hate me?
Okay, don't use that dog.
So what we're using at the moment is we're using Grafana tempo, not an ad,
but it sends it straight to S3, so you get this ability to do all the things you can do with S3
to store this stuff and you can use any other back end as well.
Storage of spans doesn't have to be expensive.
I think that's just a choice that the industry may.
But the other thing is, yeah, once you've gotten to the case where you're spending
we're not, we're just not spending that much.
I think, you know, I'm not going to quote a number, but we're spending like, like,
not point, not one X.
What we're spending on running the applications and we're using auto instrumentation.
So, and all I can tell you is that the way that we've done that, the strategy was
saving money by using a tool like Grafana tempo, which pushes it to S3,
which gives us those economies of S3 rather than the economies of maybe something like,
you know, data dog as an example.
And very specifically, I do call that data dog because many people have come to me and said,
we love this idea, but we don't do it because it's so expensive and I find out that the reason
is that they're using data dog, if you're listening data dog, lower your prices.
Thank you, James.
I think that's it.