We welcome for the new people who joined Welcome to the Go Devroom again.
Our next talk is really interesting because we often talking about instruments in a
lot today but we never talked about to do it without actually changing a single line
of code.
So I have two amazing speakers on stage at the same time which is Hannah and Kemal which
are both going to talk about instrumentation.
A pass!
Okay, so yeah, we're going to talk about instrumentation.
We talk a lot about profiling that was cool.
Parker got a lot of mentions.
I used to maintain Parker so I'm glad that it's getting traction.
No, okay, it's good.
Okay, should I repeat myself?
Yeah, let's go.
Okay.
Let's take a look at that.
Okay.
So, cool, yesterday I was actually when I was walking thinking about this talk and like
zero code changes whatnot and I realized that maybe all the things we're going to
mention today, maybe they are not needed because in the future only the AI are going
to write the code.
We can just give it away and okay, just instrument our go applications and we can get
away with it.
But until that time, let's see what we can do today.
So first of all, why do we care?
Right?
So the observability promise you understand the whole system behavior, they buy production
issues and then prevent any other objects or like troubleshoot when they happen.
What happens is distributed complexity, partial visibility, you don't see all your like
metrics, pens or like any metrics that you are collecting and then okay, it's very
some my machine but just happen.
So how do you achieve this?
You need to pay the tax right?
Today we're going to strictly talk about or like our example is going to be about distributed
tracing, collecting the traces, whatnot, but technically this could apply any of the
signals that you collect from your processes.
So you import an SDK into your go application, initialize your tracer, grab your handlers
if this is an online HTTP based application, propagate context everywhere because if you
are now making an HTTP request from the HTTP client or like making a VBC code query and
you need to propagate all these contexts and then you need to shut down everything, gracefully
make sure everything flush to your observability system whatnot.
And now if you are like working for a big enterprise company and if you have like hundreds
of services, microservices, you need to do this again and again and again and again.
So where the instrumentation gets away, so it's one of the things that like let to open
telemetry to rise up.
You are like getting something from data dog, something from Neveralic or like other APM
vendors.
You put that in your code base and now it's your like the third vendor specific code and
then this is not specific to your vendor or like business logic but then it's inconsistent.
Maybe one of the teams are implemented something in their service and it exposes like
different attributes and the label, labels, the other service they don't have it, like
how do you ensure that is the case and like how do you ensure that you don't clutter
your code base right and then there's this anxiety of like I'm collecting all these data
signals and they're like useful and everything but then like what is the cost of it,
what is the overhead of like doing all these sort of instrumentation.
So we can't do anything about the last part or maybe we can but it's like harder to tweak
the performance part but what about like getting the toilet of the way, toilet of instrumentation
out of the way and directly provide the value of observability.
So that means like no SDK imports, no wrapping any handlers or functions, no context
propagation, we don't need to deal about it and then just observability just make it work.
So first of all like let's step aside and talk about instrumentation, what do you mean
by instrumentation?
You have your application, let's say then you have your like backend, your application
interacts with your backend or any other services, this could get easily complicated but
then what's happening in between, maybe you need to understand like what's happening
in the ingress point, what's happening in if I'm like calling a microservice about the
auto, what not, how do you do that?
So you can use logs, like it's the easiest, developers, loud em, right?
You can just put a log line, you see it on your local machine and push that to some service
but the problem is like yes, it's super convenient to add the logs but it's one of the
hardest signals to like store, search and make understand that, it's a challenging task.
Yes, then you have metrics, there's a reader, they're easier to collect but they're aggregated
data, right?
And they don't tell a story about your individual transactions, then there does the tracing
come from, right?
It's about transaction, you have like hierarchical data, you have the context propagation,
it's rich, you can derive metrics and logs from the traces, you can build a fat event system
based on it, it gives you more data but again, it's complicated to store, let's not
talk about the storage but it's complicated to instrument and collect these data.
So that's where the auto instrumentation comes in.
Cool.
So the point of our talk is about auto instrumentation so let's talk a little bit about
that but before we get into all of that we need to talk about all of the manual toy
all that comes into manual instrumentation that come all was talking about.
So let's say that we have this HTTP request handler that's just doing something coding.
Let's say I want to instrument this.
I need to add around 15 lines of code for every single handler that I'm having so every
health check that you have every endpoint that you have.
You've got to start a span, you've got to stop the span, you've got to set their attributes
somehow, it goes on and on, this doesn't include starting the tracer, everything like
that.
So what's the point of doing all this work?
I am lazy, I don't, I want to have all this data but I don't want to do it myself
and as Kamal said maybe AI will do this for me eventually but so far, haven't heard
of anything.
I need to do something in between having my application and actually getting data to
actually profit from having this application.
So this is exactly where auto instrumentation comes in.
Auto instrumentation as you can probably tell by the name is a way of instrumenting your
code without having to make any code changes.
And there are two different types that we're going to talk about today.
The first one is run time auto instrumentation which you can probably tell happens during
run time.
This is often a source of making code changes and if you know about go which is a compiled
language, we can't make source code changes.
So luckily for us in go there are other alternatives that don't require source code changes
and we're going to talk about that.
The other type is compile time instrumentation which happens at compile time surprisingly.
And this works a lot better for languages like go which are compiled because this will
never require require you to make source code changes.
Okay, so let's go more in depth into each of these approaches.
The first one for run time basically all of the approaches and go talk about EBPF.
And if you were at the EBPF Devroom yesterday, you've heard a lot about it and you probably
know but as a overview, it used things called hooks.
You probes are in the user space, K probes are in the kernel space and then there's also
static hooks called USTT and all of these hooks give you an opportunity to skip from
your application to some kind of tracing code.
There's another approach that's called library injection that takes advantage of LDPRLO
to basically do some weird magic that I don't understand to shove some code into your application
who knows.
So ignoring LDPRLO, let's talk about EBPF.
So as I mentioned earlier, it gives you a hook but when you have your basic application,
you're most likely going to do some communication with the kernel, whether you're in
user space or kernel space and as you're doing all of this work, that happens behind
the scenes.
EBPF gives you a hook that tells the code, hey, jump to this tracing code, make some spans,
make some stresses for me.
There's an example of EBPF Automate Instrumentation, this is an example of the OTAL auto instrumentation
for a go and it's very, very simple, all you have to do is create this config file.
There's basically nothing here about tracing, all you have to do is bring in the library
that they have and you don't have to make any source code changes.
This will give you all of your traces and this one takes advantage of Upros, we'll talk a
little bit more about this later, but goes to show how very simple this is.
The other EBPF or POSHA we see in go is called OBI.
This one is a library that was originally created by Bella but was donated to the open
telemetry community and this one also uses EBPF to trace your go code.
So a little bit more about go code or OBI.
It supports many different languages, not just go.
So if you are for some reason not using go, you can still probably use OBI and it also has
a lot of different coverage for different protocols that you want to use.
So HTTP, GRPC, etc.
The thing about EBPF that it's very important to know is that while you're accessing the
kernel, it's going to require you to give it a administrative privileges and root access,
which is something will delve into a little bit more later on, but this is very important
to keep in mind.
And the way the OBI works is pretty similar to what I mentioned previously.
You have your application in the kernel, there's a hook and after this hook, there's
a OBI side card that does all of the tracing for you.
And again, you don't have to make any source code changes.
Similar to the other EBPF application that we saw, this is also very, very easy to set up.
It doesn't require any source code changes, and you can even do this without having to
stop your application, which is kind of crazy.
All you have to do is tell it, what poor things being sent to, don't want traces, don't
want metrics, use promethias, etc., and as soon as you run this very fancy command with
or without your application still running, you get traces.
So pretty magical.
So that's it for now for runtime approaches.
Let's talk about the other side of things, which is compile time instrumentation.
There are two main things that we want to talk about today.
One is, or a cast-year-un, which I don't know if we mentioned this, but we both work at
data dog.
So our cast-year-un is one of our projects, and is in a compile time approach that takes
advantage of tool exec and go.
And open the telemetry compile time instrumentation, sig, is a special interest group that
is the collaboration between data dog, Ali Baba, and Otel.
This is basically going to be one big library to support all of your compile time instrumentation
needs.
This one is still a work in progress, but we're working hard on it, and hopefully it will
be available more widely soon.
So how does compile time instrumentation work?
As you all probably know, you start over for your code, a bunch of compiling things happen,
and then you end up with an executable at some point that is actually what is being run
when you run your application.
Inside the compiler, there are a bunch of different steps, but we're going to just look
at a few of them.
The first thing that happens is that your code is broken down into abstract syntax trees,
or ASTs, which are then turned into intermediate representations or IRs.
And what AST is very briefly, is just a tree that consists of a bunch of nodes that represent
your functions, your packages, your variables, et cetera.
Once you have your IR and your ASTs, that gets broken down into machine code, a bunch
of other steps, including linking, happen, and then you get an executable, very cool.
The thing that we want to focus in on here is the compile, or the AST stuff, and using
tool exec.
So specifically for orchestrion, we use the tool exec tool, and this is also what the Otel
compile time approach does.
Using tool exec to go into the compile time steps, trace through the entire AST, and edit
the nodes in the tree.
This is called abstract oriented code, and you basically give joint points, which points
to different nodes in the tree, and then an advice will tell you how to change the node.
This is not just data dog specific, we support the open telemetry standard, and it supports
a bunch of different packages and other dependencies that can happen automatically.
However, if you are like us, and you don't want to use the data dog libraries under
the hood, you can do something else, which is to create a config file, and what we did
for the purposes of this presentation is to edit orchestrion to instead of using our
data dog tracers to use Otel tracers.
This is an extremely simplified config file, because I had very limited space on this
slide, so if you try to copy this is probably not going to work, but basically what
it does, it tells the orchestrion library, what joint points you look at, so for example
the main function of my main package, and then what advice it does, which is to, at the beginning
of the function, start up a tracer, and then start a span.
So very simple stuff, and that means you're not limited to using data dog if you don't
want to.
As I mentioned previously, this uses tool exec, but of course if you don't want to do all
these fancy tool exec things, you can also just use the orchestrion command, and also
you can use environment variables if you so wish to, get things run.
This will edit your AST while your code is compiling, so by the time it turns into an executable
file, all the tracer tracing is injected, and your source code is untouched.
Okay, now that we've done all that, finally the meat of the presentation.
Okay, so the always one, like we have a bunch of other things that we are working on
to tackle the auto instrumentation things, we're going to talk about it, but like we really
wanted to see like what would be the effect of these things, like versus, okay, I have
the baseline application, then I have a manual instrumentation, then I use orchestrion and
injected some instrumentation, and then okay, I use OBI, like what are the trade-offs.
For that we basically wrote several applications, and then example applications, and set
it up with all the open source tooling, everything is available in a repo, we will share
the link in the end, we use a Docker-based observability stack, we come up with some
archetypes to actually generate the representative workloads, because like, okay, like
your app can be ideal, it can be IO1, CPU1d, or maybe it's just like a mix bag of that,
and we would like to see the all the differences and how it behaves.
We also collected CPU memory from the hosted self, because when you run an BPA agent
as your process, you need to actually consider both of their resources, not just the process
one, we try to collect those things, and then, yeah, as I've mentioned before, but these
are just HTTP simple applications and have some traces.
To actually focus on the effect of the app approaches, these are not doing a lot of
complicating stuff like making calls to the other services, so that you don't introduce
any other noises caused by the external services, it's just like pure spend creation
and trace creation in the end.
On the methodology of the benchmarking and how to make this kind of cleanly, we also
talk a lot about this in the software performance room this year, probably if you want
to watch that talk as well, you would know about the methodology whatnot, but like, you
can also check the repo, pull, and test yourself, right?
For that we have scenarios that we tested, default baseline, no instrumentation, manual
instrumentation, then the OBI itself, EBPF auto, like these projects eventually go on
a merge, but they are now like separate projects, and our question, because it's the
ready one, but eventually it will be an OTLC, we are calling it, open telemetry
compile time instrumentation, and again, we have a go application containers for each of these
scenarios, we have an OTL Collector, you can check everything with the EGR traces,
Prometheus metrics, and the Generate load with K6, but then we also have a simulation layer,
so when you have a request, we simulate like CPU one, the operation, IO one, the operation,
whatnot, and we use identical bare metal hardware to actually run these things, and then
we like sustain the load for eight minutes, so how do they compare?
All right, exciting.
Okay, so just to show off some data dog dashboards, when we make this public, we'll make
it available using Grafana, it's just that we're more familiar with data dog.
We did a few measurements, the first one being about latency of the requests, the throughput
of all of the requests coming in, I know this is really hard to see, so I'm just going to
go through this quickly, we have a summary, some CPU and memory metrics, and some host
metrics, again, for memory and CPU, so hopefully this is slightly easier to read if not a
little bit less crowded, though it kind of still is, but we're comparing the baseline, which
again, is the default instrumentation, or no instrumentation, rather, the manual instrumentation,
which is just using the Oto SDK, EBPF, and the tool train approach, which is still using
Oto under the hood, so shouldn't introduce too much noise. In the first column, you see, we
have CPU, and as expected for different approaches, we're actually not expected, sorry, we updated
these numbers up pretty recently, strangely enough, where's my pointer? No pointer. The EBPF
and the tool train approach seem to be using less CPUs surprisingly, and I'll do a little bit
more talking about this after I go through all of the columns, for memory, as expected, we're
using a lot more memory, or a little bit more memory for each approach compared to the baseline.
For latency, we're seeing that requests are going through quicker, and then as a result, the
throughput is higher than the baseline, so maybe not what you expected to see.
Evidently, I did not pay attention to Kamal and Augusta's talk about benchmarking,
but if you were there at the talk, you know there were a bunch of tips that you could be using,
and of course benchmarks are a little finicky, so this is just the latest results, but
let's, in addition to benchmarking, talk about who the winner is.
So, in addition to just peer numbers, we want to know how easy it is, and how
quote-unquote good each approach is. So, for this, we're just going to be looking at the
EBPF approach and the tool chain approach across four different aspects.
The first one is performance, and as you saw from the previous slides, we're using more
memory, so obviously there's going to be a little bit of overhead when you start instrumenting your
code, so a little bit of a trade-off if you want to get data. For stability, as I mentioned earlier,
EBPF requires you to use probes, which often require you to know offsets of all of your functions
in your variables in your code, which means that it's a little hard to use, and if you're
rerunning or rebuilding your code, sometimes things can break. On the other hand, for tool chain approaches,
all you really have to know is that your code is compiling, so it's a little bit more stable in
terms of like run to run. In terms of security, as I mentioned earlier, and I hope everyone
remember this, because I told you to remember this, the EBPF approach requires you to give a
administrative privileges to the library, which is a little scary. It's accessing your roots,
and it's not great if there are potentially dangerous people on your nutwork. The tool chain
approach doesn't do this, so it's a little safer. And then last of all, for portability or like
ease of use, EBPF is great, because for things like OBI, you don't even have to restart your
application, and you can just drop in the sidecar and then things start popping up. However,
it does require you to have a Linux kernel, so if you're using a MacBook, you're kind of out of luck.
But the tool chain approach, though, again, as long as your code is compiling, it works,
but it does require you to restart your application. So they have their pros and cons.
So the winner, I'm sorry, there's no clear winner. I'm kind of bidded everyone. It really depends.
If you want to use EBPF, that's great if you want flexibility in terms of use, but for
stability and security, protecting your code, compile time instrumentation might be the way to go for you.
And a little bit more about EBPF hooks and probes and things like that. Donia and her
coworker Chris had a fantastic talk yesterday about the gotchas of using hooks and probes.
And our coworker was somehow had another talk yesterday about the performance impacts and
other issues that we've had with EBPF. So if you're interested in learning a little bit more about this,
you can go check out their slides and their recordings.
Do you have time? Yeah. Yeah, we have enough time to talk about it. But again, as we told
talked before, we are still thinking about this. And we still try to find ways to push the boundaries
and really find a way to like auto instrument the go application, sorry. So one of the things
are available already in the other languages or the systems is having USDT. This is
statistically defined tracing points for the user space. And some of the languages, some of
the binary runtime, they already have this. You can enable them. This is basically injecting
couple of empty bytes to each function prologue and you can use these places to
either inject a library or like hook with the U-props system. What it gets into table,
the all the downsides of like or beyond needs to write right now deal with like calculating the
offsets where to find the binaries which memory to read and do this for each and every go version.
And most of the things are can easily break because if you temper with the go stack when it's
executing, if you go runtime would just panic and just you're at all that gets crashed.
But if we can put USTTs into go application, actually we can make these things stable and we can
benefit for the both of the world. So how do we do this right now? There's a library called
SELP, but it's not and it's using the sub-st, which is a native binary and it's basically
that generates what's required and link them in the runtime. But again, then you have those knobs,
there's no runtime to it. There's no execution costate and they are just like dormant and when you hook
some BPPF trace program or any other BPPF program and then you can utilize that event collecting
to data. I'm going to just give you our earlier running out of the time. And the same strategies
enabled like for the other runtime so I'm not. But again, this is this isn't working for like
latest goal versions. SELP is like out of data and this is basically also like dynamic time
library opening, whatnot, it's not exactly secure. The other way that you can also do is like
again the dark magic of injecting a third party native library and hook into the actual calls and
generate these spans. There is a framework for that called Frida and I also experimented with that
V also examples of the LIP-STP in the library but since they're like so unstable, we do
it and include them to the benchmarks. But there is a way that we can craft by supporting
these libraries whatnot. But actually one of the, this is another like approach to do this as well.
They are building their own quarks, they are building their own framework based on CGO and
extending to runtime but it's basically an injection framework. So but there's also another idea
that I want to pursue and I've been working on this for a while. So come like one of the things
that recently we, the goal runtime added is the flight recorder. And flight recorder is for like
getting traces out of the scheduler, GC whatnot, it's about the internals of the goal runtime itself.
But I thought like okay like if we can extend it and add some more like tracing points in the end
and then make it just like aggregate the data and stream out of the system and we can actually
use this for our purposes as well. And for that I'm working on a POC to see that if it's actually
can work and I got some results but it needs a lot of performance improvements that's why we also
that included that to the benchmarks. We also don't know if the goal runtime team
gonna agree with us. So there's a long way ahead for this proof of concept. And the second one,
the another POC that we are working on is injecting USTD probes directly to the goal compile time.
This is achievable. USTT is work like you just need to add another f-section to your binary
and then it they are discoverable and they are stable. And if you if we actually implement this
in the goal compile time and the tool chain we can have these probes and we can add these probes
to the standard library and this is there is no runtime and execution overhead of this and we can
just enable them for the Linux systems and then an EVPS system or an injection system can
hook into these tracing points that are defined. I come up with some tooling. I actually implemented
these. I have a proof of concept PR whatnot to discover these points and maybe generate
BFF trace to demo these things whatnot and like the API is definitely like looks like this right
USTD at a probe and then collect the data. Maybe capture some arguments whatnot. I'm still
working on it. It's not stable. That's why it's not made the cut for the benchmarks but I will
keep updating the repo that we're going to share. And yeah if you are interested you can check it
out and see how it goes. So instrumentation is helpful because of the ability, helpful.
Auto instrumentation is possible with a lot of trade-offs. You need to pick your battles. You can use
the BFF page approach but it's brittle. You can use compile time approach but then you need to change
how you build your application. Even it's minor. It's a change. You can still contribute all of these.
The open source CIGs are working. There's an OBI CIG and an open telemetry. There's a compile
time CIG that we are working on. There is the auto injector framework. You can extend auto
injector to include go and you can always contribute to the go time. If you are interested in any
of these subjects, try to discover and join these CIGs and say hi and let's start working on that.
With that, thanks for listening. Thank you. Thank you so much.