Thank you.
Thank you, folks.
Good morning.
Thank you for being here so early on Sunday.
This is maybe it was hard for you.
It was not that hard for me with this a bit.
So here we're here to talk about open telemetry instrumentation.
I was supposed to be giving this talk with my friend Israel, but he cannot come to the
maybe it's because he went viral a couple of days ago when he got a job offer or from a recruiter
asking for three years of experience in deep seek, which is pretty weird.
So now he's famous and he cannot come.
But yeah, and you are stuck with me and I'm also, I work now as a product leader with
telemetrics.
I guess you didn't know there is a cloud product and I also work at the university teaching physics
to some vet students in Madrid in Spain and yeah, I have a past like everybody does.
And I also love playing heavy metal and seemed racing so if you do these things and you
want to talk about them later, there will be super cool as well.
So we are here to talk about instrumentation and open telemetry.
So you will be hearing this a lot, so I will try to make it super quick.
So open telemetry is this project, the superheets and gives you APIs, SDKs and tools to understand
or create observability and this means metric flux and traces.
So things start to get a bit complicated and overwhelming because this means that these
APIs as the case and tools need to produce or ingest and show metric flux and traces.
And if you think about it, then you need to implement this in a lot of languages.
So it's like, okay, now it's like this is super overwhelming.
The good part is that open telemetry said, hey, or this super clever guy said, okay, let's
establish a protocol.
So we don't care about what happens in these tools and we have interoperability.
So the protocol helps to abstract that and, okay, I don't care, with language, you
are programming your tool or mine, they just talk it's other.
And also it unifies in the protocol, metric flux and traces.
So you don't need a separate protocol for each of them, which is really convenient.
And it also provides instrumentation libraries and out instrumentation for the code.
So you don't need to think about it or put a lot of different libraries when you go to
the code.
So what is this thing about instrumentation?
Let's start with zero code because it's fancy.
And so it's something that tells you effortless helps you to create metric flux and traces
in your code.
This looks super cool and it's great for your managers and because then you don't need
to spend money doing this, but it has some inconveniences, as well as we will see.
It's agent driven.
This means that just imagine you are in Kubernetes, you would have an operator that
will inject or activate some observability signals and in your code.
And how is it done?
And in three different mechanisms mostly, this monkey patching or byte code manipulations
if we are talking about Java or a BFF, right?
Like I'm calling the kernel, sorry, there is an event in the kernel and then I say,
you want to send a message and then I produce this metric or this span.
But there are two super interesting calls later to talk later today about these topics.
So I will let them talk more about that.
I will be focused more on manual instrumentation.
However, if you are curious about how this monkey patching works, for example, you can
look at Wikipedia, super simple, just imagine you are using this.
This is an example before AI, just imagine you are using a Python library and you want to
say, hey, what's the number pi, right?
What's 3.14?
But then what you can do is, okay, at runtime, I change the behavior of this library.
So at runtime, I change the value and now it's like 3.2.
The same as it happens with sensitivity, by the way, but then it doesn't change the library itself.
So open telemetry or this mechanism helps you to, you have your library that is behaving
in one way, but also you can at runtime change the behavior.
So it starts sending metric locks and traces.
So that's the principle.
And you do it by configuring libraries and exporters.
For example, I want to generate metrics and send them to this endpoint.
I want to generate.
So this is what you will be doing when you do serial code instrumentation.
So there's nothing super fancy for the user.
So it's pretty convenient and it's great for edges and libraries.
What I mean by this is you have your code, you will be, for example, calling your database.
And when, and you want to understand what's going on there.
So the guys, super clever, of that database, they created some instrumentation for you.
So you can activate or deactivate.
So you can understand if you have issues in your code or in the database communication itself.
So this is super good, but at the end of the day, not every language support that.
I come from the C++ world and this doesn't work there.
It's also really hard to control the data because you can be generating a lot of signaling.
And if you don't treat it properly, then you can have a lot of increasing costs.
And at the end of the day, we want to understand what's going on in our code.
We want to understand if it's failing because of us and you need to do something.
So let's talk about manual instrumentation.
That's why we are here.
So when you start to include your open telemetry libraries inside your code base, it's kind of hard.
So actually, that there was this survey a couple of months ago, months ago,
that people were asking the community, hey, we need more examples.
We need the documentation to be more thorough.
So actually, this is why this talk is because when I was using to instrument my code,
I was using an open telemetry, I had some doubts.
So the first thing you need to understand, so now this is what we're going to talk about,
how is this structure, how you can do it.
The cool thing as every other library, it provides an API and an SDK.
So the cool part is that it's the couple.
So you can abstract yourself from, hey, where I'm sending my data.
I don't carry my code, right?
When I am processing my request, I don't really care where I'm sending my metrics,
where I just want to don't pollute my code.
So this is one of the principles I have my API in which I say, hey, this is my plus one to this metric,
but I don't care about the rest and then you have the SDK.
So let's start discussing how you do this for metrics.
And we will also do it for traces and logs.
Formatrix, I say what's in.
You normally have your function in which you want to add an error.
So at the end of the day, what you want to do is say, okay, I have one more error,
and then I don't care, I just throw it and someone else will pick it up.
So you do this by calling your API and when you start really in the documentation,
it can be, as every documentation, it can be boring.
And that's why I'm trying to put here some fancy pictures, maybe it's more clear.
So you have these concepts that are measurements, instruments,
meter, meter provider, and you have, okay, but I just want to do like plus plus my metric, right?
My measurement.
And then you also have the SDK which provides you and reader and a supporter and what is that?
So this is one I'm trying to explain now.
So they tell me, okay, we have a meter provider.
When I want to create a measurement, I need to call a meter provider,
and a meter provider is this thing that will give me a meter.
And what is a meter?
So it's my interface, the meter provider.
The meter is something that tells me which resource is generating the metrics.
So it's bound, for example, to a library, right?
In this case, if I'm using a MongoDB library,
it will be a meter with this name, right?
And then I can ask for this meter for an instrument.
What is an instrument?
In metrics, you can have different set of kinds of metrics of instruments,
so a counter or a histogram, so that's a meter.
So then I'm done, because now I have my counter, right?
My instrument, and I can tell, okay, I have one more error, and I push it, right?
So we are now done with our goal.
We know how to say, okay, give me one more error,
or I want to report that I had an error.
Well, just we just need one more thing, which is, if we go just to real code,
what you will be having is you create, well, this is a plus, of course,
so lines can get long, but you create a set of labels, hopefully,
you know, with no high cardinality, and you will say, okay,
this ID, or this response code, and then I added to my labels to my metric.
And I also, the number, I just say, okay, just have one more, but you can put them, right?
So this is it, right?
So now these concepts, coming again, I have a measurement, which is what I want to add,
and I added to an instrument, which is my counter.
And then the other things are for the API.
So what's, okay, so that's cool, but what happens after that?
Because I was wondering, like, okay, I just throw this metric and now what, right?
Well, it gets a stored, you know, into an aggregation store, into a local
a storage, and then the cool thing, and this is what I love about this project,
is that it says, okay, now you define a reader, and an expert.
So from this reader, you are able to say, okay, every second, every five seconds,
every half a second, you say, now you get this data, you aggregated, and you send it out.
Where, in this case, okay, to an open-tylometric collector, but wait for it,
because you can put another reader and say, you know what, leave this metric here,
because I promise you guys will come and take them.
So now, I don't really care if I, in my code, if I decided that I would use a
permission, sorry, if I would use open-tylometric forever, whatever, because now I'm done,
and how you define this, you define this, for example, in a separate file, or part in your code,
you will be doing a single tone, or whatever, and this is how you do it.
So super quick, here, I'm showing you how you define a reader, you know, like,
this, every second, you get the metrics and you take them out,
and you will have some, define some parameters that you can do via configuration files.
For example, hey, I have my open-tylometric collector, or my backend there, so do you define the
exporter, so this is like configuration file, and yeah, a lot of code, and you know, to make things work,
so it's a single tone, but that's it, right? So, and of course, I just, this one was an example
for the open-tylometric exporter, but you will have the similar function for promises,
if you are, if you want both in my example. So, this is it, right? I have a measurement,
I will throw it to my instrument, and that's part of the API, and then there will be a store,
and sense, whatever I want, period, I'm done. So now we know how to instrument our code
with open-tylometric, in C++, well, no, it's not only C++, because the cool thing about open-tylometric
is that it gives you this, for it's an specification for every language, so every language
that implements open-tylometric libraries need to do this. Of course, you will have nuances,
but that's it. So now you know how to instrument in every language in the world, it's pretty cool.
Now, traces, we are done with methods, let's go with traces. Normally,
what I see is that everybody thinks they know traces, but then they have some misconceptions,
so if you allow me, I will do a super quick introduction to traces, because normally,
what I see is people thinking, no, this is just a low level log debug, and then I'm done,
well, no, this is a different thing at this in this notation. So, what is a trace?
Where will we meet for a second? It's just some spans I put together under a trace ID.
Nothing else, nothing fancy, spans under the same trace ID. And what is a span?
It's a representation of an operation, and it has, because every operation happens in time,
I have this time dimension, which is pretty cool, because it will help me in the end. So,
and why are this important? Because when I have a distributed system, and a guy is doing some
stuff in JavaScript, that goes to a go API, and then goes to my database, and then something fails,
then I cannot save my day by doing QCTL logs, because then you are like, okay, what do I do with all of this, right?
So, if you have a span, you can track, if you have a trace, you can track, you can trace your call,
and with this span, it's like, okay, this service call, this function, that's a span,
and then I have this other function, that's another span, and then I have all the conflicts.
So, this is this distributed tracing, okay, and at the end of the day, as I was saying,
you can, you can take it in a gunster, which is a normal visualization.
Now, let's get to instrumenting fun. Now, we are more experts, so maybe we don't need to go deep
dive into all of them, but this is even simpler, so I started a spoiler error, I started with the
most difficult one. Now, tracing is even simpler, because there are no different instruments,
I was saying, okay, you have metry counters, or histograms, there are different, here we have a span,
there's no different sets of span, you have a span, and that's it. So, our API gives that
a trace provider, a tracer, and the span, so I will be reporting a span, and they will go to the
SDK to export it. So, it's the same kind of representation, and let's start from, from the end
this time, this is just, okay, I will be defining my resource, this is an important concept in
open telemetry, because I tell, hey, this instrumentation, or this signaling is coming from this
resource, in this case my library, or my program, so I have that, so at the end, when I want to
take the data, I can understand where this is coming from, so I put my resource, I also say my,
where I want to explore my data, like, okay, I have an open telemetry collector run there,
and some options like batching, for example, I have processing source, I can process my data,
so I can say, okay, I can say, let's send it every, you know, every megabyte,
I don't want to have a lot of signaling, right? Another thing I, so that's for the SDK,
another thing I want to recommend mostly in C++, because how it works,
is to create this kind of auxiliary function that will help you to create child response,
because what we want to do is to propagate context, don't forget that, so I want to have a function
in which I say, okay, create me a span out of this one, so it's just propagating the context,
nothing fancy, but it will help you, because when you are in the real world, and you say, okay,
I'm going to send an HTTP request, okay, this is our example, let's go with that.
I want to send an HTTP request, so hey, let's start my span, and I will set some attributes,
which attributes, how do you decide, how to call this attribute, how are they going to function in the end?
Well, that's a great question, because the open telemetry project defines where they are called
semantic conventions, and there is a semantic convention for a lot of things,
you can say, I don't know, like for CICD or function as a service, or for example HTTP,
which is our case, so by using the open telemetry library, you will have all these things
already in your code, because you know, okay, I am sending a request that is going to this,
you are, how do I put this in a span, so I can take it in the end, so it's so easy, okay,
you just use this, you adjust it to the span, you just use the primitives from the library,
and you are done, you don't need to overthink or redo, so it's pretty cool. After that, so I added,
write my URL to my span, I send my request, and what I do now is say, okay, my counter, remember,
we had our counter, okay, plus one, I sent a request, so that's another counter, it's not another,
and I add an event, which is another very important concept for traces and spans,
is not only about having this, this one called this one, no, you can add your logs,
we all love to say, hey, I was here, hey, this fail, hey, I don't like this, if I am here,
we are all screwed, right, this kind of things we want to add them, so these are events in this jargon,
and yeah, so, okay, I send my, this is just continuing with your sample, I send my request,
when I get my response, I will be adding another event, hey, I also, you know, in a super short demo,
but then I can also validate my answer, and if it went back, I can also add another event,
or set my span as this span was an error, so I can take in the end, the trace, just, and we are
done, however, we just need one more thing, if you are wondering how to help this propagates,
well, because you need to propagate the context, this service at the other side needs to
understand your span and your library and things, so how does this travel, it travels, for example,
in HTTP, it travels via headers, so, but then you need to invent them, no, this is standardized,
and actually these are the headers, and you would just just do it your hint, so the library
will handle this for you, in C++, if you want to do it yourself, you know, like, it's super big,
and everything, but it's just a couple of functions that you will say, the concept is super simple,
is hey, I have this request coming from HTTP, please inject them in this span, or I am sending
a request, okay, let's take from my context, you know, the library, the primitive, and put them
into HTTP headers, and I'm done, so that's the only part we were left, so yeah, we can do this super
quick, just to show how, like, why is this important and what, how does it look like, I have here,
I have here a client that is sending 100 requests per second to a dummy server, and so let's give it
sometimes, so it generates data, and here we go, and let's refer us, and at the end of the day,
what I'm going to look is to some spans, you know, like a trace, and I can take them, and why
all of these that I was saying is important? Well, we have now, I know it's super small, but we have
this now, this spans, because what I'm doing in my client is I send a get request, then I get the response,
I send a put request, then I get the response, so I can take the behavior of this other service,
the other side, so the cool thing is, because I was instrumental in my spans with this
open telemetry standards and libraries, now I can inspect my span, and I can take here that,
okay, this span corresponds to a post method, and it received a 200 code, so everything is okay,
what's I do not need to worry about these things, it was in OTP format, it was a client,
you remember I was saying this resource, okay, I said client, now I have here a client, I know
this span is coming from a client, and also the URL I was calling was this one, and I can also have
some information about the library that is generating this, so I can see that, and the other cool part
is I have here my logs, right, so I was talking about metrics, log and traces, traces,
when you start using them, and understanding it's power, maybe you start to put less logs
to the pipeline, and in case you were wondering for metrics, yeah, now we have them,
this is by the way I was just sending these metrics and and traces to an open telemetry collector,
and of course, storing them in Jagger and here in a victory metrics instance, and here,
you know, this is open telemetry, in case you are not used to open telemetry, I'm more to
permit you, you can visualize it the same, so you don't need to for the user, at least this is
not an issue, right, so you can easily take that, so that's it, for the demo, let me go super quick
to logs and we are done, yeah, and in case the demo didn't work, I had those screenshots, logs,
you all know logs, that's the first thing we do as programmers, hello world, that's a log, that's
observability, right, we are all experts in observability and logging, so what I am proposing you here,
right, is use, well, me, everyone is metrics, logs, and traces with open telemetry library,
and that's it, why? After all, you will tell me, yeah, I like to just bring to the console,
yeah, whatever, we all do it, actually if you go to the open telemetry library, they have in
their examples, they are printing to the console as well, so why shouldn't I? Well, if you continue
doing this, first of all, you have already the library, you are just need for metrics and traces,
so why not for logs? You can add attributes, for example, in an open telemetry collector, if you
use it to collect the logs, you can add the attributes which are standardized, like hey, this is coming
from this container or this name space, and you can also filter and but, so you don't have a lot of
signaling, you can transform and process the data, so it's pretty convenient, you can also help
for correlation. For example, logs, if you add a trace ID, can you imagine how cool is that? Because
now, if, because you like to look logs, okay, and you do your kip city a lot, if you see a
spanity, now I can go together and take, okay, this is all the context, cool, right? So there are
many reasons why you should be doing this. Also, yeah, we all love, for example, in C++, when you are
instrumenting logs, then you start to reinvent the wheel, because you start, how do I activate or
deactivate the login? How do I reduce the login level? How do I, and then you start, we all have
that coworker or even inside ourselves that likes to reinvent the wheel and create another library
for this and another, and things can get messy already, have it, why don't we don't use it?
However, I know you guys will be still using it, I will still be doing that, and I will just,
hey, I have an error, I just put it to the console and I'm done. Okay, if that's your case, it's okay,
because if you, one means putting your errors to the console, in this that when Linux is a file,
you are putting your logs into a file. So the open telemetic collector can read them,
process them, and do all these things, and send them. Of course, you will lose some things,
like correlation, you cannot correlate the idea if you didn't instrument it with a spam,
but at least you get something out of it. So it's not that bad. And okay, you are experts now in
this, you will be creating your resource, and adding it to an exporter, and the processor,
and all of that, so you already know how this works, it's the same for all, for all, once, and
and this is how it looks. So it's not that bad. I mean, it's super quick. You just say, okay,
give me my locker, and I love my data. However, what I would recommend is if you are using traces,
let's add your spine IDs, because now you can add the end correlate. So it's just a small thing.
But, and this is the last part. Why, how does it look like, right? We know, hey,
ever here, that's a log, this is a log in open telemetric, okay? It has those resources,
it has maybe some time stamps, which is good because, at the end, you like the time stamps,
you want to add them, right? It also has all this information about my library that I was showing,
so this is all standardized. So my only message here is what's out, because these things can be
expensive. We love to put a lot of logs there, but we will have, as I was saying at the beginning,
there is another talk on how good our body's auto instrumentation, the same goes for manual instrumentation,
normally, at least in my experience, one of the biggest issues we had in performance
was because we were logging a lot, and we needed to produce them. And also with costs, right?
Okay, so as a summary, if I want to give you some takeaways, or I would like you to take
some things home, try to mix both worlds because automatic instrumentation is also nice,
but now hopefully you know how to do manual one. Open telemetric will help you decoupling the SDK
from the API, so you can abstract your business logic from where you are supporting data and
change in the supporters, in this case, and semantic conventions, don't forget them, they are
super important because they help us to correlate all the data and understand what the hell is going
and okay, if you still want to, you know, like print, then you use a collector, like the
open telemetric collector, it will help you to inject some data and enrich your data so you can
have good observability in the end. So that's it, and now we are open for questions, thank you.
Thank you.
Can we have any questions?
What's rough, last night, right? I'm out of question about using it in embedded settings,
so I have an embedded Linux system that I want to apply open telemetric too. Sorry, I kind of
hear that the mic is the mic, or maybe it's an embedded Linux system, I want to apply
open telemetric too, but I'm worried about things like the aggregator, how long is the retainer,
across reboots and stuff, have you tried using an outside of like a Kubernetes cluster?
So you mean inside the code? I'm just talking like instead of on a cloud somewhere, like doing it
on an embedded device with a battery and limited resources, you know how well it's a
lot of people there, or? Yeah, so if I understand correctly, you want to use the open telemetric
API and you are seeing that it's aggregating data and maybe it has some performance.
Just imagine or in my experience, what I saw is that it's actually convenient to have that,
because otherwise you will be sending all the signals one to one, and then you have like
a lot of networking issues, because you are having all that bandwidth. Of course, you will need to
find you in the end how this thing is, but I have an experience in issue with these parameters,
it's just more like, okay, really, I just send one second of aggregation and I'm done.
Of course, if your process goes down, then you may lose some data, but maybe when things explode,
it's not easy to debug, because it's a very question, thanks.
In the upcoming profiling signal, will there be also automatic and manual instrumentation?
That's, so if profiling signals will have automatic instrumentation,
I'm not an expert in that field, so I will not like to you, I believe, yes, because
I believe it's done with EVP-Ed, but I will set up and maybe there are other talks later,
so maybe they will have you with that one.
Start it.
And so, thank you.
Thank you very much.
So do we have the next speaker present?