More than 80% kind of, so that's great, so people repeat this because you will really enjoy.
Today we are going to learn how we convert a hard file into an open telemetry trace, and how
that's really fine in our browser observability.
Always before I start a talk, I would like to say like, what is the goal?
We want to monitor the bar and optimize better web applications.
That is the reason why we are here, and now we are going to understand how we are going
to do it.
For the agenda today, we are going to have a quick intro about open telemetry, which
is the tool that we are going to do for that project.
For hard files, some of you might know what they are, but let's go a little bit deep,
and first analyze what we are implementing it.
Later, we are going to explain how we are doing that hiring to open telemetry,
our architecture, how we are building it,
and of course, we are going to end up with a lesson about open project.
So now you know the topic, let's meet also the speaker,
Antonio, come in from Spain, working in Tisco-Phersonized,
and specialized on observability.
Go to the intro.
I would like to ask all of you to go to the following cure code if internet allowed.
And let me know what came to your mind when you think about open telemetry.
When you hear about open telemetry, what came to your mind.
So my early of the suns are coming from a rehearsal that I did before,
but many people think this is mainly open source, observability,
metrics, traces, locks, all of those is true in a sense.
But the way how I like to explain this is our observability,
open source, observability framework.
But this is not a single thing.
It's a tools like the collector.
I guess many of you know the open telemetry collector.
APIs, SDKs, SDKs for the different languages,
like go Java, many more.
It is mainly used to manage telemetry data,
because we want to instrument generate and export that telemetry data,
so we can make use of.
So this is open telemetry, and we also would like to
regulate one for size different signals.
We have traces, metrics, locks, and profiling.
We are going to focus today on traces.
I hope all of you know about this.
And now let's move on.
Let's talk about the collector.
Collected one of the pieces that we are using in this project,
and this is mainly used to receive data in a variety of different formats,
coming from different source, like it could kind from OTLP,
Yager, Prometheus, but you can even implement your own component
to read data from a database or from read data from a browser or whatever.
This is done in parallel or the reception,
and now you go to process that data.
This is done sequential.
And here you can do many things also.
You can filter out data that you don't want.
You can enrich that data when it's extra,
you can buy it for performance reasons.
And then, similar as receiver, we have exported.
We export that data outside to the different observability buttons,
or even as I said before into a file.
This is totally up to developer.
There are full flexibility here.
And we have also the concept of extension.
They are serve component over all the collector.
And they are like, help for the collector, profiling,
memory, storage, and many more.
So now you know about the open-telling design.
Let's talk about the hard.
Hard is nothing else than the browser interaction.
I might many of you have seen that before that picture.
Like, you are opening a web space,
and then you see all the resources that are milled and loaded.
All the requests that are being performed.
This is the hard file.
But what is the problem of the hard file?
That it's contained since it is data.
Like, it can contain your cookies,
it can contain your headers,
like with your password, authorization,
user, email, all the things.
So it's quite sensitive.
Usually it's engaged,
and it's for debugging purposes.
When you have a problem, you send out hard file
to other person who helped you here.
This looks like something like that.
It's a list of requests,
and each request is going to have a variety of different elements.
Like the method, the URL that you're hitting,
headers, body, response code, response,
a object, and many more.
Let's talk about quickly about fossilized.
We are using those two standards.
In fossilized and network as well as platform.
We monitor the network.
And we have AIN all around the globe.
Those AIN are acting as real user.
So we have AIN, a Europe Asia, America.
And we can also have AIN inside of your own infrastructure.
Because you might also want to know what's the problem inside of your infrastructure,
of your employees or in your office.
And all this data is in real time.
Because you want to discover when the problem is.
At the time that it is happening,
not like 5 or 10 minutes before.
Sorry, after.
Inside of fossilized, I would like to emphasize
about the paid slow test.
It's a test that is using the hard file.
That's just render our webpades as it was a user.
And then capture all the hard file or the hard file data.
And it's only live on the webpades of fossilized.
That's why we say, okay, let's convert that into a trace
and send it over to our customer.
And this is what we are going to discuss now.
We have our hard file.
And we want to convert it to a student tracing.
I might, my idea of you have seen that before.
This is like all the spans that are one after the other.
The half hierarchy between the other.
And you may be thinking, how you represent a hard.
We're going to have like a root span that represent the URL that you are downloading.
You're downloading a, let's say, fiscal.com.
And you are getting that what pays.
That is going to be the root span.
It's going to have the time that it starts and the time that it ends.
And now for every resource on the webpades.
We're going to have an span.
That is why it's going to have like HTTP data.
It's going to have the URL, the method, all this info.
That appear on the hard file.
We're going to convert it into open telemetic span.
And all of those are going to be child of the root.
Because there is no dependency between those as span.
The dependency is only with the root.
And the cool part is like you have also the time of it's span.
So you can see the performance of your webpades perfectly.
Each span is going to have things like the name that contain the method on the URL.
The kind is going to be always climb because we are doing our request.
We are doing our request to our webpades.
We are doing our request to our server.
We have also, as I said before, the start time and end time.
The status, if there was any error, we are going to put the start to error.
And of course, it's going to have the span ID.
And now let's talk about which attribute we put in those as span.
Here is a key pillar of open telemetics, the semantic convention.
They don't only say how you have to write down your attribute and what is the format.
You also provide extra meaning.
And for me, this is the most important about semantic convention.
Because it's teaching me method means the method that you have been used to make that request.
And not only that, it's also explained to you how it is going to be breathing every observably.
So everyone is going to understand it perfectly.
So we are following the open telemetry.
Record attribute like method, address, for URL full and the status code.
And those are key to represent a request if you think about it.
Sorry, I want to do it.
And you may also thinking about headers.
Heather are optional in the convention.
And they represent like the list of it's array.
They represent the list of header.
But the problem is that they might contain sensitive information.
For that reason, in 1000 as we have decided not to include it because that could lead into PII reasons problem.
So they are not being at.
Then we have also the concept of resource in an span.
A resource is nothing else that who is generating that telemetry data.
For us, it's 1000ize, test, and the alien.
So we are going to have all the information.
1000 is the CD, the name, and the type.
Also the alien, as I said before, 1000 is half alien or around the world or in your infrastructure.
So those say you are going to have a location.
And alien ID and alien 9.
So you can identify who is sending a query in that web page.
This is how a trace is going to look like just in a JSON.
So you're going to have all the resource attribute.
And after that, you're going to have all the root span.
After the root span, you're going to have all the child span.
Quite simple, just a trace in JSON.
Now let's deep a little bit more into our architecture.
How we are building that, how we are transforming the data.
We have two collectors, as I said before, we are using open telemetry collector for that.
So we have two collectors, one is for reading the half data and converting it into a trace.
And the other one is mainly to route in that data.
I want to mention something else.
We have the concept of integration in 1000 as an integration.
It's like with data, you want to send to which observability back end.
And that's quite important that why we have the integrations collector that say that data go to that observability back end.
That could be a splung, giant trace, graphana, any other.
And then the reason why we have also two collectors, because they are going to scale out at a different piece.
In the left side, you're going to have the trace collectors, you're going to scale depending on how much data you receive from Kafka.
On the right side, at least from my perspective, the integration collector is going to be the scaling based on how many integration you have.
Because for each integration, we are going to have an open telemetry, or it'll be exported, sending the data to the destination.
Let's zoom in on each of the collectors.
First, we're going to receive the data in Kafka.
We're going to transform that data into an open telemetry, so we just explained before.
We're going to have in the processor, we're going to filter that data.
Why? Because in my data point, it's not going to be sent to any observability back end.
We don't want to go through all the processor and export and so if it is not going to be sent.
So we filter out there.
We also have the attribute processor to enrich that data.
Because what came from Kafka is nothing else like TSID, ANID.
So you're missing like the test name or you're missing like, sorry.
You're missing like the ANNN, all the data is added at that processor time.
And we also have a batching processor for performance reasons.
And then we export the data to the next collector.
And in the next collector, what we're doing is receiving the data and we have our integration processor.
What are we doing here?
Imagine a data point, the customer wants to be sent to Grafana and Dynatres.
So we need to duplicate that data point here and inject the new attribute, which is the trace AD.
Sorry, the integration AD.
We add that attribute and then in the routing exported, the only thing that you do is like, you see the integration AD.
You say, okay, that's go for Grafana and then you are just exporting that data.
So if you think about it, it's like a router.
And now I want to mention also the extension.
We have extension for the health check to know when the collector is ready.
Store it together all the cuts data about the test, Aging and so on.
Event bus.
So it's quite useful because every time that something happened on our collector, we sent a,
it is a pop-sup system.
We sent a event.
And then another component might receive that event like the metric collector.
Sorry, the metrics extension.
We also have the sentry extension, which it's intercepting every panic and error,
and then sending it to the sentry platform.
Okay, so now you know about all of this.
So let's go into the deep and see an example.
In thoughtsonize, we're going to create a simple page load test that go to fiscal.con, as we said before.
That 1000S test is going to contain data like how much times take to load that word page, how many completion.
We are running test periodically, like every minute, every number of seconds you decide.
And then it's also generally what I mentioned before, the hard data, which is the key in that presentation.
But that data only leaves in 1000S.
That way we need to export it to the different observability backends.
We create that integration as I said before, where we specify where we are sending that data.
In that case, you're going to be younger using the RPC.
We as I am running local thoughts, and don't have to put any authentication.
But you could add in your observability backends.
And then you select also the test data that you want to send.
And that's all. You don't need anything else.
And now we are in younger.
And we can start seeing our data and our traces as we are familiar.
So we serve for traces. And this is one for this call.com.
And you can see all the spans that represent all the resources and requests of our what page.
For each span as we agree before, it's going to have all the semantic convention, like HTTP status code, method, real full.
And it's going to have also all the resources achieved, like the 1000S test ID, test name,
agent name, and all those details.
And you may be thinking of whether or not what that is the advantage.
You have the same that you have before in 1000S, right?
This is not true because now you have all the flexibility and functionality for any observability backends.
If you think about it, now you can do things like, this is an screenshot from before.
Now you can do things like, I want to filter only for the traces that have an error or a status code for 100.
And now you go to your work page and you find that there are a resource that have at that error have a battery quest.
So you are going to discover that you have customer having the problem.
And this is quite useful because you can create alerts. You can see how often that happened.
As you have the agent location, you can see that this happening in Asia or that it's having on Europe or any other place.
So this is quite useful because now you have all the capabilities from our observability backends to understand your traces.
But not only that, you can see, okay, I want to see what are the timing of my spans.
Remember the time is going to be the time that you need to load up your page.
For fiscal.com, it's taking around, I don't see from here, but around 5 seconds, 7 seconds.
So I want to see why there are those different, are those different because they are in different regions, are those different because they are more aspiring one and the other.
So you have the capability in error to compare two traces.
And that's quite cool. This is a huge trail, but if you're familiar with that, you're going to identify that.
We sure what's downloaded here, not here. Why? Are these sequential? If one failed, the other is not called.
So you can find out those problems that your customer are hitting and you don't even realize.
Now that we have seen all the process from architecture and so on, now let's go to the lesson line.
We are using tracing always for microservices to get each other.
If you think about it, this is another use case, and it's quite useful because our customer can take advantage of.
The open telemetry is super helpful for us. The community, we were having some bugs with home.
They were solved where it needs some component. They were helping us.
Also for the semantic convention, it's what you find is mature enough to be used and I recommend everyone. This is one of the pillars.
And the open telemetry collector at the beginning was really hard to understand, really hard to use.
But after a few spring, you start creating your own component and play in your own. It's quite much you.
And let's go to the beginning of the session. What is our goal? We want to understand the performance of our browser, better.
Improve our monitoring and identify performances.
Thank you so much. I hope you enjoy the session.
I don't know if someone will have any questions. I will be more happy to answer.
Okay. So there are a few there. I will be in the door right after the meeting. Sorry, the presentation.
So happy to answer any other question in person.
Thank you for your presentation. I'd like to ask you how do you export the hard files from your browser?
I don't know. Carf, carf. Okay. That's a detail from the first one.
I assume that everyone already have the hard file. What we are doing here is like we have a nailing.
And that nailing is collecting all the hard data and is converting into a adhesion and sending it over to Kafka.
And then we have our collector just reading from that Kafka. So this is more detail in the first one.
And I suppose this is not that why it didn't go deeper. But yeah, this is, it could be up to you how you want to get the data into Kafka.
Or you can read it directly.
As you said, you can create your own component receiver that just load the web page.
You'll have the flexibility. Thank you. Thank you.
Okay. Yep.
We've all the extensions that you can have.
How is the integration between Magento and Google Cloud for telemetry data?
What do you mean sorry, the integration?
Because there are various regions from what I understood and various extensions to a process data.
Magento does a lot of things under the hood and I was wondering if there was some integration in that.
There might be some integration that I know the work session before.
We have a nailing that this acting as a user.
So it's actually loading the data from a Chrome.
So we are acting extracting of the data from a Chrome browser and we are getting into a heart.
So we are not using any of those extensions that you're mentioning, but for sure there might be.
Okay. There might be a wrong.
Anyone else? Sure. On the other side.
Did you open source any of it? Like the conversion part to hotel?
Yeah. I talked with the community before.
Yeah. We are willing to help there and create that component.
Again, it would be like, it could be the only other processure because depending from where you are getting the data.
It could be a processure that just convert from a JSON into a JSON half into an open telemetry trace.
Or it could be a receiver time, but more than happy to do with the community.
We have been working with the community before, but for, I'm sure.
Giving a component is like a quite silent process because there are many components already.
And there are many that are doing similar things.
So it's quite silent, but we are more than happy to contribute the data to the community.
That's totally part of this content, a vision.
So you're putting the hard JSON on Kafka and then reading it with Kafka receiver.
Do you have a custom extension encoding extension to the parse that JSON from hard into hotel P?
Correct. Absolutely. This is what we are doing.
We are using a field in Kafka receiver called extension and we implement our own extension encoding.
And that's really useful for us.
Thanks for watching.