Okay, let's start right into it. The next talk is about combining traces, namely the
kernel F trace and the editing and the user space trace. So, who am I? That's me, I'm Felix,
I work as a embedded Linux developer at Siemens Foundation Technology in Munich. And if you
ever want to contact me or reach out to me, this is where you find me, I'll get to help
master it on and also via email. So, before we start with the actual technical stuff, I
click Paul, who of you has worked with F trace? Ah, that's very good. With U.S. work with
editing and G tracing. Okay, let's people, user space or kernel, kernel, user space?
Both, okay, okay. And you of you use both F trace and editing and G tracing at the same
time. No one. Maybe that changes from now on. Let's see. So, where do we come from? This is
not an artificial problem we want to solve here, but one we actually had in the field. So,
we needed tooling to analyze a complex real-time communication box. So, in the field, we had a device
and there did some real-time communication. And the device was also secure booted, that means
you cannot easily make modifications to the kernel or load kernel modules. In general, I really
dislike out of three modules. So, the editing and G module from this perspective is a bit problematic.
And apart from that, it was also really hard to exchange a replace be kernel we are running
there. Simply because it was tedious. So, it was easier to use the infrastructure we already
had in the kernel. And combined that was the things we can exchange. Like, we were able
to deploy containers there. Also, privileged containers, which is good if you want to
configure the tracing infrastructure. And by that, we needed a solution, which worked for our
developers, which are familiar with the trace compass tool to analyze traces. And in some
of our applications, we also already had user space trace points. So, LTT and GUST trace points.
And the developers really liked them because they are low overhead. If you turn them off,
they are also low overhead. If you turn them on, but of course, you have some. And they
are quite easy to integrate into C or C++ applications. So, we made a plan with the goal
to instrument the applications that do not yet have the trace points. In it was LTT and
GUST trace points. And use the F trace infrastructure in the kernel. And in the end,
then the developers can use the trace compass tool. Also, from the officials to view the combined
trace. And it was combined. I mean, they should not view two traces, but they really want
to view a perfectly time synchronized trace, both from GUST trace and kernel. For the
diligence, there is an F trace reader plug-in for trace compass. We tried it, but it was
horribly slow. And in the end, we also wanted a more streamlined solution. So, from that
we derived some requirements, like we want to convert the kernel F trace into the common
trace format, because that is what the trace compass application can read. Also, some other
tracing tools. And another really important thing is the trace tracing and the analysis
of the trace should happen on two different systems, because the system we have in the
field is an embedded device. It doesn't have a UI or whatever. So, we need a way to capture
the traces there. Copy them over to the machine where we analyze things. And then run
the analysis there. I was that before we start implementing, we had to look at some conceptual
challenges. One was how to read the trace, buff us in a portable way, this is what I just
said, so that you can trace on one system, copied over to the other one. Then how to convert
thousands of trace events, like basically all the F trace events, without writing, I want
to one conversion for each of the events, because that is really, really tedious. Then
how to synchronize the trace clocks, both in user space and kernel. And finally, no solution
really works out if the developers don't accept it. So, you know the solution which also
works for non-experts like application developers, for instance. And with that, let's dive
into it. The first challenge is how to read the trace buff us. I hope you can see it in
the dark mode here as well. In the diagram, you see that we use trace command on the device,
which in the end is just a front end to the trace of S in the kernel, which you can ship
in a privileged container, that the trace command binary. And with that, we can record both
the flight records, so we trace itself, the binary trace as well as all the metadata we need,
like NDNS, the event descriptions, process command lines, and a couple more things. And
then we copy this thing over to the other system where we do the analysis. And there, we
have these great libraries from Stephen. The trace command, the trace event and the trace
of S to open up the file and extract the data there. And in the trace, thought file, you
have basically a set of metadata, which are the kernel command lines, the descriptions
of the events, event classes, and then the flight records, which are usually one buffer
per CPU, but you can also have more buffers per CPU if you use sub buffers, for instance.
And then we use the right-line event converter, which then converts the events, which
we have in the flight record from one representation, like the kernel representation, into
the intermediate representation. So the next thing next challenge here is how to create
the common trace from a trace. And for that, there's the great bubble trace tool and
the library, which is in the end of the screen processing application, which is plug-in
base. So you have a set of input plugins, which can read an arbitrary format. And these
are the green ones here, which that we can implement plug-in, which can read the F trace,
on the trace, and then you have a set of max plug-ins of filter plugins, which combine
the traces from multiple input plugins. And then you have an output plugin. And in this
case, we can just use the plugins, which are already there, like the CTF writer plugin,
to then write our trace in this common trace format. And I would really recommend you, if
you didn't do it before, to look into that project, because this is quite flexible. And
once you get a grip on it, it's easy to implement basically any trace format A to trace
format B conversions. So how to technically convert the trace events, the first thing, which
is noticeable, is that the trace, the F trace events are plausible. There is not a binary trace
format, it's not part of the kernel ABI, that means you need to capture these descriptions.
But in the kernel, if you look at the trace entries here, like events, I accuse of the
IQ for instance format. Each trace point has one of these format descriptions here.
While you get the internal type definition of the trace event, like here, you see the types,
you see the offset, the size. And by that, you can parse the data in the binary flight
record. Also, the trace events are a name, like you have a system, a name, and then the
format file. And the good thing is you don't have to do this parsing by yourself, because
for that, you can use the lip trace event, which already can parse this textual representation
of the event. And then, yeah, provide you a live, provide you APIs to extract the event type
and all the field offsets and field types. On the other side, we have the Bobbi trace library,
can programmatically create events, or more technically correct event classes. So, what
we do there is we just implement a mapping, by reading the event classes out of the trace
start, out of the trace that metadata, then we recreate them in the, in Bobbi trace, in the
intermediate representation. And then we store these mapings in a hash table, because when
we read the flight record, we just get event IDs. And this is potentially hundreds of
megabate bytes of data. This really needs to be efficient. Otherwise, the whole implementation
is horribly slow. And even as it's out of bandwidth, still, you want to have some performance
otherwise, it just takes too long to do the conversion of the traces. The second part
is semantics. So, as I said, there is this LTTNG kernel module, which directly can write
the common trace format traces. But it has some slight differences in the semantics of the
events compared to the F trace. So, one thing is that in the F trace, you have the events
named after the system and then the name. And in LTTNG, you don't have that system name. So,
for events like the IRQ events, where you have the system IRQ, and then an event which doesn't
start with the IRQ prefix, you have to add the prefix so that then the trace tooling, the
analysis tooling like trace compass, is able to understand that this now actually is an IRQ event,
because it expects the LTTNG semantics and not the F trace semantics. So, this part of the conversion
is easy. And then you also have field name conversions like in the kernel, you have the process
ID or pit. And LTTNG consistently uses the term thread ID or tit for the same thing.
Then you have some field values, you have to convert like in the scheduling events, you have
the priority values. In LTTNG, they are adjusted by the max RT price. So, you have to subtract
100 in this case. And then there are some events which only exist in the F trace or they only
exist in the LTTNG kernel module trace. For the F trace events, we can just add them because in
the analysis tooling doesn't care if they are more events which it can't understand.
For events, we don't have an F trace, we can't do anything because we only have F trace on
the system. So, either we add these trace events to the kernel and eventually profit from that
or we just accept that we don't have them. Another quite funny thing is that the LTTNG pit
refers to the LTTNG, the kernel pit is named tit. And in LTTNG, we also have a pit. But this
refers to the kernel thread group ID or tit, which unfortunately is not captured in the
scared process for event. So, in the F trace case, we don't have that event, but in LTTNG,
we want to have it for the analysis. It also works without that, but this is something I want
to contribute to the kernel to also capture that in the escape process for event. I guess it's
cheap to capture and because then we can really do one-to-one mapping. The next challenge is how to
synchronize the trace clocks. And this might look simpler than it actually is. So, the first thing,
you have to be aware of that the user space trace and the F trace are captured individually.
There's no synchronization between the two. So, what you do is you start the F trace and then
start the LTTNG user space trace or the other way around. And once you run into the issue or
into the problem, you want to analyze, you have to stop both. For the F trace, you can just use a
trigger which then turns off the tracing for LTTNG. There's a similar thing you can use,
but you also have to program it individually. And if you use two different trace clocks for
both traces, there is really no way to reliably correlate them at a micro-sevent or below that
granularity. So, what you need to do is to use the same clock and the LTTNG user space uses the
clock monotonic. In the kernel, usually the trace clock is used, but you can switch steps and
by switching it to clock monotonic, both traces use the same clock. So, they are running at the same
pace and by that they are they are already aligned. Then there's one more thing to consider that
is in LTTNG user space. The trace events are also adjusted by the world clock. But luckily,
the offset to the world clock is also recorded in the trace metadata. So, for a particular value
of the clock monotonic, what the value of the world clock was there. So, what we do then is to extract
that and just add that offset to the F trace clock as well when doing the conversion. And by that,
we have perfectly aligned traces, which are which is quite good if you look at the larger scale,
which are even a world clock adjusted. How to make it usable for non-experts? The first thing
is bubble trace provides a very interface that means it can discover all the plugins. And then
when you run bubble trace like here where you know we don't have it here, here we have it bubble
trace and then you provide a file. And bubble trace will all ask all the plugins. Can you read
that file? Or do you have some processing mechanism for this input string here? And then the
plugin can say either yes or no of some confidence. And then the plugin was the highest confidence
wins and bubble trace fires the plugin up and registers it as a source plugin. And by that,
you can just use this short format here, bubble trace, trace starts to already read the trace
start and the events from there and then they just grow your console. The second thing is we don't
want to have them on the console, but we want to convert them to the common trace format. So,
we wrote a replication F trace to CTF which in the end does exactly that it takes and
trace start file and it then converts it to the common trace format and here you specify an
output directory where then the traces are written to. And the easiest user faces one with a lot
of command line parameters. No, I'm just joking. Usually you just can call it, usually you can
just call it like that. So, we have here this F trace to CTF. Then you specify I want to have
ATT and Gsementic. So, I also do the semantic conversion where I convert the pitch to the
tit and so on and the event names. Then I provide my user space trace. This is also a directory in
this case and the output directory. And what the tool internally does, it first reads the first event
from the user space trace. From there we extract the clock definition to get the offsets to the
wall clock. And then we set up the second trace processing graph where we feed in the trace
that file and some metadata like the clock definitions. And we also register the CTF reader to read
the user space trace through that as well. And then we just follow the standard bubble trace logic
that we use a max to interleave the traces and by time and then the CTF thing to write them
out to a directory. And this is what you all were looking for. Probably here we have a snapshot
of the trace compass application. It's a bit small here but you have to trust me. There we use
the tit of the bugger application and edit a couple of user space trace points and we use the
F trace to capture the scheduling trace event. And here you'll see that the trace compass is
able to correct the analyze these traces and also show them together. So with that we are done
and I think we have time for questions. If you're interested in the project have a look at
GitHub Siemens BD2F trace to CTF. And while implementing that we also did a couple of upstream
contributions mostly bug fixes which I also list here.
Thank you, Kultalk. So what I wanted to see in the picture before. Do you have like a
drill down from the user space trace point like start an end for a function and sort of
align them to see like like costac wise basically. So to see like it goes to an IO function
to go to some kernel so IRQ. Yes, so in theory you could use the function graph tracing as well
and then you would see the call stacks but in this case we only use trace events so we don't have
a call stack but you can use the I think it's called red analysis or control flow here and
the bottom you see it where you see the control flows through user space and then kernel as well.
Thank you.