Hello everyone, my name is Mattude and I am CEO of Ephesias.
I'm here with Olivier Zion, works with me at Ephesias.
We are here to introduce the Exat Tracer, so it's a project we've been working on for
the past two years, mainly with AMD and Lawrence Livermore National Laboratory, and we're
basically adapting the LTTNG Tracer, which has been really focusing on the embedded
and telecom world to HPC.
So I'm just going to backtrack a little bit and give some context so that we're on the
same page in the room.
So quick introduction about what is tracing and how it differs from other strategies
for observation of systems like profiling.
So it's based on instrumentation of code, you can either statically insert or dynamically
enable instrumentation or it can be done fully dynamically.
You then collect or react to a sequence of events that are emitted during the code execution.
So one important aspect is to have many more introduces the stiffness on the workload,
so that the issues you face reproduce on the observation.
So for that low overhead is really key, it can be compared to logging, however it needs
to handle very high event throughput in the order of millions of events per second.
So why tracing?
So why would you want to use tracing?
So one of the main reason for using it is to identify the root cause of issues.
So Eisenberg, so bugs that are hard to reproduce that only reproduce once in a blue
moon, issues that disappear on the observation, identify the root cause of performance
bottlenecks, protocol API, API specifications, yeah, violations, and to monitor program behavior
and then react.
So one example is hooking on application code and gathering a detailed snapshot of the events
that led to that problem.
So you can then identify performance and outliers and root cause of errors.
So tracing and profiling.
So I know that profiling is, so both have their own, I would say, strength.
So profiling is more lightweight because then it gather samples, tracing needs to be inherently
lightweight because it extracts a lot of information, so where does each strategy fit?
So they are both complementary, profiling is very good at identifying active usage of resources.
Tracing can do that as well, but the profiling as the benefit of just taking samples to do
that, so it's more lightweight.
However, tracing really excels at identifying resource misuse.
So when the CPU is not actually used, so you're not using 100% of your CPU, you don't
know why, and you're waiting for another system, for IO, for some other thread to complete
something.
So identifying those more complex patterns where your resources are not used as they should
be, tracing is excellent at that, so this is where you're tracing really shines.
So a little bit of history on the LTTNG evolution.
So I started the project in 2005, the main focus has been since then mainly telecom embedded
in real-time system, large multi-core systems, scaling to many, many cores.
Linux kernel and application tracing, we've created the common trace format version 1 and 2.
It's been developed in collaboration with the development of people doing trace analyzers.
So bubble trace, trace compass, we'll come back to that.
So a few words about the common trace format, so I created that format with the tool infrastructure
work group around 2010 and the one that was released 2012.
So it was based on a domain-specific language to describe the metadata.
It provides a structured type system.
It can be defined statically or dynamically, the static definition would be more for embedded
systems where you can ship the description of the trace with your SDK and then you just gather
the hardware, the trace from hardware in binary.
Clock descriptions, allow correlating between traces.
It's a binary trace format, so it's really compact, it's fast to produce.
So in it, it's a trade-off between compactness and really not using a lot of CPU when you
write it out.
So we're not talking about compression there.
We're talking about writing binary data that can be either aligned or not aligned depending
on the architectures so we can get the most of the performance of the various architectures
that are targeted.
So it can be easily generated by hardware and software component.
So CTF2 has been released last year, 2024.
So the main difference is we switch from a custom DSL to JSON.
So it's a super set of the CTF1.8 type system.
It has new built-in types and its main strength is to allow user-defined metadata and
extensions.
So you can tag specific fields saying, well, this field, this array of four bytes, it's actually
in my own name space, it's IPv4 and I know that in my tooling, I need to print an
inter-pret IPv4 as a network address.
So it's a binary trace format.
So the binary trace format is the same as CTF1.8 plus additions.
So the new types that are supported into are being added and that's pretty much it.
Bubble trace.
So that's the reference implementation of CTF as trace reader and producer.
So it can convert filter, seek, and analyze trace.
It's both a plug-in system and also a CC++ library API with Python bindings and a common line
interface.
So you can basically support our betrayary trace formats and convert to CTF or analyze
them on the fly just by implementing plugins to read those other trace formats.
So it's really meant to bring all the data together.
It's a bit below adding custom phases.
It does the correlation also across traces by times them.
So you have various information sources and you correlate them in its model.
And it has been built in plugins to both read and write CTF1.8 and CTF2.
So trace compass.
It's a graphical interface to view in analyze traces.
We'll show some more in the coming slides.
It supports the common trace format as well as other trace formats and logs and custom
logs.
It has plug-in system for views in analysis faces.
So you can create your own and add them into that project.
There are many built-in views in analysis.
For the Linux kernel, for instance, this guy is a hardware resources, CPURQ usage, many
more.
There's a critical path analysis, log correlation with traces, network packet correlation
and so on.
The visual example of how trace compass look like at the top.
We have the resource view showing each of the CPUs, their state, the frequency, they run
on.
So that's useful to know what the CPUs are doing over time.
So the x-axis is time.
Under that, we have the control flow view.
I would call that also maybe the tread view.
So it's on a per tread basis.
We show what each tread is doing over time.
And then we can see the waiting and the unblocking between threads.
We have a detailed event list and then some Instagrams at the bottom.
So the main features of LTTNG, so being low over-ed, it has a really fast per CPU ring
buffer in the order of 100 nanosecond per event.
It has a user configurable dynamic runtime filter per evaluation.
Let's say of a two integer that would be around 15 nanosecond so when you want to filter
out a lot of the event that you don't need.
Event streaming to disk or network, it's an absolute mode where you do flight recorder
tracing in memory and when something interesting happens, then you capture from memory.
So you remove all the IO that you otherwise need to do to stream out the data and you just
capture when it's needed.
So that's really useful.
It has trigger mechanism event notifications with payload capture that's to react to what
is happening on the system, rather than buffer things.
In a session rotation mode, that's mainly meant for basically splitting traces into chunks
so that each can be sent and analyzed faster than just waiting for the entire experiment
to complete.
Trace correlation.
We do correlation across tracing domains, then external user space, and across host.
Across host, we are based on the time synchronization of the machine, whether you use
NTP or PTP.
There are also algorithms in trace compass that real-line traces at post-processing based
on network communication, so it basically take the TCP sequence numbers and real-line traces
based on that communication.
So I'm doing that part as well.
Yeah.
HPC collaboration.
So in the past two years, two years, we've started working with the Oregon National Laboratory.
They've developed TAPI.
It's based on the LTTNG and Babel Trace.
They provide instrumentation of OpenCL, Level 0, Kuda runtime, HIP, OpenMPT.
It's developed for tracing Aurora, the Aurora cluster with 10,000 node, 9 million cores.
It captures millions of events per second per node, and it runs for hours when tracing
is enabled.
So that's the scale.
TAPI provides nice summary about what is happening on the system as it runs.
So we are also collaborating with the Lawrence Livermore National Laboratory on the Exatracer
Project, so we're developing it in current elaboration with AMD.
So we've created the instrumentation of RAKEM for Exatracer, so it connects LTTNG to RAKEM
and MPI.
So the HIPHSA, RaktX, GPU kernel, dispatch layers, OpenMPI, KrayMPI, so that all goes
into LTTNG.
It's been developed to trace L capita with 11K nodes and 11 million cores.
So we are collaborating on a research project with Polytechnic Morayal as well on trace
compass.
It's a trace analyzer and visualizer, and we're working with them on improvement of scalability
to large traces, so more than 100 gigabytes in that's really pertinent in the context of
HPC.
And some of the goals there is to reduce the reaction time, so basically interactivity
when navigating in a very large trace, and also reduce analysis delay.
So this is the delay between, let's say you take a 12 hour run, that's a huge trace
that you gathered.
You don't want to wait another 12 to 24 hours of pre-computation before you can start
navigating in your data.
So ideally, we want to have some kind of pipeline in there, so that the analysis can
be done on other resources in parallel with your workload.
So let's welcome back to that.
So here's a stack diagram of the HPC software stack that we target.
So you can see that on the target, at the application level, we instrument in green, that's
where tracing is available, so CC++ application, GPU kernel dispatch.
We have partially, tracing partially available at Python and Java levels with logger integration.
At the runtime, we have partially available at the lib C, somewhere is instrumented.
HSA, HIP, open at the MPIs are instrumented.
And then at the system level, we have instrumentation of the linear kernel.
And that can all be funneled into the same tools on the align timeline, and you can really
see correlated information from across all those layers.
So that's one of the strength of the LTTNG tooling.
And then we have on the side the Exat Tracer that connects on each of those level of instrumentation.
We have the SDK, BarCitF will become maybe useful in the picture to trace on the GPU itself,
but that's not there yet.
Our guntapy also is there with bubble trace and trace compass.
All right.
So now I will leave Olivier.
So the Exat Tracer is targeting many of the layers of rock M, basically, and MPI.
So we have MPI, open MPI, create MPI, those are the two implementations that we have tested.
We have HSA, HIP, rock TX, and GPU kernel dispatch.
So how this is done, basically, for non-programmers here, these, for example, HSA and HIP are
libraries in C++, and there's header files that describe the API.
And we parse this API with client, which is a compiler for C and C++.
And from there, we derive a definition of trace points, and we generate wrappers, automatically
based on this definition of the API, and it's the same for MPI also.
And so with this wrapper generated, we basically do interception.
So for the HIP and HSA, they have mechanism inside of the runtime to basically allow interception
of every call, and for MPI and rock TX, we rely on symbol overwriting.
So symbol overwriting is something software to allow you to, basically, hijack symbols during
the runtime, and so you can call your function instead of the defined function by the
programmers, and then you can call the next function, which is the original one.
And so this is the advantage that you don't have to recompiles your software.
You only need to, the LD preload symbol over right for example, is just a environment variable,
and you don't have to recompile.
And the GPU kernel is basically instrumented during this patch, using the Roperfather SDK.
This is an example of a trace generated by bubble trace, so bubble trace, the common line
interface is basically just a simple convention, so we can read a trace and put it on the terminal.
So as you can see, it's quite verbote, there's a lot of HIP in here, I have two events.
So where we have a time stamp, we have a relative time as the event before it.
We have the host name, and we have the provider in the trace point, so in this case is HIP event.
It's a mailbox or a memory location, and it's the entry of that function.
And then we have context fields, so for example, this is the CPU ID 17, so we know which
chip you has been running this event, and then we have application context also, for example,
we have the MPI rank, so we know that the event happened on rank 1, instead of rank 0 for
the second event.
And then we have the payload, so the payload is basically what's interesting in terms
of argument of the function, so we have the correlation ID to determine which thread and
which function call happened, because we also have exit of the function, so we can do
a correlation between entry and exit, and you have the pointer of the, that was mallocate
in the size of it, and the next event is similar, it's basically a send of MPI, and we
can see for example, well, it's a return, so we don't see it in that much, but we know
that the function has returned basically.
This is a view in trace compass, and I must mention that the previous slide here, this
is the default output of bubble trace, but bubble trace is meant to be a program instead,
so this is really like just to see the trace, but you can, like, tap he has done, do many
more things than that, this is a view of trace compass, it's just a really small view
of it, I don't know if we can see, yeah, so we have the top, the top we have hip, so
the two event of hip, we have the malloc, and we have a main copy on the memory, and yeah,
and we have MPI under it, and the MPI event are sorted by rank, so we have MPI send, followed
by MPI finalize, and we have for rank 0, and for rank 1 we have an event which we cannot
see the name, and we have MPI finalize, and so we can see in this timeline of view, so the
x-axis is actually a time of the day, and we can kind of correlate between the nodes,
the communication between the nodes, in that way, so this is just trivial view, but we
can have much more detail if we want, so the challenge of tracing the HPC cluster,
basically, is that there's a really large volume of data, and this impulse execution
overed when you want to do tracing, there's also a lot of memory footprint and bandwidth
associated with that, and also high throughput, a value, for example, to this good networking,
and also to the storage of the data itself, at the end, like Matthew mentioned, there's
a waiting time in between the trade generation, and it's visualization for analysis results,
so if you have 12 hours run of something, you don't want to spend another day to wait
for visualizing the results, so pipelining is a way to circumference that, and then there's
also the interactivity of a trade visualization at scale, so when you have a lot of data,
if just zoom in in the trace to see a more detailed view, it's lagging the view, it's not
very interesting for the users, so that's another challenge, and also there's the precision
of trace collaboration across hosts, well, Matthew mentioned that we already support that,
but maybe the correction could be better in some cases, and so for the future work that we plan
is that we want to improve the granularity of coverage, so for example, we use client to
basically parse the editor file in C, but C is not really a good semantic for that, it is
an express what is an input in an output parameter, for example, so we cannot assume that we can
access this memory on the output or the output, so basically we're blind in some cases, so using
API annotation instead, we could, for example, yeah, we could annotate the function argument
input output or tag in unions, and we also have the site project, it's an API defining an
nextable type system for instrumentation, and it's important is that component type, and it's
a runtime in a ways, agnostic, yeah, I'll skip that, that's all, questions?
If you have some kind of API results in the idea, you look with things to implement some
arbitrary program language tracing, or if you want to add custom processing, but back in
a day when I did something, so if you find it in Python and copy it and turn out that it's
really crucial, and you can do a lot of interesting things, if you have this kind of API kind of
something that's not in tracing only in a part of the program, there are really interested
while not tracing everywhere else, for example, you can't stop if there was some API, so
is there one? Yeah, so there are CAPIs to control tracing, yes, so we provide APIs, I guess it's
to control tracing, but as well as for instrumentation, right, of various languages, so the answer is
we have some, but I would say the main interfaces for trace control is idricum in line, or C
C++ APIs, but then we can do bindings, and I think we have Python bindings, so you could
create your own. For the instrumentation, the direction I'm going is the lib side, the project,
SID, and I'm basically defining an API across run times and languages for that, but so that's coming.
So currently, the trace point of LTTNG, USD is really more CR-oriented, so you need to do bindings.
Yeah, thank you very much. Welcome, another question? Yeah.
So the question is, in the context of GPU profiling, which profiling and tracing, so do you mean tracing,
what is happening on the GPU or the dispatch? So what we instrument at the moment, the main focus
initially was really what is happening on the CPU controlling the dispatch of stuff to the GPU.
That includes, however, the kernel dispatch, so then you know which kernels have completed running,
and when they've started, so that provides extremely useful information, so that's the level we
currently open, but there's currently research at polytechnic trying to add instrumentation directly
in the GPU to extract traces from a GPU ring buffer consumed by the CPU, but that's further work.
Thanks.
I used the Chrome tracing set up with like gladiations on this tree, to be like, how does more
you'll take to form a query with that. So your question is, how do our trace format CTF compare
to Chrome tracing? As I recall, I hope I'm not so it's based on protobuff, the Chrome tracing format,
as I recall, or is it JSON based? Yeah. Okay, so in that case, there's a limitation
on the big the traces can be, in that case, that was one of the core limitation. CTF, we can scale
to huge huge traces, and we're binary traces, and it's self-described with the metadata. So basically,
we get the best of both worlds. We have a structured data type, but it's binary, so it's much faster
and much more compact. And I think that's it. Thank you. Yeah, that's all we have to find out
you