A little about us before we get started. I'm a Nanya. I'm a software engineer at Kufara.
I was one of the founding engineers on tempo, which was our just a bit of tracing back
in. And now I work on the machine learning team. And based in Seattle. And yeah, I'll
pass it off to Mark Twentries. Yeah. My name is Mark Twentries. I live in Berlin.
I'm working in Mingerafana in the Belarus Quatt. And this is our first picture together.
And we didn't know. And it's me there. And Nanya is there.
Yeah. And in fact, it's our only picture together in the same frame.
We're in. Cool. So yeah, I'll set this stage for our talk. We'll split it roughly in two
halves. In the first half, we'll talk about the challenges we're deploying GPUs to production.
And the current solutions out there to monitor GPU performance. And in the second half,
we'll talk about our proposed solution to monitor GPU performance using EPPFN bail-on.
Cool. And with that, let's meet Randy. So Randy's room in the sense that he has a YouTube
channel where he does comedy. But for the purposes of this talk, we'll say Randy is an entrepreneur
who's raised money to start an AI research lab. And with this money, Randy's bought a bunch
of GPUs and deployed a fleet. 5 GPUs is not a fleet, but a few more. And these GPUs have been
passed off for different teams for experimentation. So typically the way this works is we have
a scheduler and different teams sort of submit tasks to the scheduler. Some of these could be
really long running tasks like model training and they can span like hours and days. While
others could be short-lived tasks like model live inference. And this scheduler decides
with GPUs are free and assigns these tasks to these GPUs. But couple days after deploying
these teams start complaining about either crashing trainings, like repeatedly crashing
trainings or very slow live inference. And turns out this is more common than you'd think.
So GPUs are incredibly complex pieces of hardware compared to CPUs. They have a lot more
failure modes. This is from the Lama 3 paper released by Meta in July last year where they talk
about how they train the model on like 16,000 GPUs over a course of 50 something days. And they had
500 odd errors like 500 odd instances where the training just stopped. And it's sort of categorized
that and you can see that GPUs right there are the top. There's a lot of, there's I think 58%
of the stalls were because of GPU issues. And turns out this number is actually pretty standard like
1 to 3% GPUs is failed like the overheat and fall off the bus. But yeah there's lots of other issues
with this. Spoiler alert, EBPF doesn't help with tracking hardware related errors. So
skip couple sets. But yeah so coming to software side right like so this is typically how we
write a program that's geared towards GPUs, a CUDA software as you'd say. So we first load data into
the CPU memory and then we allocate memory on the GPU side for the device. And then we transfer
bits of that data that we want for our computation over to the GPU. Then we launch kernels which
are basically like functions or basic computations that you launch on the GPU. They could be like
matrix multiplications or softmax, lot of math that you launch on the GPUs. And the results
of those computation are then copied back over to the CPU. So the tail DR is that CPUs the
orchestrator for all of the GPU tasks. So kernels are launched from the CPU so it's important to know
how many of those were launched where the dependencies between kernels was the output of one
kernel being consumed by the other and did that cost load downs. And then memory is also
allocated and deallocated from the CPU side. So it's kind of important to know how much was
allocated was it freed up before the next operation was done. And our data transfer happening
async to the computation right typically there should be done async while other computation
tasks are underway. So like all of us would want a monitor GPU performance randomly goes to the
internet and searches for how do I how do I do this right and what are the solutions out there.
And the the first thing that you kind of read is to set up hardware metrics and there's some
pretty solid exporters out there. The first screenshot is from Nvidia's DCGM exporter which is
transferred data center GPU manager and it plugs into hardware APIs of the GPU and reports
some pretty interesting metrics like the temperature temperature tracking is pretty important.
Flex GPUs can overheat and die and also like frame by free usage GPU utilization all of
this stuff. There's also a slurm job exporter which is linked from this slide. So slurm is another
container orchestration system which is more popular in the HPC world. So most of the top
super computers would use slurm and then this screenshot is from Azure HPC node health which is
an open source repository of node health checks that run that are targeted towards GPU. So similar
to the health checks that we have in Kubernetes but specifically for GPUs they'll check things like
anxiety errors and if there's any errors related to the GPU they won't bring it online so that the
cleaning doesn't stall. With the problem with hardware metrics is that while they're helpful
to know with GPU or with job fail and the failure states it doesn't lead you to the root cause of
the problem. So again next advice is to use profiling systems and again there's some pretty solid
profiling systems out there. This is actually a good point in time to get a show of hands like how
many of you all are using GPUs in production. So there's a couple so you might be familiar with
this if yeah maybe you're using it to profile your CUDA programs. So the first one is Nvidia
and site systems and it helps break down all of your Python functions into the operations that are
launched on the GPU and the second one is the PyTorch profiler and this also kind of helps break down
how much time was spent in different parts of your code. But these are pretty complex to use like
this is an example of the command that found in one of the user forums and it's like you can kind
of start seeing the issues with this. First off every team in Randy's company now needs to learn
how to use these that are learn which libraries to profile whether or not to sample the CPU
call stack and by the way if you enable CPU call stack sampling you have a performance overhead
because the profiler is running in the same process as your main Python code and people have seen
up to 2x the performance overhead like it takes twice as long to run your code if you have CPU
called stack sampling enabled and then finally the output is on your local machine that you have to
load into a GUI it's not like a platform capability that can be provided to multiple teams
and if you're in more complex if you have a multi GPU setup so each of those would generate their own
output files. So again there's performance overhead and in terms of like PyTorch Profiler you have
to manually instrument the code to enable profiling there's also lack of CPU context before and
after GPU events so that's also something that we want to highlight. And finally like what
if Randy runs out of money and is like I cannot afford Nvidia GPUs anymore we're going to switch
to AMD's and you know now with the deep C can Nvidia stuff going on I don't know what the
price of GPUs is going to be but suppose we swap out some of these for AMD's right like now
your whole profiling architecture each of these teams learned how to use the Nvidia Profiler
now you have to learn all of that again for the AMD tool set and all of that. So these were the
problems that we decided to tackle at Profano and yeah and now we're going to talk about how
we're going to use bailer which is Defano's EVPF based open source auto instrumentation tool
which is also going to be part of open telemetry and how we're going to use this to monitor the
performance of GPUs and I'll pass it off to Mark and I'll try to. Thank you. Yeah so what
are the advantages of using EVPF in this context? First of all as the instrumentation so that means
that we can use deploy bailer in our classes that we use GPU and we're going to have
by default automatic instrumentation in for every CUDA call that happens in the in the system.
It is framework agnostic so it means that whether you use PyTorch or any other machine learning
libraries does a mother since you're if you're using CUDA it's all good and it has a lower
overhead you probably might know but EVPF is quite fast we haven't measured the performance yet
but we assume that is lower overhead like the other probes that we have in bailer how these how
these works please take it with a pinch of salt is an experimental feature
things my broken but the first of all we have to identify the important CUDA calls that
we have available and we got a small subset of them and just to test and then we start in
right in some probes they're quite straightforward and so complicated and then we start we can
get in data and see which metrics and which labels we can start creating from them we have to do
some process of expecting the CUDA libraries and a model discovery that's going to be necessary
to get the symbols and the names of the kernels and it's important to mention that we have access
to the CPU context before and after the GPU call so we are not able to instrument the GPU
but we're going to be able to instrument every time that the CPU is doing a call to a GPU
this is how it works roughly imagine that we have a prompt to our favorite AI assistant
and it goes to an LLM the LLM and needs to do some sort of operation and for that it triggers
this operation CUDA launch kernel and this typically goes to a GPU and it does some calculation
and it returns back to the CPU. Baylight's capable to add a proof to inspect what's going on there
in the CUDA launch kernel operation and we create this metric GPU kernel launch
costoto is ingested by promithios or a pentalometry to later be visualizing in graphana
this is the first function that we try to instrument is the CUDA launch kernel we can see that we
have the funk offset here that is going to be used later to to get the name of the kernel and we
have this grid and block coordinates that are useful in order to figure out the cardinality
of the operation we're going to talk about this in a minute here we can see in promithios the visualization
of this of this function sorry of this metric and we can like track which which of them are
more called another and since Baylight's Kubernetes for citizen we are able to automatically
Kubernetes method decoration to all the metrics so we are able to figure out for this metric
for this operation in which port and which name spaces running so we can track which teams
are running more operations than other here we have a graphana dashboard
visualizing the dimensions of the CUDA launch kernel function and in the image above
it's the panel above we can see like the average and of the cardinality or the
grid cardinality and the block cardinality so we can identify which kernel functions are using more
more using more resources in our GPU in terms of blocks and grid in the panel below we can see
the rate of this cardinality how it grows over time this is right away like running a prompt
in a loop but if we run if we run a second prompt we can see in a spike in the cardinality
over time which makes sense because we just need to do more operations we also are tracking the
CUDA malloc but this is not so useful because this this call is only when you load the model
the model on the bootstrap but we do have the CUDA main copy and it allows us to identify
host to device or device to host which in this case is CPU to GPU and GPU to CPU
and we can look at things like for example this panel where on the left side we can see that
again like over time there's was the quays that we have before the amount of kilobytes
that they're transferred to from the CPU to the GPU and the other way around and of course
is usually we usually send more data to the GPU than the other way around because
the GPU took that typically only returns one parameter or one result and the CPU has to send
the whole text of your query for example and on the right side we can see how the rate
of the CUDA kernel launches is correlated to the memory allocations
we can also do profiling this is a piece of code that we have in the in our first probe that we
show to capture the stack trace thanks to this bbf helper and yeah we can see some interesting
stuff like piterch kernels and vlm kernels because we have support for that but the rest of the
stack is gone that's because we use frame this is using frame pointers and they are optimized
so they are gone this is how we want how we envision to have our CPU profiling for this operation
and this is like a screenshot of gdb our idea is to use the auto
open telemetry profiler the native stack on wind it's a bit tricky to make it work with in our
platform but we are working progress and that's it's going to give us the full picture
some of the limitations with a bf approach there is not a information available on kernel execution
time that means that we cannot for example do the latency of a kernel function execution
and we don't have access to the GPU hardware itself so we cannot measure things like the temperature
just to recap the idea here is to close the gap between traditional GPU monitoring
and modern monitoring solutions like khafana and baila in the future we would like to
support more architectures like as an audience say like I am indeed for example
but at the same time we don't want to work on instrument every lm and every framework out there
we also like to capture the context before and after a GPU call what does it mean
like currently we are just generating a metric but we can also generate the whole trace
when the request is made the time that it take to the request like the prompt and how these impacts
in the number of kernel executions that it has
year prompt so we could also do a cost association to like depending on on which operation are you
running we could track the amount of GPU intensity that is that operation and finally we would like
to instrument more cool operations and I think that's it thank you
there are any questions that we can do do we have time for questions
yeah that's something we'd like to try
okay it's it's easy so that is the problem the way the kernels are launched there's no
way to track there are two times but maybe we can try a couple other approaches see if the
books any other questions
so okay we have worked running oh sorry so the question was how long have we been running it in
production so right now we're using it to monitor just like a single GPU that we have running
some internal queries and stuff we don't have a large scale GPU deployment at Grafana if there's
anyone in the crowd that does please reach out we can collaborate and work on this and yeah so
right now it's been couple months since we've been running it internally on like one GPU but that's
not the ideal use case thank you
you