We see that this workout has been running for nearly two weeks and I checked today and
they are still running.
So it has been more than three weeks.
The GPU utilization has been so high.
On the contrary, I have some examples of interactive workout here.
These are two instances of Oama and we can see that the notebooks have pretty
varying utilization here.
For example, this notebook has zero average GPU utilization in past days so that
is quite waste.
And the same applies to Oama instances because they just reply to user
questions at some times.
And this is a closer comparison of two different notebooks of two different
users.
The first users notebook, we can see that the GPU and CPU utilization happen at the same time
for some reason.
And there are quite gaps between that.
For example, this is one whole day.
So the user has created the notebook but is not using it all the time and it is waste.
So this user, however, we can see that even there are times when no resources are used.
For example, here it's not very well visible, maybe.
But this is GPU utilization, then the GPU is idle but CPU is used.
Which means that the user is somehow interacting with the notebook.
And this means that we cannot just decide based on GPU utilization to maybe
kill the notebook or somehow remove this workload.
Because first it's interactive, user might interact with it later.
And second, GPU utilization itself is not the marker of whether it's used or not.
However, we might want to tell user one that he should be using the jobs or something similar.
I don't know why he's using the notebooks.
So the obvious observation is that we experience low resource utilization.
And I think that many customers and administrators know that there is low resource utilization.
But especially with GPU utilization, it's painful because it's expensive.
And there are some existing solutions such as time slicing or multi-instance GPUs.
Which allow us to maybe use GPU more, but it is not possible to maybe kill the workload using the GPU
and retain the progress on the GPU.
And that's why we suggest using GPU checkpoint restore, which does not help this problem.
Thank you, Ricky.
So transparent checkpoint, as Victor mentioned, allows us to essentially preempt running applications
and resume their execution either on the same or different notes.
This allows us to, for example, check, essentially say, yes.
So this allows us to improve the utilization by essentially pre-ortizing different workloads
and preserving their state when, for example, they have different priority.
The challenge, however, in implementing GPU checkpoint,
is that we have to save the internal state of the GPU and restore it back.
And this is often proprietary and we have to provide low performance overhead
and make sure that the application is inconsistent state.
So there have been several methods proposed in the literature on how to checkpoint and restore GPU state.
And the most commonly used one is what is known as device proxy interception.
For essentially, it is a line called a shared library that is pre-loaded for the application
and replaces every device API call with essentially client server mechanism that keeps a record of,
essentially every memory transfer and every API call made to the GPU.
And this allows us to separate the CPU state from the GPU state and during restore,
we can replay the API calls and restore the application.
There are several challenges with this approach.
First, this device proxy mechanism has to be able to load the code kernels
or the GPU kernels from the binary, which means it has to implement the same
mechanisms used, for example, by the Kudarante.
It adds an interception overhead for every API call and keeps a record of every cost to device memory transfer.
And it requires also a more specific implementation.
So it doesn't work for all GPUs.
You have to essentially implement the GPU support yourself.
And one of the main challenges is that it requires dynamic linking.
So, for example, if you want to use PyTorch with this mechanism,
you have to recompile PyTorch with dynamic linking.
And we did a simple experiment with an open source tool called cricket
that implements this checkpointing mechanism.
And we can observe that essentially loading a starting the PyTorch application.
For example, it's much slower than essentially the standard execution.
And every API code that is used during a new network training essentially has a lot
of overhead.
So, we are exploring a new mechanism called essentially relying on the GPU driver
to provide the checkpoint restore mechanism that is integrated into two-code
Creole.
Creole stands for checkpoint restore user space.
It's an open source tool that allows us to save and restore the state-of-links processes.
And it has been integrated into several container-and-times.
Such as Docker, Podmon, and more recently in Kubernetes.
And it provides GPU checkpointing support with the plugin mechanism for both
ambient and video GPUs.
The way it works is using essentially the CUDA plugin allows
to track point and video applications.
And it uses a two-code checkpoint, which was recently released by Nvidia,
which communicates with the driver and essentially stops the application from running
on the GPU and moves the state of the application to host memory.
And this allows us to create a unified checkpoint that contains both the CPU and GPU state.
The advantages of this approach is that it doesn't require preloading of shared libraries,
which is very helpful for containers because essentially injecting the library into the container
and setting up the environment variables is challenging.
It also doesn't introduce overhead of API coin perceptions and recording the memory transfers.
And it works both static and dynamic linking.
And because it's implemented in the GPU driver itself, it works for all types of GPUs.
The way this mechanism is intended.
Yes?
Yes, only works for Nvidia.
We have a separate plugin for AMD GPUs.
This is using the essentially an implementation in the Linux kernel.
I have a separate slide, but due to time constraints, I focus on the Nvidia implementation.
And the integration with Kubernetes is based on Adrian's work, Adrian Reber,
who implemented support for ANZIP checkpoint in Kubernetes recently.
And since these plugins are integrated with Kubernetes,
we don't have to make additional changes.
For example, to preload libraries or similar additional features.
And checkpoint itself is essentially working on individual containers.
So it allows us to save the state of individual containers,
we can pause and restore them later on.
And I have a short demo.
So in this demo, we have a Kubernetes instance running,
essentially opened with UI with Olamma and Jupyter notebook.
So in the Jupyter notebook, we have essentially a neural network training
that has high GPU utilization.
But it will be running for a long time.
So it's a low-prior task that essentially occupies about
8 gigabytes of GPU memory and 100% of GPU utilization.
And we have two users using something similar to tragedy,
where we can essentially perform model inference.
But because the GPU is over 100% utilized,
these requests are very slow and takes long time to get response back.
So this is a common problem.
These type of applications use a user layer require low response time.
And optimizing the way this model's response is one of the main challenges today.
So we have a checkpoint mechanism that allows us to essentially
install the training task, the Jupyter notebook,
and then move the state into host memory.
This is what the checkpoint showscriptives.
And this allows us for the inference jobs to complete much quicker
because that essentially we can preempt the training task
and then we can resume it later on with the rest of script.
And in our evaluation, we focused on what are the main factors that add latency
and how does this mechanism scale with multiple GPU devices?
We observe that because large language models usually have a lot of GPU state.
Essentially moving the state from the GPU to host memory is fairly fast.
This is what we can see for with a 100 on the left graph.
And then saving this essentially the whole state to disk is usually very slow.
And restoring it back is also, it takes long time.
We in this graph we can observe that the blue bars show the total restore time.
So this is the time we it takes to essentially load the GPU state in host memory
and then from host memory back into the GPU.
And the GPU restore time is in orange and the unlocked time is essentially
synchronization mechanism that allows us to restore the application consistent state
and this is negligible.
The checkpoint size depends on essentially the model parameters
and the quantization of the model.
And we can observe that most like 90% of the checkpoint size is GPU state
and it could be up to 97% for large models.
I just want to take a moment to acknowledge other people who contribute to this work.
So Andre from Google cases and Steven from the NVATM and Felix and his team from Andy
and Adrian who contributed with the Kubernetes implementation and
look us with providing access to the two Kubernetes clusters.
And thank you so much for coming so early today.
And in summary we present fully transparent GPU track pointy mechanism
that works both with Andy and NVG GPUs and has been integrated with Kubernetes.
And we will be have transfer of any questions.
Can you take the mic so we get the questions on the recording as well.
Any questions for Redstone and then in Victoria?
Yeah.
Well, please one, don't repeat the question.
So first part of the question is to what extent does this checkpoint mechanism
support checkpoint networking protocols or networking interactions?
And second part of the question is to what extent do you have to restart on the same GPU
or can you use let's say another Nvidia type of GPU and restart there?
Yeah.
Thanks so much for the question.
So for networking.
So we can about stateless protocol such as TCP.
UDP is not stateless protocol.
So we can simply check on restore UDP sockets.
For TCP sockets, the kernel has essentially a functionality called TCP repair
that essentially allows us to create a TCP socket, set it in the correct state
and then resume essentially the TCP connection.
We have a locking mechanism that essentially allows us to drop incoming packets
so that essentially when the client is not running, when the socket is closed,
the kernel doesn't send responses back.
So the connection stays open.
So there are a few mechanisms that have been introduced to handle network connections
and for different types of GPUs.
So we talk with Stephen and he says about essentially if it might be possible to migrate
for example from 8100 to a 100 GPU,
but this will be very difficult because essentially the way it's implemented.
It's very different for every GPU.
But it is possible to migrate from essentially between different GPUs as long as they are the same.
And you have the same number of GPUs.
More questions.
Hey, thank you. Great talk.
I have a question. Do you have any insight on when to checkpoint?
Like, what's the trigger point that you would check point a job
and then restore it obviously when there's interaction going,
but when to know when to check point.
I mean, sometimes maybe you're just before the user then interact again with the job, right?
Yeah, so I mean, there are many research papers on this topic.
So there are essentially some of the papers focused on providing photo neurons.
So essentially periodically check point and then using the latest checkpoint to recover in the case of failure.
Microsoft recently published a paper called Just In Time Checkpointing,
which aims to essentially optimize this mechanism.
And there's also serverless use cases.
For example, it's much cheaper to run this type of inference workloads in serverless platforms.
And then checkpoints are usually used to decrease the start-up times of inference jobs.
Did that answer your question?
More questions?
So another question is a little bit more on the amount of data.
Do you do anything in terms of data redundancy reductions?
Like, if you compute a multiple GPUs often the workload contains the same amount of data,
is there some way to basically just drop the stuff that's similar on the GPUs
or is it just take the data from the GPU?
Can you continue from there?
Yes. So we evaluated scalability with data parallel workloads.
This is essentially when we have the model using multiple GPUs and
separating, for example, the data set to run on multiple GPUs.
And this scales linearly.
Essentially, we have to save the state of every GPU when you have multiple GPUs and
restore it back.
The mechanism that we're currently looking at is compression.
So compression is very effective.
We have seen some promising results.
And this usually allows us to, it's the most efficient way of
compressing the data, reducing the size of the checkpoint.
And there is a research paper that will appear later this year discussing this as well.
Any more questions?
So this was now for Nvidia and AMD GPUs.
In our environment, we were interested in doing something same,
but totally different environment, which is an embedded software.
And we don't have that.
Usually, other type of GPUs are Mali or Broadcom video course.
So if you want to extend this, have this mechanism to work on all the GPUs.
What would you advise?
How to approach that and in your presentation?
I had the impression that it was an API approach in the beginning with AMD.
And then there's the approach that you do with Nvidia, which is more the driver.
It's held that preparing the data structures or something.
What will you advise?
Yes.
So the API mechanism is the current state of the art.
So this is, for example, what Microsoft are using internally.
They also announced that this is the mechanism that he used, for example,
to run charge IP key and to provide photo learns for their training workloads.
And we're presenting a different approach.
So our approach doesn't require this API interception mechanism.
And it's based on what Nvidia have introduced recently into the 4D GPU drivers,
and what AMD contributed to the pre-project.
And in terms of different GPUs, essentially you have to,
so what I can say is that I tested the GPU checkpoint mechanism with Nvidia
with about 10 different GPUs and confirm that it works.
But yeah, you just have to see and run it and see if it works for the GPUs that you have.
But so for us, it's not Nvidia.
It's another GPU.
So we would have to get somehow get the vendor doing a PUC and trying to get
replicate what Nvidia is doing.
Okay.
Yeah.
For the GPUs of 4D, you need to find the signal.
If you want additional GPUs of 4D, you need to find the support.
Because we, in theory, upstream, have seen that the GPU part is something you cannot do as an open source.
But you can do as an open source implementation,
but you need to have the inside knowledge of the GPU without that it's,
I would say it's impossible to unique the vendor.
Yeah.
What question?
I'm curious about the, the Kubernetes, sorry, the interactive Jupyter users.
Yes.
And if you've been using this mechanism, how do you let them know that they're being preempted
and when it's going to be available again, and obviously they might be a bit surprised.
Yeah.
Not using exactly GPU checkpointing right now, because it's very new,
and we haven't tested it for like production use case, I would say.
But what we already do is, like, stupid killing of this workload,
and we let people know in advance with like regular automatic emails.
We are monitoring the usage, like in Grafana,
and we set up some politics based on our own decision that like if you are,
I don't for two days, then like we are killing you.
And because we have a email for every user, we are always in the name of it.
If you want download your data, just like save something,
because we're going to do this.
And it has been very helpful, and originally, like a year ago,
we had these politics on seven days, and very quickly decreased it to one day,
because it was not manageable this way.
And we would rather prefer now to start using checkpointing,
because we know that for some people there,
progress is important, but if there are somewhere on holidays or what,
they don't check for emails, and we get automatic replies back that they are not checking emails.
So it's unfortunate for them, and that would be better.