Good evening everyone, my name is Yosh, I work at Perkona as a quality engineer and recently
I have been playing around with the GPUs and this session will be regarding what I learned,
how you can partition using mega approach and what things you can do with that.
So it will include setting things up and understanding how to monitor me as well.
So the overview of this session will be like we will see what are the available sharing
methods, then we will see how mix works, what are the concepts of mix that you should
be aware of if you want to partition a GPU and we will also explore a workload.
So for video generation what were the things that happened for MPS as well as mix partitioning
method.
So yeah then we will explore ways in which you can monitor the mixed instances and then
we will conclude with the suggestions.
So what are the ways in which you can share a GPU, there are broadly two types, temporal
and spatial so you can share GPUs with time like a process which is with other processes
and uses a full GPU so that is a time sharing then there is an MPS where you have
an entire GPU which is shared by processes within it.
So they all work at the same time and they share the GPU resources.
It is a software based approach and then there is make the thing that we will be discussing
today.
So make is an actual isolation of your GPU.
So if you have a large GPU you can partition it in a strict isolated manner and run
processes on it.
So it is an hardware based GPU isolation, what do we mean by hardware based?
It is not actually hardware lines they are there but you enable it using the make commands
and all.
So that is what we mean by the hardware based but it is controlled using the CLI.
So this is a broad comparison like how make is different compared to time slicing MPS.
So GPUs split into isolated portions there is no context which overhead because in
time slicing your entire process which is with other processes so memory and compute
is used.
So there is a lot of switch between the processes MPS is a shared pool and there also there
is some context switching so that is one more thing but in make your process is isolated
completely so it does not need any sharing.
So there is no context switch within that particular process.
Also yeah it is a fixed resource setup so you do not have to worry about time you do not
have to worry about other resources hogging your particular resource.
Also there is some limitation you can have 7 small size slices in case of time slicing
that is like a single workload so you cannot do much with that.
You can quickly switch things in MPS you have many processes that can use a single GPU but
in make you have 7 small slices so if you have a workload that you want to distribute
you can have maximum 7 and based on the size of the workload you can adjust that.
So for strict isolation requirement make is best because it also guarantee quality of
service.
Now what do we mean by quality of service?
You are bandwidth your SM utilisation they all are nearly guaranteed in make partition.
This certain you know things which are not completely entered percent guaranteed so but
most of the time it will work as per expectation.
So yeah so how does make look like?
So on the left hand side you can see a GPU it is partitioned into 7 GPU instances.
So make partitions it into 7 instances so this is an example of the smallest slice of a
Megan instance on the right hand side you can see there is a diagram for NVIDIA A100 GPU.
It has 8 slices of memory and 7 slices of compute.
So your GPU is essentially divided into 2 parts there is a GPU instance which is divided
into 2 parts GPU slices and GPU engines.
So GPU slices consist of those memory slices in the above diagram and compute slices.
GPU engines are a separate thing which is allowed to the GPU instance based on the portion
that you partition so that is another thing.
Now each you can see there are 8 memory slice and 7 compute slice they are not using
exactly 8 divisions or 7 divisions it is almost that we will see further how the slicing
happens.
Yeah so the slice hierarchy is like this you partition memory first so you cut memory and
then you assign compute to it it is not the other way around you cannot assign compute
you cannot slice compute first and then assign memory.
So this is a 2 level hierarchy process so you have to strictly follow that steps.
So what are the partitions that we can do?
So on the left hand side you can see the smallest partitions so we sliced 5GB of memory
which is the smallest available partition in the H1 and A1 and RGPU and then we
alerted 1 compute to it so it is the smallest compute with 1G 5GB as an oven enclosure.
There is also another one where you can share the memory pool with multiple compute instances.
So in the middle one the figure H you can see that there is a huge chunk around 20GB
of memory allocated to 4 compute instances so each compute instance X has a separate compute
instance but they share the memory pool so that is one thing and in last you can have
a large GPU instance where you can have large memory and large compute so it is an isolated
instance but it is bigger compared to the smallest size which is figure G.
So what can happen? I mean the smallest size you might have issues with your workload because
it might require more resource so it is not good for that but for a smaller workload it works
really well because you can have 7 of those instances. On the right hand side you have a big
instance but it is not evenly distributed. What do we mean by not even a utilised sorry?
It is not even a utilised. So if you have a workload which uses only 10GB of memory and two
computes you are 10GB of memory and two computes are wasted on the side so there is a potential
on use of the compute. On the middle one where you can see there are multiple compute sharing
a big memory chunk you can have out of memory issues like all the computes are competing for
memory and eventually some processes collapse because of no memory. So what are the overhead
in make? I mean you are getting this full isolation so there must be something you are compromising.
So you compromise on exact compute divisions. So as I said earlier it was around not exact 7 division.
You can see in an example of H100 the smallest slice which is one compute slice has around 16
asms but overall GPU has around 132 asms so it is not exactly divided by 7 you are missing
around 2 asms. So that is one thing and you can also see on the middle one around 60 asms are
available for 3G and 4G which is not exactly half of one 32 it is just half of 120 so you are
living around 12 asms. So yeah there are a couple of things that you compromise if you are trying
to use make so that is something you should you know keep into consideration. Then let us see how
you can create a partition. I mean we have access to a GPU and we list those available
partitions using this command on the left hand side and figure L. It shows you what are the
divisions that are available to you. Using the profile IDs you can create make profiles.
You can see on the steps that first of all you need to enable the make mode.
GPUs are not enabled by default for make. So you have to enable it you don't have to install
a new utilities for the new GPUs they are by default installed. So you just have to enable this mode.
Once the mode is enabled you can create a GPU instance first for the particular GPU. So you can see
from figure K that we have GPU 0 and we are using GPU 0 in the step 2 to create two compute
instance for profile ID 9. So we got two 3G 40 GB profiles using step 2. So we created our
GPU instance in this second step. Now we will create an assign compute instance to it. So here
we are assigning the entire compute instance to it which is supported by those two GPU instance.
So yeah the third step is that but if you want to be even more specific and divide that
GPU instance to an assign multiple compute instance to it you can do that with the small step
at the bottom. You add GPU ID and then you assign particular profiles that you want.
You can also view the listed compute instance and GPU instance using the commands shown.
Okay so after we create what actually it looks like using NVIDIA SML SMI L command you can see
that there are multiple making instances available. You can see that UUID which we will use
to run our process in the further steps. So this is how it looks. On the figure L you can also see
that there are no free instances available. So we are done with the maximum profile that we possibly
can. So yeah after creation this command can help you in identifying which are available and
which are not available for partitioning. Okay so now let us look at the combinations.
So there are multiple combinations apart from the overhead that we saw earlier.
There are certain combinations of we partitioning that can destroy and waste compute and memory.
So we look into that and we will also see which combinations use all the available profiles
slices. So your GPU is at least used completely apart from the overhead.
So yeah this is one example. So you can see this is an H100 GPU with around 80 GB RAM.
So we divided into two portions three compute instance each and 40 GB each.
So you can see there is one compute instance which is wasted which is not loaded to anyone.
So that is one thing that can happen. If you are partitioning things in certain ways you are missing
out on the compute capacity. So that is one thing. It is around one seventh of your GPUs like
14 percent you are simply just wasting. You are not using it anywhere. So yeah that and yeah
for the memory is similar thing. So if you have seven identical instances with small compute
instances you just have 10 GB of memory wasted. So you are simply using seventh eighth of your
GPU capacity. So yeah you can see from here this is taken from the documentation.
Certain combinations were compute misses out. So and if you observe it properly it is around six.
So whenever you are using the combination of six like three compute instance is three compute
three to one compute instance, three one one one. You are missing out on one compute instance
and based on that you are simply just wasting things. Okay so which combinations don't
leave any slices. These are the combinations which I have listed. There are you can also see
in this table that six is missing. So it is not working out for that particular combination.
Okay so yeah this is the same slides I will let it go. Okay so how can we execute things on
our making instance. We created a make instance now we want to run our things on it. So you can see
using the NVIDIA SMI L command you can view multiple available instances. You use those UIDs
after creating your make instance and then simply export it as a variable. So there is this
visible device is variable where you assign your particular make device UID and then run it.
The highlighted one is for the GPU it is not for the make device. So yeah.
Okay so B200 also has similar things like the one we saw earlier for H100.
We are showing this because we will be showing performance of the video generation model on B200
and H200. So these are the partition methods which are available to us for B200.
Okay so for the good part like video generation. We have around two models that we will be testing
things. 14.1 which is a very large video generation model. It uses prompt image and audio
to generate a video. There is another one which is text to video generator which is a smaller model
of 5 billion parameter. Well it is not small but for video generation contact it is like a smaller one.
So yeah we have also kept an observed the peak and constant V RAM utilization for this. So they
are mentioned over here. Okay so what did we observed? Like the baseline compute is for H100 GPU.
At the end there is an observation where we simply run make instances on B200 and show it along with
this. So as you can see in certain workloads it takes some time and you are only able to run one of them.
While in other workloads which is on B200 you are able to run multiple models in parallel.
So I will show you even detailed one the next slide. So yeah so how many images were we able to
generate? Like using the MPS method isolation was not guaranteed but it it worked. It worked around
two instances and three staggered manner. So we run one process then we wait then for one minute
then another then again we run the third one. So it worked in that manner. For the make part
we had guaranteed isolation for H100 and B200. So there was no need of you know doing things but
in the case of larger GPU B200 we might assume that we will have a lot more slices available but
that's not the case. The slices combination that we have are quite more than we require in terms
of memory. So in case of 5 billion parameter model for B200 make we are wasting around 10
GB of memory for generating one model in parallel. So that's certain combination where things don't
work out but that is one more thing. You cannot allow other work to be run on this unused GPU.
But for MPS part you can do it. So that is cons to that but isolation is completely guaranteed
in the make process. You won't have any issues or any failures. So possible failures. So I tried
running a variable workload where a model uses more memory and then stops. So it uses for like
5 seconds. It's a small time but still it fails. So you can see that for a variable workload and
if the variance is very high even for a shorter time this make partitioning is not good idea.
So yeah and also for the single GI instance and multiple GI instance O M occurs in the similar fashion.
Okay so now we have make setup and we understood what it is. How can we monitor the instances?
So there are two ways. First is the NVIDIA SMI if you want to click quick CLI based setup.
You can monitor it using watch and see but there is another one. The DCGM exporter.
This exporter collects matrix just like any other exporter like node exporter and then you can combine
it with Prometheus and Grafana. So yeah this is the monitoring setup. What else can we do?
Like we can slice and create make but we can also combine things together.
Like you can have a make instance which can run MPS demand because MPS is essentially a service
and you simply enable it within a make instance and it will work flawlessly. But there is one more thing.
You cannot run make after enabling MPS. You need to disable MPS and then you can create make.
So that's one thing. Yeah so conclusion for high variance workloads provision profiles with high
buffer. So yeah this will result in memory, news memory and a lot of things but still it will
provide isolation and we will get into it. Then for stable workloads make works well for high
variance use MPS whenever possible don't rely too much on make because you will simply waste
some resources which might be used by other processes. And for the large video generation
models H100 is not enough. You need B200 at least for generating at least two to three videos
in parallel. So yeah that's that's all. Thank you very much.
Yeah you have one question.
I think it's because of the architecture. Yeah can you repeat the question again.
Yeah so the question is that why do we have certain partitions and when can we you know expect certain
resources to be utilized properly in the partitions? Well I don't have right answer but it is because
of the architecture. So I believe if NVIDIA improves on the GPU architecture we might get even better
profiles which are more dynamic. So yeah hopefully that happens. Yeah we have another question.
Are the profiles same for all the period teams like H100, H100, H100, H100 do they all have
exactly same. Yeah combinations available for all GPUs same or do different GPUs have different
available profiles. Well the answer is there are certain combinations which are similar for certain
architectures but for different architectures they have different combinations available.
So yeah also one more question. Yeah I haven't explored AMD or Intel I have worked with
NVIDIA hopefully. Okay so we have two more questions. Yeah.
Can you repeat speak loudly a bit.
Yeah so I tried generating videos on the different make partitions compared to the
MPs. The make ones had bit delays. Now the reason was it was not using complete SMs in these
partitions because certain SMs were left over and these video generation models are SM intensive
apart from the RAM requirement. So that caused around two to three minutes of delays in generating
them. So I generated around one minute not one minute 40 seconds of videos and it was taking
delays. Okay we have another question.
Okay so the question was like do AMD have things? Yeah they do. So one person from the audience
explained what are the ways in which they can be partition. So thanks thank you for that.
Okay we have one more question.
Yeah so the question is like can we have multiple GPUs where you can provision workload using
make partitions? Yes it can only be done if you have 4x H100s available. So in this case it's
in single instance but you can orchestrate your workload by different make partitions and enabling
those partitions and turning them. No there are many available out there like 4x H100
data also so on for B200 also there are certain combinations. It depends on the provider.
So I use Worlda. So they are also providing combinations which are in smaller portions.
So I use one combination.
Thank you.