Hi, everyone. My name is Josh. I work at Perkona as a quality engineer, mainly testing databases.
So recently I've been playing around with the GPUs and I was curious like what works,
how can we run multiple models and what are the observations that I had. So this session
will be regarding how you can run multiple models, what methods you have available and what
are the good ways to do that. So the overview of this session will be we will understand why we
need to partition a GPU. We will explore GPU sharing methods, then we will get a use case of
video generation. So we will use a van model and test it across the multiple sharing methods,
then we will see what crashes we encounter and then we will optimize the workload and we will
compare it with the H100 and B200 GPUs. So why should we partition a GPU? I mean there are
four key reasons which I identified number one, some models don't support batching. So you will have
to run particular instance of that model again. Now you cannot get a new GPU all the time and I look
at full resources to it. You need to partition it. So that's one reason. Second is your application
requires separate you know compute capacity of the GPU. So you need to partition it because of
that or you want to sell the compute. So you partition GPU and you sell it like model does.
So there are a couple of companies doing that or the last one which is my favorite you cannot afford
another GPU. So you just have one method of partitioning. You have one GPU to play around with
and you just partition that. So what methods do we have at all disposal? So there are two
key ways in which you can share a GPU. The one is time based. Another is spatial based. So
temporal that is time slicing usually has one process allocating the entire GPU for a certain period
of time. So there is a large context switch. So process one process two process three. This
switch amongst each other as they you know work and the GPU is allocated to all of them.
In the spatial one you have two methods. The first one is MPS. Another one is make. Now
of these two methods MPS is not strict isolation. You have GPU it is been shared by multiple
process. Make however has strict isolation. So you partition the GPU with dedicated
partitions and your process is worked in isolation. So there is no you know problem between
the processes. So yeah this is a table comparing the sharing methods that are available to us.
You can see time slicing has full GPU availability. MPS also has the full GPU availability
but the process is shared sharing the GPU. Make however has each process allocated certain
exposure of the GPUs. So it is not complete GPU sharing. It is partial GPU sharing by each
process. Times slicing has high context overhead. So your process runs for the particular time.
It switches. So it has to put thing in memory again process two comes into the picture. It uses
the GPU. So it is a large context which overhead. MPS has context switching because there are
multiple process running. So certain context which happens. However on make there is no context
which overhead because process runs independently completely isolated from other processes.
So time slicing is like you are renting GPU for certain time. It is time sensitive. So if your
workload is large you cannot run multiple workload together. Resource sensitive is MPS. So you have
particular resource but it is being shared and if a process is using too much resources other
process might fail. So MPS is resource sensitive. Make has fixed resource. So there is no problem
regarding that. Times slicing you can run single workload at a time but it is running at full capacity.
MPS however you can run 48 processes in parallel. Make has fixed size. So the lowest size
you can do is like 7. So there are 7 fixed separate instances that you can run using Miga approach.
So what are the where should we run time slicing? I mean it is good for a workload which can
wait. You do not need it urgently and it takes little time. So those workload are best suited for
time slicing. MPS where if you know the nature of your workload you should focus on MPS because
it uses full GPU and you won't have to worry much about you know partitioning things and
everything like that. Make however works best for the isolated workloads. If you have workload
that requires strict isolation. No other sharing or anything like that you focus on MIG.
So quality of service is guaranteed in two things times slicing because it is full GPU
allocation and in MIG however in MPS there is no guarantee of quality of service. I mean what do
we mean by quality of service here? By this we mean memory bandwidth and SM usage. So that is the
drawback of MPS. So what are the methods which GPUs support which methods? So MPS and MIG is
supported on all enterprise GPUs like MPR, Blackwell and Hopper. They also support MPS method
and MIG. Professional GPUs support MPS. Few MPR and Blackwell based like A6000, Aida and all.
They support MIG partitioning. Provided you have the latest drivers. Then consumer GPUs support MPS
but not MIG. So these are the things you should take into consideration before you try to
start partitioning things because consumer grade GPUs are just not useful and not manageable
with MIG work. So let us see the first method. It is MPS. The full form is multi-processed
service. So essentially it is a service running on your server. So what does this service do?
I mean it is a CUDA implementation. So you have CUDA API using that. There are three things which
are comprising the MPS. The first one is control demand. So this demand is what manages
the MPS server. So you have an MPS server running which is sharing GPU connections with the
clients. Client here is any process. Any process that uses CUDA is essentially the client.
After you have enabled the control demand. So here you can see in the diagram that process A,
B and C. They all pass through service controller and then multiprocessor service is actually
assigning GPU portions to it. So you can see the green portion is assigned to C, purple one to
B and orange one to A. So this is the basic overview of the MPS. So how can we set up MPS?
Well it is pretty straightforward. You first of all select the GPU device. So in case you have four
the numbering might be for the device one it will be zero. Then you have one two three.
So if you have four H100s it will be named accordingly. So first of all we select the GPU
that we want and then we start the MPS department. By default this is installed on all H100s
and the price level servers. You don't need to install anything. It is enabled but the demand
is not started by default. So you start it yourself and then whenever you run any process that
uses CUDA driver it will simply act as a MPS client. So you won't have to do any other thing.
Now let us see the MIG approach. What does MIG actually mean? So the MIG method as you can
see on the figure C there is a GPU. It is partitioned into six instances. So there is this concept
of GPU instance. Now these GPU instances are completely isolated from each other. It is a hardware
based partitioning method where there is no software you know switching things between the processes.
It is enabled and this is how it looks like. So a MIG instance comprises of GPU slices and you can
see in the figure D. It is an example of A100. So there are seven different memory slices. You can
see the number 5 GB. There are eight such partitions of the memory and seven compute slices.
So GPU slices plus GPU engines equals a GPU instance on the left hand side. So GPU slice is basically
a memory slice combination with a compute slice and one memory slice is roughly around one
eighth of the total GPU. Similarly a compute slice is around one seventh of the GPU.
So here for sake of simplicity we are writing essence as compute. So how do slices occur? I mean can
I slice compute first and then slice memory? No. First of all you slice memory. You assign memory
and then you assign the compute to that particular sliced memory. So it has a hierarchy and a step.
So you follow that step to you know create a MIG slice. So what are the combinations that we have?
So on the left hand side is the smallest instance combination where a small memory is selected
and a smallest isolated compute instance is selected. So it is like one G 5 GB instance, one G
means one compute and 5 GB means 5 GB of VRAM. Now this workload is completely isolated from the
point of view of memory because no other block of memory is added into this combination.
It is the smallest combination available and yeah this is a small instance. So size might be
issue for your workload. Then you have a multiple isolated compute instances. Now what does this mean?
You have a memory slice on top which is a combination of 5 or 4 5 GB instances and a one
compute instance is. So this entire structure is partition into second level where one compute
is enabled within the four compute setup. So one C 4 G will actually mean that you have 4 G
per instance but that 4 G is partition and one compute is used from that. So that 4 G is again
partition into four different parts. Yeah so here you can see that memory is still shared between
those four compute. So essentially this might create problems if you have 4 isolated compute
instances but they are sharing the same memory. So your workload might overwhelm the memory
that is 20 GB and you might get some issues. There is another approach where you can have multiple
large chunks of compute as well as memory. This essentially is a single instance with a large
you know GPU compute and memory. So yeah there is one drawback that if your workload is not
that intensive your compute might be wasted or your memory might be wasted. It is sitting idle
so you might not be using that. So what are the overheads in make? I mean you are using the
mega approach you are getting guaranteed quality but there is one problem with this approach.
You are compromising certain things. You will compromise on certain memory leftovers,
certain SM not being utilized and there are certain combinations in which you are essentially
wasting your compute capacity. So this diagram displays an H 100 nick profiles.
As you can see that 1 G has around 16 SMs. It is not exactly 1 7th of 132. So I mean you can see
there are certain things which are being not added. 3 G also it is not exactly 3 times 16.
It is more than that. So yeah there are certain drawbacks of using make.
So how do we create a make-per-cut partition? So as you can see from figure k you are listing
out the GPUs that you have. So we have a device named GPU 0 which is an H 100 ADGB instance.
Then you see what are the profiles that are available to you? What are free profiles available to you?
And then you partition them. So let us see how we can create a make instance.
So first of all we need to enable the make-mode. This mode is not enabled by default.
So you create this make-mode using the first step.
Then on the second step you create a make-instance on the GPU 0.
Hyphen i0 is basically saying that use GPU 0 and create a GPU instance. CGI means GPU
instance. And 9 number that you are seeing is the profile ID. You can see the profile ID on the
figure L. There is it is it is 3 G 40 GB. So you are creating 2 such instances. So 9
is basically 2 profiles of 4 G 40 GB. Sorry not 3 G 40 GB. So you can list individually like
with GPU instance you have and which compute instance you have. You can see that step 3
and step 2. They have order. You first create a GPU instance and then you assign compute
instance to it. So what are the other things that you can do with the make partitioning?
Well if you partition and create make you can enable MPS demand within it. So MPS is basically
a service which can be managed. So yeah you can create a combination like this. You can have variable
workload like P1, P2, P3 running on MPS within a make-instance while you can have a completely
separate isolated workload on make-instance 2. There is one more thing that you should keep in mind
if you have enabled make service first you cannot create MPS. Sorry if you have MPS service
enabled first you cannot create make. There is an order to it so it disable MPS demand and then
you can create make. So let us look at the example that we tested. So when 2.2 model is what we
used. There are 2 models. One is a large 14 billion parameter one. Then there is another one which
is text input based, 5 billion parameter model. The left hand side, the larger one uses prompt
image and audio to generate a video. So I will study for 4-itip pixels how much memory is needed
what is the constant workload. So we know the nature of this workload like it stays constant
late 55 GB VM for a certain duration of time but at the end when the files are being created it
goes and shoots up to 58 GB. So peak means that particular 30 second duration. So there is a medium
parameter model which is the text input one. It generates 720p pixel videos. So we encountered
out of memory. We are testing these things on MPS and you can see in the diagram that B is where
you can monitor the CUDA process. So what I did was I created terminal started the NVIDIA SMI
and kept it in watch. I ran 2 processes in the bottom shell CND just to see what happens.
Two of them were running. Both of them are very large models. It needs 55 GB. Now while both of them
started at the same time they need more resource. Now one process was able to outcompit the other one
and the other one failed. So you can see on the C that we got the out of memory error.
Okay, so what else did we tested? We tested running 3, 5 billion parameter models using MPS.
Where workload was constant? We were getting around 22 GB memory equally distributed by all the
processes but just before the end it crashed because one of them required 33 GB and that another
workload got cancelled. Good thing about this is that it tells you which process is causing the issue
like because of process 2, 2, 1 and 3, 7, 4 things got crashed and this process is exited.
So what were the tests that we found? Well you can see with this graph we were able to run full
GPU baseline tests. Time to video generation was good but it failed in a couple of them.
So this is the overview of the performance test that happened. The last one is if you want to run
a big model you cannot do it on H100. You need a B200. So we also tested that in a B200.
But there is one more method. Like this process head around equally 22 GB usage by 3 processes.
We can run them in certain order in which we can make sure that three of them runs properly.
So yeah this is where we use that. So we know that at the end 30 seconds is where problem occur.
So why not we simply start each process one minute after the other. Like staggering things.
You start certain things first and then wait and process other things in parallel. So we were
able to successfully do that and it took 10 minute and 30 seconds to generate three videos which is
quite quick compared to all the workloads that we have. So changing your workflow strategy can
improve your performance and throughput. Also let us see on the B200. Like B200 has good profiles.
But you can maximum generate three make profiles because even though it is 180 GB it is not equally
partitioned. There are certain combinations which are not used full for our workload and same
goes for the 14 billion. You can only have two. So we tested this thing and we were able to generate
three large videos and it worked in a really good time. So this is the comparison of both methods
on H100 and B200. As you can see the H100 using MPS we were able to run two without any issues
three using staggered approach. We were only able to run one large model but when we used B200 we were
able to three run three of them in parallel without any issues. Well for the smaller model we were
able to run five in parallel with other issues. So that is a good thing about using a big GPU.
Then for the make partitions there was not much improvement. You can only have three
based on the partitioning approach which will be faster but using four it will be a bit slower.
So what is the conclusion? There is no single method which is great. Make partition seems
really good but it is not good at certain cases. For large models it is recommended to use B200.
You can try to fit workload but you will need to optimize your model way more which will compromise
the quality of the output. And yeah that is the last thing. Optimize your model as much as you can.
Also I will be having two more talks. One will be mix specific and another will be monitoring.
So monitoring will be a bit more interesting like how you can monitor the GPUs. What are the key
things? So yeah see you.
Thank you very much and fortunately no time for questions but you can grab this speaker afterwards.