Beyond NVIDIA SMI. So, it will be a session regarding a beginner who is exploring GPUs
and what are the pitfalls that you encounter. So, I work at Perkorn as a quality engineer
and recently due to the rise of GPU and AI I was curious about GPUs like how can I
measure how much GPU performance is going on. So, that was the core intention of you know
going this session. So, we will see how GPUs are like the fundamentals of the GPU common
monitoring mistakes then we will look at an example how you can identify potential left over potential
of your GPU then we will understand the key matrix which are needed then we will go into the
workload specific matrix like what workloads specific matrix you need to make sure are working
and then we will see some utilities which are good. So, yeah. So, GPUs are primary
comprised of compute memory and decoders. So, unlike CPUs compute portion of GPUs are very large
if you want to simplify things and there are certain decoders specific to certain functions.
Compute is the main thing here and streaming multiprocessors are a vague way of saying that
your computes are you know running. So, it is a portion which is executing the computations
based on the data types that you have. So, yeah there are three common mistakes that I found
relying too much on NVIDIA SMI it is a basic utility that you run and you see okay I am getting
and dead percent GPU utilization I am getting this much DRM utilization. I mean it is a good thing
you can use it for certain workloads but it is not saying you all picture. So, to identify that
you need to monitor other matrix. So, there are three matrix which I have also able to identify
the tensor core matrix then SM utilization and DRM. So, yeah the combination of these tells you
the picture like what actually your workload is doing. So, let us look at one example we are
watching NVIDIA SMI and we are seeing one hundred percent all the time. So, obviously we think like
CPUs it might be using a hundred percent of my GPU but it is actually wrong if you look at only
this metric and try to you know by new GPUs you will lose a lot of money. So, it is not a reliable
metric as such. So, one such example is this I ran an FP32 workload on the left hand side
and FP16 on the right hand side. You can see that NVIDIA SMI showing me a hundred percent on both
the cases but is it actually true? I mean why is it showing the same thing for both different
workloads. Does that mean that I can not run much more workload on it. So, that is what we will
see. So, yeah this is the case of idle tensor course where I am using the same set of that was
previously done but we are showing what actually is going on in details. So, that is this utility
called DCGMI. So, it is a data center monitoring tool by NVIDIA certain portion of it is open source
there are certain methods which they are not releasing to the public. So, that is proprietary
but it is the exporters and all those things you can basically contribute to the project. So,
it is an open core utility and you can see that in the first case there is zero tensor core
utilization but on the other it is using completely. Now, why is this happening? So, by default on
H 100s and even on certain GPUs F p 32 is not tensor supported. So, it is using CUDA course
instead of the tensor course. So, this behavior is due to that and on the second one when we ran
around the F p 16 workload it is using the tensor course which means you are actually getting
more out of your GPU compared to the previous one. So, relying only on 100 percent utilization
is a bad idea if you are trying to you know do this. So, now we saw we can conclude that GPU
utilization is not same as GPU efficiency because in this picture you can see a tensor core
example tensor core has multiple supported data types that you can execute apart from the CUDA course.
So, you need to understand your nature there work your load that is be executed what I exactly
data types they are using can be used the data types which tensor core supports. So, yeah
but what are tensor cores actually? So, they are basically matrix multiplication support
hardware implementation by Nvidia. So, the hardware level they are doing matrix multiplication
and that is what drives the entire AI workload. So, that is one thing that you should take into consideration.
So, 100 SMI what exactly does that mean? It is simply measures how many SMs are doing work like
it simply measures if SMs are doing work not exactly how many are doing work it does not give you
much activity details like SM activity or tensor core usage. So, it is not good idea if you want
much details. So, how exactly is this utility working? So, you instead of a data center package
then you start a service which is a Nvidia DCGM service it runs in the back end it collects the
matrix and yeah you can monitor your matrix using the matrix code. So, DCGM has a certain set of
codes that you can use to monitor this thing and it checks things at a certain duration of time.
So, it is a snapshot of a time. So, it averages out for certain matrix.
Okay. So, what are the matrix that we should measure? So, first of all if you are running an
AI inference workload you should focus more on the tensor core utilization because it is the
performance driven thing and if you are not using that you are leaving too much at the table.
Then there is memory bandwidth there is SM occupancy, SMI activity and estimation of the
team drops. So, the performance benchmark that Nvidia gives you how much are you actually using.
So, you can get some estimation of the team drops rather than writing code inside your
models and all those things. So, let us this is a diagram of tensor course. So, you can see
there are multiple matrix multiplication as well as additions happening at the same time. So,
this is something that they implemented in their hardware and if you are using that you are
speeding up your inference. So, this is the thing that you should take into consideration.
So, one SM consists of multiple tensor course. So, this is just a portion of SM there are many
more in a GPU. So, yeah. So, yeah. So, yeah as I talked earlier the potential performance
improvement. So, they have performance benchmarks and you can see that particular data type
workload how much T-flops are expected, how much you can you know get out of it. So, if you are
using a simple CUDA course setup of F 16 you are getting only 120 T-flops but if you use it
using tensor core you are getting around 1000 T-flops. It is like 8 to 10 times more faster. So,
that is the thing enable tensor course whenever possible. So, how can we monitor it? So, there are
certain codes that we discussed earlier. So, these are the matrix which are collected by DCGM
and they have certain codes what they actually check is written on the right hand side. So, yeah
the matrix multiplication is the key thing in case of tensors. So, you have simple matrix
multiplication integral F 16 which is half bit and double precision that is F 64. So, these are
the matrix that you should focus on and look at it. So, this is an example of an F 16 workload.
You can see that tensor course are being used but we do not know exactly what tensor course are
using. So, we get details using certain codes. So, yeah this is an F 16 example. So, you can say F
P 16 and tensor usage are same because they are using the same particular thing. So, tensor is
basically a generic overview of your tensor core uses but the other three matrix are
detailed view of those particular things. So, yeah this is another example of F 64 workload. So,
you can see double precision is something which is being shown similarly for intay to workload.
So, what are actually SM's isometric that we are looking at. So, everything executes as a thread.
So, the block of threads is a combination called wrap and warp basically run on the streaming
multiprocessors. So, this is something that you should see. So, SM occupancy is a metric that you should
be aware of. So, it is like SM's are occupied. The things are about to run but what actually
is running is seen using SM activity metric. So, these are the two things. Now, one thing that you
should keep in mind is that these matrix are averaged out. So, you do not get exact value for SM.
It is averaged over all the SM's that are available. So, this is an example of an high occupancy
you can see that SM activity is around 0.97. But, SM occupancy is like 0.66. So, this is actually
doing some work if you try to infer from this. A low occupancy means that memory is being used
but the SM's that you are seeing right. They are not doing much work. They are sitting idle
for things. So, that is one inference that you can do from this. So, how can we measure T
flops. So, as we saw like certain things are being executed using SM's. So, we can multiply
those percentages. You can get idea like what exactly is happening using the peak T flops which
are given in the benchmark. So, you take the peak T flops, you multiply it with the SM activity
as well as SM occupancy and then you multiply it by the tenser activity that you have. So,
it will give you a brief idea an estimated idea you do not get exact value out of it. But,
you can just identify that this thing is doing this portion of the given benchmarks.
Where this might not work is for the burstable workload like if you have very variance in the
workload this is not a good idea to do. But, for the stable workloads this method will probably
give you an good estimation. So, yeah, things we should keep in mind by themselves,
GPUs are useless, you need CPUs and CPUs are transferring things to the GPUs. Now, the thing is
that this transfer is the bottleneck nowadays. You can get more compute, but the transfer of the
data from the CPU memory to the GPU memory is where things are breaking. So, once such observation
you might see is that you get high SM occupancy, low SM activity and a high VRAM. What this means is
that you have so much stuff and so much activity, but you do not do things. So, SMs are occupied,
but they are doing very less work. That is what you can identify from this. And this is okay.
If you get this kind of workload it is assumed that it is an inference workload.
Inference workloads do not use your GPUs to the fullest in terms of compute.
Yeah, also the DCGMI utility that we saw has one more cool feature like you can do certain profiles
so what do we mean by profile like for NVIDIA SMI you can get brief idea about things.
You can see the temperature as well, but you cannot monitor things using CLI. This utility has
in built that setup. So, we set a policy for 60 degree centigrade temperature and we are registering
that policy. So, when we run a workload it will listen for violation. You can see on the top that
temperature is rising and as soon as it hits 60 it will give you an notification on the terminal.
So, rather than watching NVIDIA SMI using this you can utilize it like this.
And it by default monitors every five seconds. So, yeah the libraries which are being used are
set to this. You cannot change those particular models because they are controlled by NVIDIA.
You can enable those models. So, that is something that you can do.
Yeah, so this is a brief overview of the graph that we saw earlier like what are the metrics,
what are the DCGMI codes and what are the purpose of it. Now, comparing with the tools
that are available out there. So, DCGMI is a good tool it is an open core tool it uses NVIDIA,
but it is like you can say community driven as well. The exporters and all those things are
something that you can contribute. NVIDIA SMI is a good thing, but it does not give you much
detail it is a very basic tool. So, relying on it can give you bad inference.
There are other tools which are proprietary. So, for example, inside compute is a SM level
activity that you can see. So, it is specific to the kernel activity that you want to monitor.
So, it gives you more details. The inside system is a micro level of your GPU but these two are
proprietary. So, yeah monitoring solutions, yeah you can also set up an exporter of the DCGMI.
So, they have the exporter available out there and combine it with promising graphana. So,
you can create a dashboard out of it. So, major workload as we discussed is FP32, the floating
data types, how we can utilize them. There is also one more thing that if you have an AI workload
and it is using FP32, it can also be converted to the tensor float 30 proof. So, make sure that
you identify that thing first, whenever you try to run those things because it will improve the
performance. So, as we discussed memory bandwidth is the key thing here. So, having a huge
VM is not a good thing. Like if you have a good GPU, which has large VM, but it is memory
bandwidth is low, you are not getting much out of it. You are simply loading models and you are
not executing it faster. So, that is one thing. So, compute is going way faster right now. So,
yeah that is it from hand.