OK, sir.
What's your name?
I think Laurence.
Laurence.
And so let me introduce Laurence and Frank,
who will be talking about GPU stack.
And we are just about to start.
So let's start.
How are everyone?
My talk is GPU stack building simple and scalable
management experience for diverse AI models.
So my name is Laurence Lee.
This is my colleague, Frank Mai.
So we are building some open source projects
for using AI models.
So this talk is mainly about GPU stack,
which is a tool to manage your GPU clusters
for running AI models.
And another one is,
GPUF parser.
GPUF parser plays an important role
in GPU stack.
It is a tool for checking GPUF files
so that we can read those metadata
and do estimations like how much memory
is should be used for this model.
And GPUF parser itself is actually a standalone project.
It is also being used by other open source projects
in local AI.
And you can also use it as standalone CI
for evaluating the GPUF model files.
So in this talk,
we will basically split it into two parts.
The first part we will introduce GPU stack,
what problems we are trying to adjust
and experience where we have
when building these projects.
And the second part, Frank will discuss
what is under the hood in the GPUF parser.
How do we compute those memory estimations stuff?
So let me do a quick recap.
How have we run LMs locally in 2024?
So we all know that LMS-CPP itself
is an outstanding piece of software.
It helps us to be able to run LMs
in many hardware, especially for those GPUs.
And all LMs and the LMs still deal are also
famous tools.
It helps many developers to be able
to serve LMs or large models on their laptop.
And actually, we also have other inference engines.
We call that like the VLM,
scheduling, and many others like TensorFlow,
or etc.
The list can be very wrong.
So how do we scale?
We know that we can use tools like LMs still deal
or let top easily.
But in terms of the scalability,
we can face some challenges.
Because most inference engines
itself does not adjust the scalability issues.
They mainly focus on the inference performance
or how to improve the throughput of your service.
So for the scalability,
we will probably need to rely on something else.
For example, you can use Ray.
It's a distributed system for running machine learning workloads.
Or you can be deep speed, which is a library,
which is useful for making distributed training
and inference effective.
And of course, it can be Kubernetes.
Kubernetes is an orchestration platform for running containers.
You can also combine multiple technologies together
to satisfy your scalability demands.
For example, you can run Kubernetes,
which is to run Ray on Kubernetes.
So that kind of stuff,
we can see many examples in production workloads.
So the problem of these ways is that,
for example, Kubernetes itself is a general purpose container
orchestration platform.
It does not have the building concepts for GPUs and AI models.
So if you use this one to run it well,
they generally need to have a certain level of expertise
to run those platforms effectively.
So what we are trying to adjust is that we want to make it easy
for the general public, not just professions to run
and to easily scale their platforms.
So how's the view of the general public?
So unlike the audience here,
many of the committee users,
we encounter actually lack of many of the knowledge
that are under the hood.
They may not know what the model architecture.
They may not know the detail of how the matrix
multiplication works for inference.
They may not know what those different kinds of
connections means.
So what they know are that they have the devices
or hardware at hand.
It can be as powerful as H100 clusters.
It can also be some old 30, 3090 GPUs
or even some data computers.
And they also know that there are many good models out there.
For example, deep-sick R1, they can just download it
free on hacking interface or somewhere else.
And when referring to AI models,
actually large language models are not the only ones.
We also have like a stable diffusion,
which is for the image generation.
We also have the embeddings and rerankers models,
which are useful for building or rack systems.
We also have like audio models.
So in terms for the end users,
they may need to, for example, do the chat
to the large range models.
In terms of the AI application developers,
they actually want to have some useful APIs
to interact with those all those kinds of models.
Actually, I should put Lama in the first place of this slide,
but it just don't have official logo.
So the user demand is simple.
So they want to get model and get it wrong
and do their job style.
But the complexity is there.
So we have the topology complexity for running AI models.
For example, if you have some powerful GPUs,
you can do fully of loading to those GPUs.
But for some users,
they may need to do partial of loading.
They even don't have the GPU.
For example, we can do a CPU of loading
to run your deep-sick R1.
And for some users,
they also might need some distributed topology.
For example, their models are too large
to fit in a single GPU or multiple GPUs
on a single node.
So distributed computing may involve in this process.
And the diversity is here.
They different users may have different highways.
For example, different vendors of the GPUs
or different accelerators.
And the inference engine can be also different.
For example, we can use lambda cdp to do our
ming inference for the whisper.
We may need to rely on something else.
And the models can be different.
The conversations can be different.
And when you want to run something well in production,
you also need to tune your parameters well.
For example, to change the bad change strategy.
For example, to optimize your cache,
including the profile cache, the kv cache strategy.
And you may also use different execution mode.
For example, eager mode or the graph mode for running the inference.
And when you have to serve a high throughput of the LM service,
you can do optimization on higher level.
When you have a bunch of requests.
So to adjust these changes,
we want to build up two that is smart.
It can understand the environment well.
It can understand the models.
Well, it can understand the inference engine as well.
So this is the screenshot of the GPUs.
To know the environment well,
we should know the detail or the information of each
node added to the cluster.
For example, the operating system, the architecture,
those matters when running your large language models.
Because that different inference engine may rely on different platforms.
For example, MLX is for the Apple device.
And when we run the VRM backend,
we usually cannot do that on Windows server.
So we need to know those information well.
And for the GPUs, like there are many different kinds of GPUs,
or when the software can group everything together.
And they can do live balancing your loads
or build inference for you.
And to know the models well,
for example, GPUs stack needs to be aware of the models.
So there are many different kinds of models,
including audio, LM image or any other.
So we can run a properly scheduled to support GPU or devices.
For example, in this screenshot,
when the user tried to run the large language model,
it cannot be fit entirely to the GPU.
So partial of loading is used here.
And for this situation,
when we detect that to run this massive model,
we need to rely on multiple GPUs on different nodes.
We will do that for you.
So this is distributed inference or class workers.
And the two needs to know the inference engines well.
So when you search or when the user search model on hacking phase,
they are many different kinds of format,
quantization, etc.
So when you choose one of the models,
we should know which inference engine fits your environment,
which inference engine fits this model,
so we can choose it automatically for you.
And you can do any optimization you want.
You can easily configure those back-hand parameters
according to your need.
And we also have the cardboard support for multiple inference engines.
What does that mean?
As we know that the innovation in AI models happens fast.
So when a new model comes up,
the inference engine needs to support it quickly.
So we can see in the most major inference engine,
their release cadence is for they need to release very fast.
So bugs can show up easily as well.
To balance the stability of running the models
and trying the new models,
we need to decouple the inference engine version
and the two itself.
So when using this GPU stack platform,
you can easily choose different inference engine
and they can serve different models
and it's totally decoupled.
And you can easily easily do its experiments.
So next, we will talk about how to do
your F-passer work under the hood.
Frank.
Thank you for joining us.
And I will take you deep dive into the Iranian memory usage
of the GPU F-passer,
a GPU of the models,
which are the kick-canger for the efficient running AI models.
As the model is at the model grows inside,
the memory management director effect the inference speed,
cause and hardware utilization.
Our team tries to simplify this process
and enable the wrapper to optimize the resource
without needing to be expressed.
Okay, let's see this.
When we load a model using LamaCp or something like LamaCp,
based on the LamaCp, just like LamaBoss,
we can get some loss of the memory usage.
As you see, we can divide the memory usage
into four parts, the footprint,
the base overhead of the loading model
and running inference server,
which the models,
the memory occupied by the models parameters,
the kick-catch,
the memory stores, the KV pair,
in the tension mechanism,
and the computation,
the memory used to influence.
For example, this is the KV 2.5 half million model.
You can see that the wage is almost one kick-buy,
and the KV-catch is almost 100 megabyte,
and the computation is almost a 300 megabyte.
This number indicator,
regulating any part, can lead the impartity resource allocation.
So let's break down the let's just see
which is the kick-factory example.
Sorry.
First, the hardware backend,
chose off the hardware backend,
such as CUDA, Recomb, Meta,
or can influence the footprint.
A typical example is that when we call the CUDA set device,
it occupies 160 megabyte in the Nvidia P40,
well, it occupies 215 megabyte
on the rear-deaf 4090.
And then the models, in this part,
the number of the models parameter tastes great impact
on the wage,
and choosing the,
choosing the, the Rycon configuration
for Mac can reduce that part.
And the embedding,
and the dimension of the embedding layer,
and the actual layer of the model,
take a very degree of the KVCatch and the communication.
And then the hyperparameters,
as you see, the hyperparameters,
the contact size, fresh attention,
and the cage type.
It can, they pay the crucial roles in the KVCatch
and the computation, integrating.
Let's break down the,
this is case for the memory use for the KVCatch.
Let's break it down.
The attention equivalence is determined by the embedding lens
and the attention had come.
And then the embedding KVCatch,
is calculated by the multiplicity,
attention equivalence by attention had come for the KV.
And the KVCatch and the KVCatch dimension, as you see,
the KVCatch dimension is,
is also calculated by multiplying the embedding KV
to QA by the contact size.
So finally, we can result in the KVCatch player
with the, with the, with the catch type
and the KVCatch dimension.
The total KVCatch can add up
each layer of the KVCatch.
This number, this shows how even small parameter
can have significant impact on the memory calculation.
So KVCatch,
integrating memory usage is more complex
when we consider the, just you see,
the fresh attention memory map and the IPC remove
all our floating and the tensors three.
So, we,
so to adjust this problem with,
we product two,
call them,
do you have a pass?
And it can, it can remove
to passing the metadata of the GVF fire
and then to assess the memory requirement
without needing to download the model.
This is the, let's do some, let's do some K study.
This is the, we will try to calculate the difference
between the actual running and the estimation
results as you see.
In the RAM part, we can,
we can maintain the difference
with in the 200 MB.
And the, and the rerbs in the 1 GPU,
we can, we can maintain the,
maintain the difference within 100 and 20 MB.
When we choose the, when we change the context size
from 4K to 60, 64K,
and then we can keep the,
the RAM in 1 GPU under the,
1,
under the 120 MB.
When we, when we use the tensors three,
average the 2 GPU with the different model
and we can compare 2 GPU,
and you can see that,
we can keep the memory use under,
under the 15 MB.
That's all.
Thank you.
Thank you very much.