So yeah, for those of you who are not familiar with VLN, VLN is basically started as the open
source projects from a couple of students from the Berkeley and it kind of exploded very quickly
because it offered a really like a nice and efficient implementation of inference for modern
LLMs and then over time it became one of the very very large and vibrant communities where
like PhD students, researchers, engineers and also like big tech companies are contributing
very actively and trying to develop like inference engine which is going to serve these modern
LLMs as efficiently as possible. The cool part about VLN is the fact that it runs and supports
various hardware so you can run it on and VLN GPUs, AMD GPUs, Google TPUs, AWS,
Neuron, Intel Gaudi and so on and so forth and usually for most of these hardware vendors like
the teams from these companies are helping out kind of developing you know, trying to make VLN run
as efficiently as possible in their hardware and then on top of that VLN supports running all
like popular modern LLMs these days and usually model vendors are working with VLN kind of behind
the scenes to always try to enable day zero support for their models when they are about about
a real system. So VLN offers a ton of like optimizations by default and like you will get
already like pretty good performance if you just deploy your model with the standard configuration.
However there is always like the traditional kind of question of trying to figure out
whether we can squeeze in more performance and this is where quantization and speculative decoding
became like very attractive topics for us because they offer the additional pathway completely
orthogonal to all optimizations that already exist in VLN to accelerate inference with these like
very, very large models. So just to bring everyone to the same page just one slide on what
quantization is all about. So quantization is a basically a process through which we are reducing
the number of bits that we use to represent either weights or activations of the model.
What I mean by this is that if we take a neural network and we plot all of the weights in the
network, we're going to get this like a normal Gaussian like her. And what's going to happen is that
the weights are going to be distributed across this curve and they can take any possible value here
and given that this is the full precision and by full precision usually these days we mean
the weights are represented in brain floats 16 for but that's that became a standard these days.
We have very high granularity so we can represent like very, very small differences between the
weight. And through the process of quantization what we're trying to do now we're trying to take
these weights and we're trying to put them in these discrete buckets and each of these buckets
represent one single value that we can represent in this now a new quantized range.
Because this like a quantization this this quantized range here has much much lower granularity
it this means that during this process we're going to have to shift the weights a bit.
For example if two weights were very close to each other in the original full precision regime
we're going to most likely one be able to represent their difference in the quantized range
so we'll have to put them in the same bucket. For some weights we'll have to move them to the
left to the right and so on and so forth. All of this to say that quantization is not the
lossless process. So this means that there will be some loss of the precision and the entire
game and entire research here is all about figuring out algorithms which are going to enable us
to do this while preserving accuracy of the model as much as as possible because we still don't
want to destroy our model in order to get some efficiency gains. So let's take a look where
quantization fits inside the server and then based on that we'll try to figure out which quantization
schemes to use in which in which in which situation. If we look at our server like at the like
the very bottom of this like a pyramid we have CPU memory. This is like our main memory it's
like very large but it's also very very slow. On top of that we have our GPUs. GPUs have
something called like high bandwidth memory or HBM and this is what you see when you run
NVIDIA. So my for example and the main characteristics of this layer is that it's much smaller
but it's also much faster than the CPU memory. And then at the very top we have GPUs
RAM and tenser cores. So this is the smallest part of the memory but it's also the fastest one.
And this is basically the part where the computation happens. This is where matrix may be
some applications actually happen and this is how we are doing the inference like with our model.
So if we try to look at what's happening when we are like trying to do one forward past
our model. So we want to put in some tokens for the model and generate an answer.
The first part that we'll have to load the model. So we have to move it from the CPU memory
to the HBM and this is something in most normal scenarios we do only once.
We load the model to the GPU memory and then for n times what we're going to do like for
every single operator in the network we're going to take like one weight matrix. We're going to
load it from HBM inside tenser cores. Do some computation with that right back results and repeat.
And this is something we're going to do for every single like a linear operator in the network
and then what's going to happen is that this is the sequence of the operations that we're
going to repeat many many times for every single token for every single letter in the network.
The reason why I'm just kind of specifying here like loading torch and a linear is because
even though the LMCs days they have many different types of layers linear layers or linear operators
are the costly pieces of this. If we were about to plot like where we spend time during the forward
pass all of the linear operators this is like the self-attention and the MLPs part.
They will take majority of the time here and they are the main target for optimization in general
here. So these are the two main components and candidates that we actually want to accelerate
through a quantization. So the first part is an acceleration of the loading. So given that the
loading part happens like when we're transferring the weights from HBM inside the tenser cores
there is nothing else we can do on the hardware level because all GPUs come with a specific bandwidth
they're nothing we can change there. However the only thing so that we can do there is
reduce the number of bits to load and those bits are the bits of our weights. So that's the first
kind of quantization scheme that we are working with that's called weight only quantization.
The second part is the computational part which actually happens inside the tenser cores and this is
where matrix matrix multiplication operations happen and they happen inside tenser cores.
So the only way to accelerate that is to find faster tenser cores. And if we take a look at
technical specifications of an H100 GPU we'll see that by default we are usually because most
models are in brainfolded. 16 format we are operating in brainfolded 16 tenser cores which you
really have around 2,000 steraphlobs. And then if we look in this sheet there are also two more tenser
cores called fp8 and in-tait tenser cores which offer two times more flops per unit of time.
So the main goal of this part is now to push all of the matrix matrix multiplications
to happen inside fp8 or in-tait tenser cores instead of brain float 16 because now we get
two times more flops per unit of time. And the way to do this now is to have both operands of
the matrix matrix multiplication quantize either to fp8 or in-tait. And this is where the second
scheme comes. Which is called like weight and activation quantization. And then these are
like the two main paradigms in the quantization field. And whenever we want to like quantize and
optimize the model like we have to pick one of these two like and that's going to like like
depend on the exact scenario that we're interested and we'll see later on how to choose this.
Now I'm going to just very quickly like go through some code samples just to like show
showcase how how relatively simple this is at this point in time. As part of the real
and project we have a library called LM compressor which is basically a library that implements
a lot of like state-of-the-art quantization algorithms which you can just use from from the
from the python interface. And it's relatively simple to apply this. You just load your model.
This is the standard hugging phase transformers like we load load our model with dot from pre-trained.
And then we instantiate the quantization modifier. We say okay we want to target all
linear layers because as I mentioned before those are the costly pieces of the inference.
And then the scheme that we want to do for quantization is let's say in this specific case FPA dynamic
scheme we call the one-shot method we provide the model, the recipe and that's it.
This is like a very simple pipeline where we don't use any calibration data. So here now we come to
the second like like a choice that we have to make when we're doing quantization. And that's
the choice of whether we're going to do quantization with or without calibration data set.
Calvation data set is basically a set of tokens that we pass through our model
to try to simulate the forward pass through the model and see how the model responds to this.
So because LMS these days have one major problem and that problem are outliers inside
activations. Outliers in activations are basically that's like a phenomenon where some specific
channels have order of magnitude higher values than other other activations in the same in the same
layer. And this makes quantization very hard because if we have one extreme large value and a lot
of very small values, if we do like quantization top of that, this very large value is going to push
all these small ones to like to zero. And if we do that, we're going to destroy the accuracy.
So then we have to address addresses and kind of the path to get there is to take some
calibration data set pass it through the model and see where these outliers appear in the model
and then based on that applies to apply some tricks. And this requires some kind of calibration
data set and this kind of here example here shows how how relatively easy it is to do the
quantization with LMS compressor, but now just accounting for the fact that we want to have a
calibration data set as well. First we load the model, the standard way as we load
any agifist transformer, then we have already prepared like based on like many years of research,
we already prepared like a proper calibration data set. Usually if you have a model
fine tune for a specific task, you want to pick calibration data set from that task so that you
have in distribution tokens. If you're just like quantizing like a general model, then this should
be a good enough enough data set to start with. We just tokenize this and then we create a
quantization recipe and here I'm trying to show you that one nice feature of LMS compressors
that you can combine multiple different quantization algorithms together. Here I'm just like
throwing in smooth quant, which is an algorithm to deal with the outliers in the activations.
And then the standard quantization modifier, I want all linear layers, but now I have changed
the quantization scheme to quantize the weights, to 8 bits and activations to 8 bits. And also I can
ignore some specific layers if I really want. One common practice is to ignore quantization of the
final LMS head because usually this takes a significant impact on the accuracy of the model.
And then we call the one shot, we provide model recipe and like contrary to the previous example,
now given that we have calibration data set, we also have to provide the data set and the number
of calibration samples. And what is this is going to do is going to pass the calibration tokens through
the model, look how activations behave, where the outliers are and then based on that apply
smooth quant and the wrong linearest quantization and provide provide us with a quantization model.
This pathway is computationally more expensive, right, because you have to have some access to
GPUs kind of to do the forward pass. The previous one was relatively easy because it did not
depend on any calibration data set, so we have been able to do that even on a CPU.
LMS compressor has a couple of nice already built-in features to deal with this,
whether you have a single GPU multiple GPUs can do the sharding, CPU floating and all these
like modern tricks to deal with that. So given that we are as I mentioned before, we are quantizing
a model, this is a lossy process, the main kind of the first question that we should ask ourselves
is how well are we recovering the accuracy of the original model. And here I'm just kind of we have
like a pretty long line of research where we're trying to show how to do quantization in a different
in different scenarios for different models. And so on but here I'm just kind of like like a
cherry-picked just one example to show how these accuracy recovery looks like in practice overall.
So what we're looking at is the like an entirely family of the deep seek R1 distilled models.
So from Lama and Quen deep seek took these models, fine tuned them for some reasoning
reasoning tasks. And then we are looking at their average performance across AIME and F500
and GPU alignment, which are the standard kind of reasoning task to do this. And we are looking
at four models, BF-16 is the unquantized baseline represented by the gray, gray bar, F-PA-188 and
in four represented by a blue, green and yellow colors represent just the different quantizations
in that we apply here. And one like most important point kind of to take away from this is that
blue and the green bars here are usually very, very close in most cases almost indistinguishable
from the gray bars, which means that F-PA-188 quantization if done properly and calibrated properly
should always yield accuracy recovery in range of 95% to 100% of the full baseline model.
One important note in addition to that is that in four quantization is usually a little bit
trickier in this in the specific regime and more specifically for the smallest models in the family.
As you can see Quen 1.5 V, Lama, A-B, we do see slightly higher drops here,
but usually if this process is also done properly, these drops should never be below 90%.
At least we have done a ton of research of this produced hundreds of models on the Hague Nesave
and we still haven't found a single use case where this the accuracy recovery would go below 90%.
So if that happens then something should be calibrated
slightly better on the quantization part. And all of the times so far I'm just talking about Lama
but all these techniques and all these techniques apply to vision language models or any other
architecture that you find, it's just that Lama's these days are the most representative use case.
So this was about the accuracy given that we are compressing a model, we usually expect some
gains on the speed upside because now we have a smaller model and we want to run inference with that.
So here what we're looking at is how the inter-token latency changes with respect to the number
of queries per second and we are looking at Lama, A-B model served on a single A-6000 GPU for
some specific use case doxling generation. And we are looking at three different models, B-F16.
So this is the unquantized baseline, entade weight and activation quantized model and
in four weight only quantized model. And they're like two interesting things to take away from
this graph. The first one is that if our server is being hit with less than four queries per second
we're going to be in this regime here where the best choice with respect to the smallest latency
that we want to get from the model would be obtained by doing in four weight only quantization.
And this is because in this specific use case here we are not bounded by compute. So we don't
have many requests coming to our model. Therefore our GPUs are going to be idle most of the time.
So what's going to happen here is that the main gain that we can get is by optimizing this
weight loading part of the pipeline. Then the next part here becomes important that when we hit
four queries per second, when we hit four queries per second what starts happening is that
weight and activation quantization starts becoming better choice than weight only quantization.
This is because at this point we have large enough inputs to keep our GPUs busy doing matrix
matrix multiplications which means that now we need to optimize the computational part. This was
the second part of the inference pipeline. And we can optimize that by doing quantization of both
operands of the mathematical which means we are leveraging lower precision times our queries.
And then if we push even more, if we push even more what's going to happen is that at some point
we're going to get in the heavily compute boundary gene. So our inputs are going to be extremely large
and weight only quantized models are going to become a voice choice than just deploying
unequantized model. In practice because right now we're going to be heavily bounded by compute
and the time needed to load the weights is going to be almost zero relative to the time that we
spend in terms of course doing models. So you have to be really careful when you're deploying a
models in order to figure out where on this graph you're deployment flies. And this depends on
all of the factors in the game or like the model size, the GPU that you have and the requests
that you that you that your server server is receiving. In order kind of to automate this because
we had to do this many many times we developed a library. It's also part of the villain project
it's called Guidelalan where you can just serve your model and then simulate real-world workloads
and see how how the model behaves in different scenarios and then based on that plot something
like this and see which quantization scheme is best for you. Given us we have five more minutes
I'm probably going to try to wrap it here just just very short like part regarding the speculative
decoding. So speculative decoding is relatively new technique. So quantization was like a lossy process
because we are moving the weights around we are shifting the weights lossless speculative decoding
is a lossless acceleration technique which means that we are not changing the model and the
text that we get at the at the end of the speculative decoding process is like guaranteed to be the
same text that we would get without it. The main caveat here is that we have to train additional
model called speculator model which we're going to serve alongside our original model. So this speculator
model like is a model which is like order of magnitude smaller than our very large model that we
are originally interested in and this model is something we're going to run many many times
like trying to produce three five tokens at a time and then we're going to take our larger model
in this case it's called a verifier just to verify the outputs of this smaller model and then
the larger model is going to say oh I agree with three out of five tokens these are the tokens
that I would produce if I was in the decoding phase so I'm going to accept these three I'm going to
reject these two and then go again and in this way if our speculator model has been trained properly
we get some kind of scheme where we are generating multiple tokens at a time with a larger model
for the cost of like running a couple of for passes through this very very small model and it's kind
of feels I tried to illustrate it here in the terrific clear but the first pass like we have a prompt
once upon the large model starts the the inference produces up and then we take this like once upon
and we put it to we give it to a speculator model and we say okay please like speculate the next
two tokens should be it's going to say time there and then we're going to take all these
and we're going to put them through the verifier and say hey are do you agree with these tokens
like are these tokens that you would produce and in this specific case the model said yes I would
produce these we got three tokens for the cost of running just one token plus some small amount of
time we spent in this speculator model then we have the we generated three tokens we pass it again
to a speculator we try say oh generate the next two tokens the the the speculator model generates
these two the verifier in this case does not agree with the scary so it's just going to discard
the scary part take the the first one which which agrees with and then it's going to combine them
and produce an x one and this entire process is great but there is one important part is that
quantization was relatively easy to apply we take a model we just quantize it in one shot
we have a model which is faster here the main cost that we have to pay that we have to train the
speculator model so we have to take a large model and then we have to generate some data set
and then we have to train smaller speculator model they're like different techniques how to train them
to mimic kind of the distribution of the larger model so this is like a computational
expensive part but it does does allow us to get like a loss as loss of speedups and for this purpose
like we also developed a library in the VLM project called speculators where you can train these
models for your own data sets or you can just go to hugging face hub and pick up some of the models
that that we already released there so given that you don't have time I'm just going to skip through
the like results yeah the cool part is that we can get the speedups of anywhere from two to five
x depending on the model size and the quality of the speculator model but I'm going to skip this so
we can maybe take some questions so yeah links like all all of the libraries are part of the VLM
project open source you can you can just play with them there are standard examples if you
don't want to do any of this you just want to get like high quality quantized or high quality
speculator models you can go to red hats hugging face hub we are kind of releasing on a daily
basis new models there it there is like more than 500 compressed models which have been already
validated before so you're sure and you can see like what the accuracy recovery saw across
different use cases so you can just download them and play with them yeah thanks a lot for your time
yeah perfect yeah okay yeah so yeah so the question is how to tweak the recipe so the recipe so
it really depends on the quantization algorithm that you want to use every single quantization algorithm
has different knobs to tweak usually what you would do it would be like the like like the standard
training loop you would take some development set like small some small subsets that you're you
know not testing on and then you would just run a couple of quantization quantization runs with
different like hyper it's like a like like tuning a training process with different hyper parameters
and then you would you would evaluate on the development split and then you would like pick the best
one and go to the test split usually there are some there are some guidance depending on which
quantization algorithm you use on how to tune some specific pieces like for example GPTQ which tries
to like approximate like the inverse Hessians then there's like some dampening therm blah blah blah
which has to be picked like on a like a model line model basis so there is no like a general rule
of time you know you should set this like this there is just like some kind of rule of time
on how many tokens you can use for calibration so like like 501K tokens at that point you start
seeing like a like a negligible improvement in the end of the results but apart from that everything
is very specific to the algorithm that you use there is a question here yes yes yes so you always run them
in parallel yeah so yeah the question is how to run speculators in vlm so basically in vlm we have
support where you don't have two instances of vlm it's like a single instance but inside of that
single instance you have a speculator model and your large model and if you do like TP2 as you said
for example in this case you are splitting that like the tenser parallelism refers to your original
model the speculator will still run on its own so it's kind of completely independent because
it's really small model you don't get any gains by kind of splitting it or doing any fancy
sharding with that so it's basically your entire model is still running in the same way you just
attach one more smaller model which just runs in parallel and it's it's it's simple as a vlm serve
you take a speculator model and do vlm serve you give a speculator and based on that it's going to
do everything automatically for you yeah you don't have to do any custom stuff yeah yeah there is
question order yeah so the question is regarding the validation of quantized models is it
are we in a danger of overfitting for example for some specific data set yes that's like a really
great question and it's still like on on going kind of problem because usually whenever we quantize
the model which we open the model card on hacking face out and we see okay like you know
mistral they published like these 10 evaluation benchmarks so our main task is okay let's try to
do quantization and recover on these benchmarks that they proposed and that's kind of the standard
setup but we usually do maybe there is something that we are missing along the way so we are always
trying to add new new benchmarks to the mix but we still haven't found a single benchmark where
the entire story that we have about accuracy recovery being about 90% fails so yeah we kind of we have
a paper where we did more than a million evaluations across many different benchmarks is called
give me bf 16 or give me death accuracy performance traders in in quantization and they they're basically
presented the study like a large scale study we took every single benchmark that exists out there
from the arena hard coding a hacking face leader board three one we two and so on and we still
haven't been able to find the single benchmark where this this fails yeah yeah
so so so the the question is relation with the llama yeah
okay yeah so the violum supports llama cpp models in general i think but i'm not exactly sure
what support for the quantize now because llama cpp has its own like way of doing the quantization
like q4 q3 and all these different schemes and they're all doing weight only quantization as far
as i'm aware of and i think that violum should support them but i'm not sure that that's
well tested path at all i think there is some kind of way to run llama cpp models but i don't think
it's super performant at least at this point yeah i know that there are like some people working on it
specifically in in redhead there is a new team which is supposed to bring llama cpp to be a first
class citizen of violum but i'm not not really like in touch with that with that pipeline we mostly
do violum like on on gp's that's kind of the main to many of the skis
great