Hello, hello, everyone. We are starting the next talk and now Tanya and Ivan. Ivan,
Ivan, Ivan will tell about the quantization and this is the talk you want to listen to
if you want to learn about quantization because Ivan is the one who is the like wild
bore of quantization or like father of quantization of which are of quantization like looks
like a witcher a little bit, yeah. So like I give it to your hands and just talk about quantization
now. Thank you. So as we go into the quantization portion and we actually don't have to talk
so right now. So this is a Q&A talk so I'll be asking questions but feel free to write your questions
if you go on this car code for Ivan and Father speakers as well. So by the end of the talk
we'll actually look who submitted what and ask them as well. But to start actually it's for us
amazing opportunity to put a real face with a gift-up profile picture. So who have seen this gift-up
profile picture? Yeah everybody who's basically looked into any kind of quantization right?
So now this is the real face but it's really hard to find your pictures on the internet
Well I'm kind of a really cool past. No, I'm this kind of guy who doesn't participate in social networks
you'll not find me anywhere except this GitHub profile. I'm not on Twitter. I'm not on Reddit.
I'm not nowhere. But before working on a CPD, so you actually come from a completely different
industry. You get physicists right? Yes, I'm a physicist. Graduated in high energy physics many,
many years ago. Then the way my life evolved I ended up working in medical physics,
specifically doing research on radiation therapy of cancer,
basically physics, tools related to that. So how is it similar in the current quantization
works and what's different? Well, when you do this sort of work, you basically it's
numerical, numerical methods of various kinds. In my previous work I was most famous for the
for the Monte Carlo work that I have done. But I have done lots of other things. One of the
problems that you have to solve there in radiation therapy is to optimize the treatment. So in
radiation therapy, when you're radiating the patient, you come from many different angles and you can also
modulate the fluids or the amount of radiation you're imparting from each angle from every position
of your beam. And so there there is this large scale optimization problem where you want to
basically optimize the fluids or the amount of radiation from the different angles in the different
positions at this angle such that to achieve an optimum dose distribution in the patient. So this
problem is associated with the so-called system matrix. I don't know anyone knows about the system
matrix. Many optimization problems have a system matrix. Basically the system matrix here is how
much dose a small beamlet J in parts into a small voxyl I in the patient. It's a giant thing.
It's billions of elements. So then while working on that at some point I realized the best way to
the basically the calculation is 99 percent dominated by the matrix multiplication of the system
matrix with your gradients relative to the dose or the same matrix with your changes of the fluids.
And so the the best way to speed up this calculation is to actually quantize it. Now it's a bit
different than LLMs. There you're happy if you can make it work without numerical instabilities
so it's 32 bit floats. So going lower than 16 bit in it would be pushing it too far so it was
a 16 32 float to 16 bit quantization. But basically what happens then is that when you do this you
realize that you have a big butt of in this matrix which is you can represent a side bit and you
have an extremely sparse matrix which is really requires the full 16 bit. So this was the thing
that my one of my encounters in quantizations. So having gone through this one evening I was
having beers with Georgie who also I live in Sofia partially and he also live in Sofia. Georgie was
the first member of the science team that I built in Sofia that I hired I don't know maybe 12 years ago
15 years ago. I don't remember. We're going out regularly for beers and then he started talking
about language models and llama and whisper and quantization and that got my that's how I
actually got into this business of quantizing language models. And then since you got in there
have been implemented a lot of different methods. It's actually if you try to track and llama
CPP some of the older methods it's sometimes they no longer supported even. So what was the
history of the evolution of the quantization method there? So did you just go lower in the number of bits
or what was you think in actually in the evolution of the methods? Well so when I started doing
the only quantizations working on that the only quantizations in llama CPP were Q40 in
then the time there was no GPU support so your best option to actually run a
perplexity calculation to see how your quantization work was to use
the blast back end and doing that we realized that the result from blast was very different
and much better than the result without blast and the reason for that was that when you did
blast you did quantize and did floats while in the back then this is now two years ago
llama CPP was quantizing the activations also to four bits and that led to a significant
precision loss. So that initiated the addition of higher bit quant in llama CPP, a
georgated Q80 and Q50 and then I did the kick once I mean I don't know the kick once
haven't changed that much maybe I have done a little bit of tweaking here and there on the
quantization algorithm but not too much. Of course lots of this is driven by the hike waves
of the internet people users start coming asking ah I saw this can I have a two bit
quantization can I have a one bit quantization so these are then you respond to these people you do
to bit the initial kick once to bit but it wasn't a very good to bit quantization so then later
the hike once came along which are much better at lower bits per wave. So with actually like since
we are talking about going below to bit quantization there is a bunch of research this
days and everybody talks about of course like combination of which layers to quantize to which
bits how to do the script quantization but also there is some new research about how to do
quantization of our training so the don't lose quality that much when you quantize to like below
to bits. So what's your take on it? Are you interested? Are you trying to implement methods
like that or everything that's like on the training side is separated? No I think so I have never
taught what seriously taught about getting involved in training because I'm just a lonely guy sitting
in a corner and hacking. I don't have the resources to do training of serious models but my
personal opinion is that people designing and training models should go to integer models
and train directly in integers Microsoft has demonstrated that you can train
partner in models so that for sure to me indicates that it will be possible to do for bit
a bit integers. So actually coming to this question about doing it floating point that we just
do integers that is a lot of debate especially on the hardware side like do actually need
floating point on your chip especially on the acceleratorships. So what's your take on it? If you
just train on integers and you're just inference on integers, do you still think like that
it's not the case? I think but I may be biased I'm old school I think integers are much nicer than
floats. Back in the day some of the computers I have worked on didn't even have a floating point
unit so you have to do software emulation and so any calculation that you could convert to integer
math you did that just for the speed. Now it's different today but even today integer math has
some very nice properties your calculation is 100% reproducibility doesn't matter if you run it
on this CPU or on that CPU or on that GPU or on this CPU you always get the same answer this is not
true for floats which makes it kind of hard to compare your results from this architecture to
this other architecture. So I think yes going fully going full-mongued to integers will be really nice.
So one of our engineers who looked a lot at your work said that it's a really interesting
combination of engineering optimization tricks and numerical designs that you use so can you
give a little bit of comment on your approach to quantization is it trial or error is it more
common from some mathematical paper and mathematical work how do you approach it in your opinion
what's the combination between engineering tricks and numerical design? So it's I had a
friend from a friend at the University of Sofia he was calling this numerology it is numerology
basically when you do quantization you're facing a mixed integer optimization problem which is
notoriously hard to solve. Now it's not impossible to solve especially if you do block-wise
quantization as it is typically done in llama CPP so it's doable and in fact in the early
in the very early work I had done an exact solution of the mix of this mixed integer
quantization problem to determine the quant. The issue there is that you don't really know what
you're wanting to minimize. If when you're doing optimization you always need a cost function
that's how we call it back in the day now people call it a loss okay so you need a loss
and it isn't immediately clear what is the loss that you want to minimize when you're converting
floating points to low bit integers typically you use root mean square error because I don't know
like of any other insight this is something that is nice to work with so back then when I started
working on it turned out that finding the exact solution of the of this minimization problem leads
to a worse result than doing an empirical minimization that stops how the way. Now maybe today
with the iMutrix which gives a little bit of indication on which weights are more important than the others
it will probably be a little bit more stable than it was back then without the iMutrix
but this empirical way of you know trying a number of steps and trying
modifications to the iMutrix weights to improve your quantization results kind of works pretty well.
Some of the some of the quantization methods actually do solve the minimization problem
exactly for instance if you look at the one bit iQ1s that does in fact use an exact solution of
the minimization it's very easy when you only have three possible states for your funds.
Yep and that is actually like I think one of the reasons why people interested especially on the hardware
side because hardware for that is just way easier to build for for just like.
They're not a model yes yes. So what is the latest quantization technique or what
what's recently I work on and where can we find it is at your own lmcpp for that you can
preview to recently or like if somebody wants to join or to track your work what's the reason
Yeah I have I'm having by my lmcpp fork when I'm quietly hacking around in the corner
the best kept secret on the internet. There are quite a few things that I have done in that repository
and some of some of the work is new quantization types. I call them iQK to which is the combination
of iQ and k and those are actually quite especially the four five and six bit are quite
quite a bit better than what is in lmcpp and then the other the lower bits are not really
better than the iQ1s but they're more efficient on the CPU and also I did those out of curiosity
iQ1s use this kind of group approach where instead of converting individual model weights into
fantastic values you take group groups of weights and you try to put them on to a given grid type
so for me this was a curiosity to see if I cannot achieve the same quantization quality without
actually doing the grid thing and yes it turns out it is possible and then the most recent
thing related to quantization is sometimes ago there was the there were the it was the Q tip high
I don't know how many of you have heard it they use trailices to basically generate a series of
quantized values I did this out of curiosity in my repo it sits on a branch it's not merged into
my main development branch because it only has Q2 inference doesn't have inference for the other
platforms that are supported there so speaking about different pecans and Q2 for are you interested
in writing specialized compute kernels to have better levels of optimization or that's the
level you don't go to specialized kernels for different hardware backends and again like
even for a video you can load kernels without Q2 there is some nice hack that tiny grid came up
or somebody else but I saw it in tiny grid the first time came up is so that you can actually
load kernels without even booting to the well I'm not a Q2 I'm not a Q2 person in fact the
kernels that I wrote from Lama CPP was the very first Q2 working in my life so I'm not an expert there
cannot really comment on that on the CPU it's quite interesting to optimize it turns out
you can speed up on the CPU the Lama CPP quantization implementation by factors up to a factor of 7
for some quantization type types some of this went into Lama file but in the meantime I have done
more progress on the on the CPU site I find the CPU kind of more cool I'm older than almost
everybody in this room I remember living through various hardware types there was mentioning
of transputers just the other talk yes I went through a porting high energy physics code to
transputers in 90s and then those transputers died away CPU will stay around so it is and it is
much nicer to actually work for so I'm looking forward to hardware vendors it's CPU manufacturers
to finally give their CPUs more memory bandwidth and then they will be not so bad to work with
especially for very large models which you cannot do on a consumer level kind of a GPU
and this what's currently available of the shelf what's your choice of hardware
what's my choice I have an updated my hardware for a while so I'm working on a Ryzen 79050s Zen4 16 core
I'm also having a little bit older the previous Zen generation 5975 I have an RTX 48 to test
the CUDA code and then I have M2 max to test the RNA on implementation and the metal
implementation so a lot of interest now goes into the direction of different architecture
like mixture of experts architecture and I mean they're all still transformable based on
can be run on LMSCPP but like if you talk about quantization techniques is there any advice
like what is the go-to method these days or like you should try different ones and compare
so what's your recommendation for people who just need to start choosing
so I thought we wanted to so the tensors different tensors in on LM have different
importance or different impact the quantization of different tensors have has a different impact on
the accuracy laws that you get through the quantization because of the quantization some tensors
are more important than others in the standard transformer attention mechanism the V tensor is
the most important one then comes the output of the attention layer and then the comes the
FFN down tensor in the feed forward part of the layer these are the most important ones now
more recent models the the importance of FFN down appears to have the gun down
then we have the R1 deep seek model the way all in the room there and their tension mechanism
is a little bit different and I have to admit that I haven't actually studied in detail how
this new tensors influence the effect the quantization error
I think they still use basically your quantization methods but just differently for different parts
yeah it's interesting the onslaught thing making this huge hype around the internet
by quantizing deep seek error R1 to IQ1s so basically I actually yesterday went to check my original
PR in Lama CPP and yes the quantizing the attention tensors to four bits is in that PR including
especially when you are using a mixture of experts why it doesn't work in Lama CPP and
onslaught needed to come along is because you need to the tensor names are slightly different
so the heuristic to detect which tensor to assign four bits just didn't work but it's basically
a three line code change to do what onslaught did that's funny so do you are interested in any
other machine learning frameworks besides like forks of Lama CPP I know you contributed to Lama
file a bit by the way just in says hi she was not able to come but any others that you at least
tracking may be not contributed to but like some ideas and them or again just Lama CPP and C++
model I work in my little fork I occasionally look around but there is not nothing yeah I always
check what the use issues people are entering in Lama file out of curiosity but I don't track
anything specifically very closely what about different papers again any collectives that you like
that research that you I interested in again maybe some I saw some papers that came about just
the importance of different layers in in the models or more on the mathematical size I'm like we
were having some quantization of a training research so any collective that you track that you
like that you can recommend people to keep an eye on like the HQQ collection if I think we'll
also discuss at some point yeah so I'll tell you a little anecdot of my life that will hopefully tell
you my approach to papers so many years ago I was much younger back then I was invited to be a consultant
to the international atomic energy agency and everybody else in the in the group were luminaries in
the field much older much more accomplished in me and at some point we were discussing some topic
and discussion became quite heated and then somebody said to me but didn't you reach such
and such paper and I answered I don't read papers there was a silence in the room and then
everybody started laughing so this has always been my approach to papers back in the back
when I was a researcher I would go to a conference and we'll learn in one day or two days what
people are doing instead of sitting for hours and looking at papers that have been published in
the last month in the journal now being semi retired I don't actually need to go and study papers
mostly when I go and look at something is because somebody I know told me about it
and you also don't write papers I also don't write papers yes prefer prefer to not prepare
those that's why we're having this session by the way I actually go into open for the questions
so I'm going to check the website but if somebody wants to send questions that's your last chance
or you can use metrics as well okay some questions do all execution exploration models in
Lama CPP GML example through the CPU model support all conversations and is performance similar
I'm not tracking very closely but my memory is that if I remember correctly all of the
quantization types were supported I wrote kernels for them in metal and in cuda it is possible that
you know cuda support was 100% so I don't really know if that's fair
we saw from Lama to you to Lama 3 that the ability to quantize the model was impacted due to
massive overtrainings also seen bitscale and laws from Christopher Raston do you think quantization
will still be used as models become more overtraint yeah I think yes this is a real effect
I don't quite follow the term overtraint the model have been trained on more tokens and it seems to
me the model weights could actually contain more information than the model weights of per previous
generation models and that's why they're more difficult to quantize or they're not more difficult
to quantize but the organization error is higher as you quantize them to the same with using the same
amount of bits this was the main motivation for developing these four five six bits I
QK ones because they do quantize Lama 3 much better than what is in in main line Lama CPP
so the other question asks if you think that dynamic quantization approach will be integrated in
Lama CPP since Lama CPP has more key to the genus strategy and what challenges they might face
in doing so because again yeah so the person who asked this question need to because as I mentioned
earlier I don't hang around on Citer and read it so I don't know what people understand on the dynamic
quantization what is dynamic quantization assign different bits to different layers basically
this has been always like that in Lama CPP since the beginning in fact if you go in Lama
CPP and search for issues entered by cover cough you'll see an issue that I have entered I don't know
somewhere around March April of 2023 talking about variable bit like quantizations
there were of course professors coming in teaching me that this LLM quantization has
not to do with video compression and therefore what I'm talking about is nonsense but yeah this
has been in Lama CPP forever yeah thank you so much actually so we're out of time so if somebody
and I haven't been able to read all the questions but feel free to ask them to one in the hallway track
at the dinner or tomorrow at the event hopefully so thank you very much I have on time yeah