Hello, everyone. We'll start the next talk. Robert will talk more about the work after Ivan
has introduced quantization. Now he's working with the result of it. And the funny thing,
the first thing, and being here and being at the Devroom, that's the first time the guys
actually met and they will talk to each other. So this is a great place to meet. Robert is
all in your hands. All right. Thank you. Hey, everybody. I'm Robert Collins. So let me just give
you a bit about myself. It's just some background. And I have to read from my notes I apologize.
I'm not used to public speaking. But I have a very background in visual arts and in front
and full stack development and enterprise software development. And lots of things that are not
related to AI. But I was interested when I went back and completed my master's in 2019,
I did a CAPST project, creating an ecosystem to scheduling notifications for e-learning apps.
And I built NullSTM model to do that scheduling. And that was a lot of fun and very interesting
to me. So I was inspired to pursue it further. I completed a elsewhere postgraduate diploma
and applied AI ML. And I was sick for a couple of years for a bone infection and it took me
out of practice for a long time. And so I found it difficult to continue in the work that I was doing
and web development. So I decided to apply myself to software in AI. So I took a position in
reinforcement learning. And I'm generally interested in AI and how AI can make immediate and
small positive impacts in people's lives. And I feel like that actually is starting to happen
in large language models, which is the thing that came right after the LSTM model that I built
in 2019. But I did actually get to see it while it was in a university early code generating
LLMs. So this project is llama GGU of optimized. It's up on GitHub. It's a toolkit for optimizing
multi-lingual model quantization through iterative refinement of importance matrices. So you'll see
they'll use the word multi-lingual related to this project and that really has to do with the
background of the project. Really what importance matrices allow you to do is to ensure demand
competencies are preserved from the model. So for example, in the community code competency
has been something that has been of interest to people. They've been building important
matrices that ensure that the model performs as well after quantization as it can in regards to
code competency. So I have a background or sorry for the remainder of the background about the
project. I have a friend who is a language instructor. She's Bolivian and she works with students.
She speaks Kechua or come from families that speak Kechua. So I kind of speak a little bit of
Spanish and also Portuguese. I mean emphasis on kind of, but I really I actually hope to
retire in Brazil. And so I have an interest in language. Maybe that's why I was interested in
this elementary model from Barcelona Supercomputing Center. And that model encompasses over 30
European languages. And I wanted to quantize that model for Olamah because it's a popular runtime
and it didn't have it. And that uses LMSE++. And I noticed that LMSE++ offers important
matrices to produce better models. But the data sets that I found just weren't really appropriate for that.
So, I mean what I mean by that is that the data sets that were available to generate
important matrices focused specifically at the easiest ones to find were just on English and
or code. So I worked through making my own for the model and using it and doing my research
with perplexity AI. And I realized when I went back to write about my experience and show what
I had done that I had made some assumptions. Thanks to the type of research that we get with
perplexity AI, we're just not actually well grounded. So I wanted to actually ground it better
than that. And I went about doing that and that led to building this project. So let me see.
So to give you some of the background that leads to the use of the importance matrices. The first
paper here shows the preservation of domain competency is orthogonal to the minimization of
loss and quantization. What this means to the authors is that you should use calibration samples
in the preservation of domain competencies. The second paper is an addendum about how to build
your data sets for important matrices. And it's not showing anything of use specifically to my
projects. Like it's not really about important matrices or LMSE++ is more about calibration data.
But the idea I think provides a default assumption that balance data sets are more appropriate
to the preservation of calibration data. And then what that means is if you have a model
that's designed to interact in English, Spanish, and Portuguese, you might be tempted to just
lazily rely on a wealth of English language data in your calibration. This would be informed
from an understanding that in pre-training, whenever you're building one of these large language
models, you use so much data that you provide information about other languages in the process
because languages are related. So just training something on English, you expect that it will
learn essentially various European languages pretty well out of the box. And that kind of is true
with just a sprinkling of other languages. But if you're really interested in preserving a wealth
of expression in other languages, if these other languages really are important to you, maybe
you shouldn't do that. And then the final two articles here really make the same claim as the
first article, at least for me, and how I'm going to use them. But they tie it specifically
to language competency. So the tools I'm using obviously are Lama C++ because it provides an
importance matrix, which is generated from a data set, and that data set is I'm using it is a
sub sample of pre-training data that you can get from models that are gracious enough to provide
that or from the teams. The data can be chosen to reflect competencies of import to the user.
And I'm also using it because Lama C++ because Lama uses it and it's a widespread and accessible
runtime. Other runtime or systems can use it as well like Lama Studio. And all of that can change.
There's I was going to say actually I wrote my notes that there's a rumor that Lama is
possibly working on the runtime or perhaps a plug-able system that actually a draft pull request
came in the day before us then started where they set up a plug-able system for MLX. So they are
in fact heading in that direction. It's worth noting also that unslawed recently build out internal
tools for dynamically leaving some weights, unquantized for multimodal and for text-only models.
And they're not saying that they won't open source at some point. Like conversationaly
fully unofficially just to me and not in any way in a binding sense. Like I don't want to
prejudice what they would do or anything. But it just sounded like in a very casual conversation
that they would be open to open sourcing at least the text-only variance of that.
And I think the challenges that are required to make this system work with theirs,
the project that I'm going to show you here, is not too big and that's because with this system
does it orchestrates the CLI largely for Lama C++, it provides a framework for using that system
in an analytical fashion. The second library is when I need to use, there are places where I prefer
to use the use Python and integrate directly rather than using the CLI and so I use Lama C++
Python. So just very quickly we've heard a better demonstration or description of this at the
beginning of the last presentation. But an important matrix is a mechanism for identifying which parts
of a neural network are most crucial for its function. So now you have a sense of what the problem
kind of is and what the approach that Lama C++ provides to deal with that, how it kind of
uses an approach that's similar to calibration data in quantization. What this project does is
addresses the challenges of quantization and quantization loss in language diversity and models
with specific domain competency. So the community, again, has largely been focused on producing
a single dataset or like a small number of them that reflect standard domain competencies
like mostly English and code. And there's been some additional focus on French and Chinese and
I'm sure you could find others if you really looked. But I feel like we could do, we can, the tools
are there to conform it to whatever the competencies are that the model was designed for. So this is
a visualization you can produce from the work that you do, working with the tool kit that I'm presenting
here. Let me describe this. On the left to right, the Z-axis, what's called KL divergent
values. Some of you may familiar with that, if not, there's something called perplexity,
which measures like the loss in the output. This is before that if a lot,
the widgets level we're looking at the individual widgets that are used to produce output.
And then the height here, the density, describes the distribution basically.
So like all of the values that are yellow, that are very high there, that's because almost all
the values are the same. And so each one that you're looking at here in depth, it's not really
a manifold that's like a series of individual quantizations. I forget what I was about to say there,
I apologize. But yeah, so there is a metric that's provided with each of these that describes
that converts that distribution into a single number. And that's just based on community input,
the community discussed, and described using the KL divergent's mean. And then also that the
90th and 95th and 99th percentile larger percentile values were more important. And so I use
like a weighted mean of those values. And it could be a different thing. It doesn't have to be
those values. You might prefer to use like I actually think in retrospect, I would like to use the
CRISPR test instead, which is a different singular value expression of that distribution that
pays attention to the outliers. But that's maybe a great tool to you for the project to allow
the modification of a metric in producing these graphs. But as opposed to having a modified
source code for production of this graph. But the important thing is if you look at this metric,
there's diminishing returns. It goes from roughly 0.39 to 0.34, 0.295, 0.294. And a thousand
you'll see it's different. And what this is, is this is the supplement for Mandala. I was describing
earlier. So on the first one, the one that's closest to us, is the unquantized model versus
quantization with no importance matrix. And then it's 250 samples per language, over the 30
some languages. And then 500 and 750, really? So yeah, that was from the initial run that I was
using to go back and verify. And the way that I approached it was slightly different. It's
demonstrated that if you selected from a data set roughly the top 10% of chunks from the data set
in terms of populations of outliers or something like the KLD values at the 95th or 99th percentile,
that you can get better results. So for example, what was shown in the discussion was that if you take
500,000 samples and we're going to get down to 45,000 samples, maybe you can get a, it's only a
couple of percent better, but you can get a better result out of it. Looking at the, there's if you
look at the metric here, there's an elbow obviously at around 500 samples. So it might make sense
if you really wanted to maximize your results to go up to 5,000 and reduce down to 500 samples per
language. This is the same chart just showing chunk by chunk metrics. So Lama C++ provides an iterative approach
and this toolkit provides the same. Like the Lama I matrix for example allows you to compose
an I matrix and then add it to the fashion. You can build a data set and with a certain amount of
text data and then you can get more text data, refer to your old data set and produce a new data set
that's even larger. In fact, I created a small poll request that you could build one data set,
build another and combine them. This toolkit is designed with the same sense of visibility.
There's an emphasis also on storage space for people who have concerns in that fashion.
And there's a usage guide. There's even a podcast on the read me for this that I did in
notebook Lama to help make it approachable. I'm going to rush through for lack of time.
The I matrix data set and KLD bench are the two core tools that you'll actually use. The other
two core tools are the tools that actually do the work, but hopefully you won't have to use them very much.
The top one is an interface to download and randomly shuffle a few so desire
data sources for making an important matrix with your test data. It's a plug-able system built
with it provides a default access to the Oscar library as it so happens but you can build it for
whatever. In fact, I started working on another model recently called Sailor II and some of
their data was only available in unlabeled and randomly shuffled and I just needed a couple of
languages from it. So I wrote a, I modified my plug-in to use a library to identify the language
and if the confidence was high and then we'd added it into the data set and it makes it work very
simply. KLD bench accepts a model or the loadages that are already generated and it orchestrates
the other two scripts. It runs compared loadages chunk by chunk and all of this is resumable.
It only keeps two loadages for the data on disk at a time. I think that's important to highlight
unless you generate it in advance. If you have terabytes of disk space, then maybe you don't
want to regenerate them over and over again whenever you're doing the baseline because you'll be
regenerating the same data so you could use generate loadsage just once. These are the visualizations
tools that are provided. I have a whole thing on early stopping but as I'm running low on time,
I'm going to skip that. The first thing that you saw was the composite comparison. There's also
a tool that provides a similar manifold looking at chunk output for the KLD version's values
for a single run on the model and then the read KLD bench marks is a plain text tool.
If you this is the early stopping that I kind of skipped but it just provides a way to generate
to segregate out a section of data that you're going to use for testing for comparison purposes
that are not going to go into later whenever you generate quantizations with important
spaces. I have to wrap up now so I apologize. I just wanted to show you that there's excellent
documentation as well and yeah that's just the final single run visualization. Thank you very much.
And of course you can talk always to Robert after the talk there's plenty of time and tomorrow
we have the a Plumber's Conference if you didn't know so we can we can meet.