Hello everyone, I would like to introduce Nihau Pavo and Igor to talk about Bielek AI, which is
like very close to my heart, as you can see, I have the baths here that I got from the guys
that they are Polish, and they are talking about the Polish language model, how they trained
that and made it possible to run on Raspberry Pi.
Isn't it?
Cheers.
Hi everyone.
How did the small open data initiative became the national phenomenon in Poland?
Well, basically by creating the domestic language model that's already being used by
small and large institutions, enterprises and startups, all across Poland.
So, my name is Michael with me, there's Spaddle and Igor.
We are a small part of the team of the entire speaker-ish community who created Bielek AI
largely in which model.
And today, guys who are from the technical team, I'm more from the business part.
Of this initiative, we'll tell you a bit about what's going on with Bielek.
Thank you, Bielek.
So, let's begin.
We will give you a little insight about what's the definition of the open source.
Of the open source, what we are creating, also what is open science.
And we will talk about data sets and training and models, but training and models will be something
like what we will tell us more.
So, as you can see, there are only three of us, but we are a part of the bigger initiative, a bigger community.
A few weeks ago, we have reached 1,800 users, 1,800 members of our community.
At this particular moment, because I remember when I joined this project, I was around 50 or 60 person
who was collaborating in this project.
But to reach this number, this open source community number,
it has some action to be taken to start it.
So, there is some kind of pretty interesting situation.
When after data science, big data science, conference in Poland, in the 2020-22 year,
our founder Sebastian Konradski and our master of data sets of data, a drink wasj,
we're talking about the lack of the linguistic corpus, which consisted only of Polish text data sets.
Of course, we are living in the big AI LLM's moment.
So, we have to find good sources of such text data sets.
And they managed to do it to provide such high quality, large language data sets.
Of course, there were some signs of our critical voices that we want managed to do it.
People were telling us that maybe 100, 200 gigabits of the text data sets.
And it will be everything. But for this moment, we are about, we have reached the level of 2.8 terabytes of text data sets.
It is open source, it is open source, it is source of the data sets in the Polish language.
Text, we are still providing more data to it.
We are collecting data via some web scraping, maybe some that also closed sources,
but we are collaborating with other institutions.
But it is for everyone. Everyone can join us to help us together,
and also everyone can download it by using our paper package.
But it is only for free lines of code. Everything is also classified.
You will know if it is low data, if it is category low, high medium of the quality.
If it is medical or not from the cooking, everything is placed here.
We have our dashboard. We are providing information about our data.
We are providing metadata to inform you what you are dealing with.
And everything is done by society.
And to reach such number, it wasn't easy task.
But I remember also one time when we were about to reach a mild stone of 1 terabyte of data.
It was about December of 2020.
And the case was before it.
Some folks reached us.
They were from the Academy of Computer Center to run it.
They were providing from 1975 a computing power,
and also IT infrastructure for the science for open projects.
And they asked us, do you want to collaborate with us?
You've got a data, you've got a specialist, you're building a big community.
We have a supercomputing power and let's collaborate.
It was a Christmas gift, which we were dreaming of,
because we didn't have a computing data computing power.
We were gathering data and it was our creme de la creme.
But with the help of Cepronet, we can now create large language models,
which is our sharing on the top.
So we managed to work together with the Cepronet.
And now we became a little Santa's helpers who are giving large language models for this society,
as open source, and it's always for free.
And we give a voice to Pa.
Well, they were, it was in January last year that they kind of started the computer.
When they were warming up, where they were warming up to the GPUs,
they have an opportunity to train the first version of everything,
everything is kind of fuzzy logic, mostly different models.
We do a special training and environment preparation.
So all pipeline is kind of reputable.
We can rerun them.
The data itself is available to all of you.
You can people install the package and download.
And it's ready for you after a couple of minutes, probably hours.
And you can, of course, run the pipeline and try to train it on your own.
Of course, it will take some days.
And you probably don't have equipment as we had before the superior computers and their reach us.
Eventually, we run a lot of experiments,
run, try to understand that there are issues with new GPUs,
because the HPC was using, it was one of the first in the world,
BH200 GPUs.
So many of the libraries completely didn't compile at all.
So we had to come up with the code and a lot of things went through.
With the model number two, we started to do the synthetic data,
get some experience with proper ideas around the synthetic data,
and how to approach the parameter, how to get the heuristic around that topic,
how to cover most of the things that we understand as a viable for the model.
Because the model itself is, we try to cover different use cases and ask business,
education people, all the kind of artist movement to give us instructions,
give us a clue what you want to have within the model.
And of course, there are users of that.
Eventually, we also started to find what are the second stage,
fine tuning, DPO, and a third stage of the training itself.
So we are doing pre-training, fine tuning, and DPO,
and eventually instruction model as a point and alignment as well.
For us right now, also evaluation is a key feature.
So to ramp up, we had the first model that is 1B, train on single,
then version number one, with a proper paper,
you can go to archive accents and see some information around it.
It was based, of course, on the mistral 1.5,
but we continued pre-training it for like a number of days,
and then a model number 1.2.
It was, in fact, a train on kind of more days on larger amount of data
around 200 billion documents that we kind of filter out.
And documents and tokens, right.
So as I said, told you about evaluation,
we prepare our own empty bench, and it's not translated.
This courage you to do translation by Google, and you have to localize it.
Because you know, that question are asking,
where are you going for holidays?
How why?
Nobody in Poland would go to Hawaii.
Sorry.
It was too costly, and it's time to, of course,
maybe some people do.
But they go for, to the seaside.
It's to some Kalushki or whatever.
And so we prepared our own kind of,
those specific parts in conjunction with what we do
about Polish culture.
And of course, for that, we open a Polish LLM leaderboard,
where we evaluate all possible models.
Within one day, when the model is published, we rerun it,
and eventually we are having very good results with
model number V2.
We do also have scripts and preparation for all the formats
that you want to have.
We are on Olama.
We are on, on hugging face.
So you can approach it.
We try to be open.
So whatever code we are talking about,
they are on the heat GitHub itself.
And you can help it.
Those are the results of those models that are quantized.
They're pretty good.
I must say that the model number eight,
quantization which was eight is even thirder.
So it's better than the model without quantization,
which surprises us a little bit.
Right now we are in progress still,
but we are training in a model even smaller model.
There's still some form,
there's still the knowledge from bigger models,
and also have our own tokenizer
that covers some specifics of a Polish language.
And we already tried out on different devices.
So this model runs on Raspberry Pi,
runs on Android devices,
some other set out boxes.
There are numbers of getting started in tutorials.
So everyone can help this guy,
administrator of some network of the school,
and can go there to see how to install,
how to use it,
that people are doing the workshops on schools and universities
to set up the rack,
to set up, of course,
we also show some use cases for local communities,
or companies,
but we try to give the instruments,
and then need to do their own work
to get to know what is inside,
what is AI,
how to use the models,
how to run the models,
and how to evaluate them.
We really want to say thank you to all the guys
that's here,
it's only parts of the core team,
and also to special thanks for AGC,
Krakow,
super computer centers,
center in Poland.
So thank you for listening,
and if you have questions,
then go ahead.
Thank you.