Let's talk about learning about Lumigators.
OK, thank you very much.
And hello, everyone.
I'm David A and Art.
As you can tell, for my T-shirt, I work on cutting-edge technology
at Mozilla AI.
I didn't change the T-shirt, sorry.
Also, they kind of ruined my joke before,
because they were telling me, nice T-shirt.
So I still hope that some of you didn't hear that part before.
So, in other things, I want to give a shout out in advance.
I kind of lied to you, so you could join this talk.
It should have been evaluating a Lambsmith simpler.
But I hope it's still fine.
I think about waiting is hard, per se, as a problem.
What we are striving to do is to make it simpler and simpler.
At some point, it will become actually simple.
I hope we are already a good way through that.
So, this is our starting point.
About one year ago, we started interviewing people from companies,
research teams, and trying to get an idea about what their experience
with our language model was.
And we kind of found that it was aligned to our own experience.
Just a few months before doing these interviews,
we were participating to these new reaps, fine tuning,
one GPU, 24 hours, one LLM contest.
And the idea was for people to fine tune the model using only one GPU
for 24 hours at most.
And they had to evaluate how their models went.
And we used what was available in terms of academic benchmarks
to evaluate these things.
And first of all, it was a daunting job.
We had to run on many different models and architectures.
We had to run evaluation that came out of the box.
And we had to rely on kind of standard data sets,
which didn't really reflect what a very specific use case could have been for anyone.
And so we decided to try and build something that would have helped
user work on their own specific use cases in the simplest possible way.
So our kind of user target is somebody who codes and works with AI and LLMs.
But it's not necessarily publishing on your reaps.
Like there's plenty of people who just want to use models without having
to develop models from scratch or fine tuning them even.
There are plenty of models coming out which are already good enough for us to be using
in some application.
We just need to understand like which is the best one for our specific use case.
And we also want to be aware about the fact that just trying all these models
take a lot of time.
And so we would like to make this step easier for everyone.
So we're a lot of people.
Right now, the actual contribution is just a dozen.
It's great that we have some which are not in our team already.
And we would like to make AI in general easier, more transparent.
Understand what is broken and possibly fix it.
We are striving to also contribute back to the project that we are using.
Also trying to avoid reinventing the wheel.
So integrating existing tools where they are available.
And finally, only building what is missing.
So sometimes we do mistakes.
We try to build something on our own.
Then find there's very good project we can rely on and then we move to that.
And just try to use what's available in the ecosystem and give back to the ecosystem.
So building the aggregator.
This is the definition of what we get or how we want the aggregator to be.
So a platform that guides users through the process of selecting the right language model for their specific needs.
It's very ambitious and you're still on the path of building the final version.
But we're already some key features which I think are worth sharing.
So first one is kind of infrastructure agnostic.
That is, it builds on a stack that can be ran on a single computer on some laptops.
On computers we don't without GPUs.
It can be built in a local cluster or directly in the cloud.
It can be hybrid.
It can be fully distributed.
You can rename it.
It relies again on not just on existing tools but also standards for interoperability.
Just to give an example, the OpenAI API which is a defect of standard for communication between the large language models and tools going to use those models.
It's something we rely on so we can easily switch from somebody using OpenAI to somebody using an alarm file or any other kind of local model.
We have built an API.
This is how our system basically relies on.
Over this API we have built an SDK and we have also built a UI so that users who want a more friendly interface can try and use it.
We decided it to be extensible not just by us but by the community and for the community.
So you might not have heard about the aggregator yet.
So it's great that you're here today.
But the project has been like publicly developed in the open since about six months ago.
We just released an announcement of our let's say first MVP.
The first version we thought it was worth really advertising somehow.
And we would like the community to tell us how to grow what is most important and useful for them.
And possibly to contribute in any possible way of capability.
Which was a very specific use case to get started with because of course evaluating any model for any use case would have been to complicated to.
Immediately provide something for people to test and we focused on summarization as a specific use case.
This is how the UI looks like.
It kind of looks like many experiment tracking tools but it's actually way easier than that.
The main idea is you have two functionalities you can upload the data set and you can run experiments on these data sets.
The fact that we kind of.
Over fit on summarization for now is allows us to already suggest you some models that we have tried for summarization and we consider good ones open source models.
We also allow people to use existing APIs for close models if they want to see how an open model compares to an existing API.
In case they are thinking about the transition from one technology to another.
And we already provide a lot of parameters which are pre chosen for how the model generally works well in the summarization use case that can of course be tuned if people want to delve deeper into this.
And for this we also allow people to directly access the API and show it in a few slides where you have a much more fine brain possibility of choosing parameters.
Finally we show different models within the same experiment.
We allow you to either death deeper in a single model to do some vibe checks evaluations so see how the model translated compared to the ground fruit that was available and see a few performance matrix that are calculated again.
Given this is the summarization example performance matrix are chosen specifically for that so even if you don't know exactly which ones are used.
You will find the ones that are pre-selected for your specific use case. Of course you don't know what the number means exactly and this is something we are working on right now.
We have also provided some documentation to tell you how to tell a part one matrix from another one and what it means for one matrix to be higher or lower.
You can also compare many models at the same time and see how they perform compared to each other and also we are starting to have some extra matrix that tell you something more to allow you choose between one model or another.
Maybe the model doesn't perform as well as GPT but I don't know it takes only 240 megabytes and so it's worth trying for your specific use case or it had a very much smaller runtime rather than another one.
Some of them have a very low one because they are just APIs that we are hitting. You might have or not have a GPU so it's worth knowing whether you can actually run it or not.
This is a very high level description of the architecture of our tool. We rely on a ray cluster for computation. One of the advantages of kind of devoting a full ray cluster that can actually run on your own laptop.
I don't think about a cluster as many computers doing the work for you. Is that automatically as kidials your jobs so you can really just tell it.
I want to run evaluation on these models and it will take care of in queueing them one after the other depending on the amount of resources you have available.
Another nice thing I saw people talking about models which are LLMs or small LLMs that are not necessarily Python models.
So ray cluster is a diagnostic of which kind of code you are running. What we are calling is basically a shell script.
So when we call a Python script we call Python script name minus minus config and configuration.
So if you have something that is not a classical Python model you can still run it in this infrastructure.
Of course we have not written any plugin for that but we can work together to make it true.
Then there is the rest API is based on fast API and surface is the different functionalities that the software is able to perform right now.
We have an SQL database which right now tracks the main information about what you shared on the system.
For instance the data sets that you have made available and the experiments that you run.
We are moving a lot of the information we were writing in the SQL database onto MLflow because we decided that to allow people who want to have a more granular way of tracking their own experiments is worth using the third party to instead of writing everything on our own.
So coming back to one of the first slides we don't want to reinvent the wheel and this is how we are acting iteratively on our tool.
And finally we have a nice three object storage. You can think about a WSS3 but what we are actually deploying for custom personal use cases is min iobase the s3 object store.
So it can run on your computer again the language it talks is the s3 API so you can easily exchange it with another technology if you want.
This is the web UI you already saw it before this is just another page is the data sets instead of the experiments.
Once you start your system and in our case it's one comment make local app after you have put directory at the repository you connect to local hosting your browser and you access this API.
Another thing you can do you connect to loading the rest API and port 8000 you can see our fast API based rest API and you can already see that with the API makes available sorry I went out of the string the API makes available is way more parameters that are usually available in the UI.
We are now navigating the trade-off between making many functionalities available and making the system actually simple for the user to use.
So we are hiding a lot of these and this is where you can come in telling us I think it would be better if this was exposing the UI instead of it's good to have it there and just going to use it as an SDK or directly call the API.
And then have the ray back hand which is what I talked to you about you can check the logs of the jobs that are running understand better what's happening but at least the central part of this is already surface in the UI so you don't have to use it to understand what's happening.
And finally as I told you before the object store in this case you can connect again locally and see what datasets you have saved and what results you have generated by running our tool.
We have the SDK, I couldn't really show you the SDK itself so I'm showing you how we are using the SDK we have made a test Jupiter notebook available that shows you how to use the SDK to programmatically call all the functionalities that we make available in our code.
And here is also some documentation about how the different matrix work and how you can interpret the results from this specific evaluation.
So the main the main decisions that we talk about this tool where you should be able to plug in and out different kind of components to move stuff around so.
First of all you can have different classes of supported models we are supporting high-infase transformer models API's again open API is the first one but there's a mystery API for instance local models BLLM, Lama files, Olamma everything which supports the open AI compatible APIs.
Now I know the question of the month is well before it was can it run doom now it's can it run deep seek so.
It couldn't run like what's what's a master cannot run deep seek I changed one line in the configuration and I could run deep seek using go Lama and the local served version of Olamma my laptop so I can say can run deep seek I think.
I'm going to put send in PR so you can do it to different type of jobs again you can customize the jobs that you run right now we have annotation inference evaluation.
But you can also have composite jobs you can choose to have a workflow that runs inference evaluation all together so you can just submit everything at once and have the results at the very end.
You can have different levels of access again API SDK UI and finally different kind of deployments is a totally before sun components can be not local you can move them around as long as you have an IP address to point.
So what next these are very high level our next steps we want to make our evaluation results easier to interpret we want to add further matrix so new ways to evaluate what we already have that is summarization.
And we want to expand our use cases and translation is one of the following one partially because it's kind of close to summarization in terms of analyzing language results with respect to ground truth but also because we found it's.
relevant use case and with potential also collaborations with the.
Firefox itself who's one of translations and kind of work as a.
Ulysses packed you know like we build the evaluation and we are also able to develop our own tools against others so whenever we do it to which is not as good as we expect people can use this other tool to evaluate whether it works well or not.
Thank you very much so if you want to know more there's a list of links which are available on this lights which are available on fuzzle website.
And if you want to contribute I like to side these power low of participation that I learned about in 2006 when I was doing my PhD.
There's really no barrier to the entrance in the last 24 hours we got two pieces of feedback one of them was oh I didn't know.
Where to get the data set I could test this stuff on and we realized we didn't provide any data set to play with we only suppose people they already had their own or already knew what they could use for that.
But it happens the other feedback I got was a PR of a one word type in the documentation.
I'm the 41 I'm Italian sorry I'm not sorry for being Italian I'm sorry.
I'm very happy for being Italian.
So it can really be nothing and you just choose your level everything is going to be super appreciated especially if it's something that helps this grow more.
So thank you very much everyone.
Do I have questions.
I'm not seeing any hands raised do we have questions in the chat.
We don't have questions.
I already know all your questions.
Oh no there's a question.
Thank you.
Let me get you the high.
Yes.
Thank you for always.
Well that's not mine.
Hi thank you.
My question for the UI.
Why did you decide to display it in a list with all the different responses?
Like this one or the previous one.
This one.
So this one is not useful to compare one model to another.
But one thing we found.
I mean we all realized.
Is that when we actually run evaluation matrix which is what you would expect in a very scientific approach.
You just have the table you see what's the biggest results and so on.
Beauty is in the eye of the beholder.
So look at this ground truth.
This is the official ground truth provided in the dialogue some data set.
This is like an academic data set.
This is provided.
It's it's a great summary.
It's a one sentence summary.
If I run an LLM on this.
I can tell it's just keep it one sentence.
But it might start giving different wording.
A bigger sentence and everything.
If I'm fine with that sure thing over there.
The result I might get from an LLM might not be great.
And so having also a chance to just see the individual examples.
And for instance sorting by every single evaluation matrix.
And comparing the ground truth what you actually get from the model.
In our opinion was a good way to always keep the user in control of what kind of mapping there is
between what they perceive as good and what they get from the evaluation matrix.
Do I have more questions?
No.
Thank you. You're free to go.
Thanks a lot.