Thank you so much for joining in.
I'm going to be talking about Huggenface ecosystem for locally IML.
When I started to write these slides, it was more supposed to be like a talk more about
Huggenface, the library that we have and so on.
I'm going to be talking about Huggenface ecosystem for locally IML.
When I started to write these slides, it was more supposed to be like a talk more about
Huggenface, the library that we have and so on.
But as I started sort of finishing this slide, it became more as a snapshot and
time of where we are as a community of locally IML.
Engineers, frameworks, you know, models and so on as of today.
So I hope like you, you take it more as a snapshot and where we are and through
time we sort of look back and see like where we were back in 2020, which here is
it five and so on.
Cool, who am I?
I'm VB.
I lead on device and open source collaborations at Huggenface.
I am a local IML.
Joyer must like like most like most of you.
So a lot of my day-to-day work involves working with our partners like
Meta, Google and so on to bring the numbers closer to you to bring the
Gemmas closer to you and so on and so forth.
I build a lot of glue work which means I make tools work better together through
our ecosystem of libraries or through our contributions and so on.
So just a quick sort of poll.
How many of you have run an LM or a model on device?
All right.
What in the right room?
Perfect.
Just the showcase like what is possible like right now.
I'm just going to do a quick demo of running an LM on web GPU.
If only I ever figured out how exactly this works.
Perfect.
So this is a very quick demo of running a 1.7-billion-parameter model on the browser using
Web GPU and this is powered by MLC which is powered by TDM.
And the idea behind this is is a very simple idea wherein you take some huge amount
of unstructured text or in this case very small.
Define a schema.
So in this case.
Can you make a bigger?
Again.
Is there a visible?
Or maybe the contrast like the lights that are above?
In the interest of time maybe.
Yeah.
Okay.
All right.
So there's an unstructured text over there which is essentially an issue where a person
shows like this is the problem that they're facing.
And then you define some sort of a schema.
So for example in this issue we want to get like description.
We want to assign some labels and so on.
In a non-ML word.
You would do this with a series of afel statements, some regular expressions.
You know some control flows and so on.
But in this case we're just going to throw this to NLLM which is small LLM to in this case.
And we ask it to take this text and give us like a nice and pretty JSON out of it.
Right.
And so what you see over here is that it created a title missing password whatever
description, labels, priority, it also like estimated like how much time it might take.
And it creates like a nice issue preview over here.
Right.
This all is just like two lines of LLM calls for the end user.
However, in the back end it's like thousands and thousands of lines of code written in
TVM which makes it possible for you to be able to do this.
Right.
And this is just one runner.
This is just TVM.
Similarly we have you know other runners like there's there's of course like coulda.
There's there's of course you know metal for like apple and so on.
Let me is it this one.
Okay.
So so now that we've we've looked into like a very small like use case of what you can use.
You know some of these sort of models for how do you run stuff like this.
Right.
There's like an entire ecosystem of first of all.
Sorry for the visually explosive slide here.
But there's an entire ecosystem of tools and apps through which like you can do a lot of this stuff.
So first of all.
The like my personal favorite is llama.cp which which sort of like creates an ecosystem of tools.
There's you know like all llama.
There is llama file.
There's Ellen studio pocket pal.
Lama cpp python come to you and so on.
These are these you know like this is then ecosystem like pretty much built on top of llama.cp.
Which is built on ggml as a Roma was talking about before.
Then there is mx.
This is seemingly a new kid on the block.
It's been I guess a year since mx has been around and you can you know it's specifically for.
Apple ecosystem and specifically for mcd's you know Apple chips.
And there's you know support for vision language models.
Ellen lens and so on and there are some like third party apps like Ellen studio local chat that you can use for.
Running stuff on your phone or on your mac and so on.
Then there is on x runtime.
This is like this has been around for a long while you know it's it's actually before my time as well.
They've been here like ancient.
And there's an ecosystem of libraries that use it.
So there is on x runtime web.
Through which you can do some some web GPU stuff.
There's normal on x through which you can run stuff on on on metal as well as good and so on.
And there's an entire ecosystem of libraries over there as well.
Then there's pvm.
My example before was powered by that as well.
They have a web GPU implementation called web lm.
Then they have MLCL lm.
Which is both for for deployments as well as for like local users as well.
Then there's of course whisper.cp which is also built on top of ggml.
And there's like apps called.
I can't have like mac was personal whisper and so on.
Then of course more on like deployment side.
There is like real lm, tgi, results of gml.
And so on which is not on the slides but I will add.
So that's like not an exhaustive but sort of like okay.
It's representation of what kind of stuff we have going on right now.
And now that we've seen like what the what the ecosystem looks like.
What's what's like the typical workflow of running these models you know on device.
And when I say on device it could be anything ranging from like you know a raspberry pie to you know your Mac to even your Apple watch whatever it may be.
The typical workflow okay this is where I need to get my.
Well okay it's fine we we don't need.
Okay so typically you would go on like hugging face dot go like a chip dot go it's like a hub of models in case.
You don't know where anyone can upload.
You know models in this snapshot you can see some of the recent like really really good models like deep sea car one.
There's missed little small 24B bunch of 3D models and so on.
And typically you go you go on on the hub like you look for a use case you can you know the use case could be.
Text generation could whatever it may be.
Then you you know you select an LM of your choice.
Let's say we choose like deep sea car 1.5D it's a distilled model which deep sea.
Recently released and like a 1.5D version of it.
And then you you know you start chatting with it.
A typical example of that would be that you just use like normal CLI.
And then the descriptor of where the model is.
It's precision.
Similarly you can do it if if you use all of our then you can use like all of our that as well.
And typically this is how being the face would look like.
Not sure if if you're familiar with the with the deep sea models.
They are known to sort of think quote and quote or like reason and this is like I find this snippet very funny because I just say hey.
And then it just like goes on like a thinking screen is like what's the user friendly to me was it approachable and so on it's like a.
I find it like a really nice exercise to.
Or like just just.
Okay.
Then let's look at some some sort of facts as of like how much like the usage of these models or like how much do we see like.
Lama rod CPV coins which are like GGUF or MLX points and and so on be used right now in this snapshot.
So we see typically on the hub we see multi billion plus plus polls every month which means that roughly like like more than like 2 billion.
billion.
Billion polls or downloads of these models are happening every month.
I hope that this continues to increase.
We there are roughly 200,000 LLMs on the hub right now which is which has spread across.
You know, Lama rod CPV coins MLX on X.
And so on.
Then there are for like GGUF as you mentioned as as I mentioned which is like the file format for Lama rod CPV.
There are roughly like 80,000 of those on the hub which people which which which people are pulling like.
As I mentioned up up there and we roughly see like 6 to 6.5 petabytes per day of egress on the hub.
Which is like people pulling these models using them for either CI for for their use cases or in production of course.
Okay.
Now comes now comes the more sort of like media part is like how do you prepare these models right like these models in in most cases are are just you know a bunch of like pet or scripts in like in like.
In terms of like modeling file and you have like either like safe transfer is equivalent in terms of like model weights.
And so like how do you go from that to actually running them you know on device.
And so actually I'm going to move this here.
So you would you would typically like start with picking your desired LLM like this could be a mystery deep seek pick your favorite one.
And this could I this could be fine key on so you could have like a fine key model for your specific use case or this could be free trained as you know the provider gave it.
You would first check if it's supported in transformers which is which is a library.
By hugging face which is typically used as the backbone for having all the modeling files and then you would.
You can you can use the model we are pytorch or jacks within that.
Then you check if it's supported in any of the frameworks that you want to use it which is you know LMLX on X and so on.
First of all if the model is not supported in transformers you either open a pull request or you look through like existing pull request to see if it's being added.
If it's there in like LMLX on X whichever one that you that you want to use it with.
If it's not there you again like you search you open a pull request or not or like follow along.
As you mean that you've made it so far you convert the model to whichever specific format that they have.
So in case of MLX so in case of LMLX it's the GGUF format in case of MLX it's the safe answers format in case of on X of course it is the on X format.
And once the model is converted the task is not just done you wipe check if the outputs are good if they match the original original model and so on.
Spoiler alert it almost never does in the first case probably because of bad quantization or because the architecture is not supported properly or whatever we will go through.
We will go through some of those in the next slide.
And last but not the least like you double check the chat template chat template is how like the model knows like how to process like in like in input conversation how to output what kind of tools to use and so on.
This is also another one of those like risk vectors where you have an issue almost like 99% of the times and then if you made it so far maybe profit.
And this this is like a fairly like complex process especially when you have a new architecture or you have a non standard architecture maybe it's it's a mamma style model maybe it's like some linear attention that you go there that you that you might need to re implement from scratch and so on and this is typically like whenever I'm like explaining someone like how you know how to run like a new architecture you know like this is how like it typically would sound like.
And this of course like you know it does sound simple if it's just five bullet points but it's it's like after having helped quite a lot of you know our collaborators and partners bring their.
Their permissive models to two different frameworks there's a lot of things that can go wrong.
First of all typically like tokenizers can go wrong that's like you know which which could be that.
Just that the tokenizer does not convert well or you know there is there is an issue with with the way it would tokenize certain unicorn characters and so on.
When Lamar 3.1 was released there was an issue with with rope with because you know when you scale from like smaller context to longer context it could be that it's not supported properly for that particular architecture.
Chat templates this is in in my opinion the root of all evil on this planet.
I'm quite excited that now like Lamar or CPP has support for ninja yes and we we can be can finally say the say goodbye to a lot of these issues.
And then there's of course like now like we see a lot of like we see a rise of multimodal models you know people who use the.
The you know.
Chaggipity or whatever advanced voice mode you see like there's there's new modalities that are like intertwined with the with the same element backbone so.
You know there's there's that which is like non standard in Lamar or CPP right now and there's there's ongoing work on refactoring it and making it easy to bring vision language models and so on.
And of course like if there is like new architecture you know like for example deep seek R1 was a MOE a mixture of expert which was a non supported architecture so you know adding support for that and so on.
Here's just some some examples which I quite enjoy this is a PR from my colleague Pedro when Lamar three was released to Lamar or CPP there was a change in the tokenizer this was this was supposed to be a seemingly two line change.
But actually if you like later on just maybe click on the ref and see how the how the PR sort of progress it became like a like a sort of seemingly complex PR.
Just to get all the ruffages and like cases map.
This is our fairly recent one to add support for deep seek me three and R1 within Lamar or CPP then this is done by son who's also going to be presenting later today.
Who's a core contributor to Lamar or CPP as well.
Where he adds support for the distilled models because distilled models had a different tokenizer.
Nice and so how does this is just once a sidebar like how does like hadn't faced contribute to this ecosystem.
Of course we have the hub where it's it's it's like we host millions of models we do not pass the cost or whatever of keeping these models up.
We continue to we would continue to keep these models up and make them continue to have them be available for everyone for free.
We have our core set of libraries there's transformers tokenizers.
Safe sensors how do you face up and and a lot of other libraries these are the.
The libraries that that like all ecosystem players beat MLX beat on X beat MLC you know.
Use in in one shape of the phone.
We have specialized libraries for different backends so there's there's optimum libraries for exporting to one X to neuron.
Also to like TRT LLM and so on.
We have some some really specialized.
On device libraries so there's there's candle which is a framework which a good friend and colleague of mine.
A lot of road.
Which is cross back in and and scale quite quite nicely.
There's an ecosystem of libraries on tim book as well.
There's mistil dot address and so on.
And we have transformers.js which which utilizes on X web runtime.
For running elements and and like models on the web.
We have a slew of converter spaces which you can use to create LMLX equivalent models.
And of course we're core contributors to LMLX, TGI and so on.
And as as we go through this year.
We want to sort of increase these quite a bit more and we want to make sure that we help.
To contribute to the ecosystem as much as possible.
And this is just a snapshot of like stuff that you can do on the hub.
Like there's there's like open conversion spaces that you can use to create your own LMLX.
And so on even if you even even if you want to create like simple I matrix coins.
You can you can do that just by going on this going on this space.
You can just pass any transformers model and it just converts it and gives it to you.
And then you can use it as you want.
This is just a snapshot of our optimum optimized inference libraries.
And I have like two quick sort of graphs from the hub about like how we see like LMLX CPT and like how we see the craze of like on device.
And like these like ecosystem players increasing.
This is like the number of coins created like in a self-serve way.
We are this space that I mentioned before.
As you can see like in April is when we started building it.
In April it was like roughly about like 600 coins created per month.
And now you can see that it's it's well over like 2,500.
This is like people just like creating it on its own.
I find it quite nice to see see this graph every now and then to see like how much interest there is by people and so on.
Another one this is very interesting for those who are interested in like development on like Apple Silicon.
This is MLX you can see that for the past one year it's just been like grinding around there.
These are like unique users of like MLX coins.
You can see that they're like grinding around like all the way to like October.
And then I don't know what happened in October but it pretty much like quadrupled around that time.
And in January I also don't know what happened at quadrupled or roughly 3x there again.
So this is something which I'm which I'm monitoring quite closely to see like how we can head there and so on.
This is my last night.
Going into this year like what am I most excited about.
There's of course there is major support or gender chat time rating support in number of CTP.
And there's a multi-moderate revamp coming up in number of CTP as well which is which is very exciting.
I'm quite excited about us shipping transformers back in and both TGI and VRLM which means that you don't really need to have different set of like architectures for it.
There's a there's a little push for DDOS which is like single file diffusion models.
That's something that I'm quite excited to sort of scale in this year.
More accurate sort of quantization schemes better calibration for the existing coins.
And yeah leaner and easier packaging like what Lama file has been doing also with the external diffusion file and whisper file and so on.
That's it and thank you so much.