Talk to you about Firefox AI runtime by Tarek.
Hey.
Sorry, it's a picture for a friend.
All right.
Hello, everyone.
I hope you're enjoying your amazing weather outside.
This is not Brussels.
I don't know what's wrong, but it's pretty amazing.
I hope you enjoy your weekend.
So I'm Tarek's Yaday.
I work at Mozilla.
I used to be a pre-involving Python community.
I created the French Python user group, FP long time ago.
And I'm part, I also wrote some book about Python.
And I'm part of the Firefox AI ML team at Mozilla.
And I have 15 minutes sharp to finish my talk.
That takes 20 minutes or this guy will punch me.
All right.
So Firefox AI runtime goal.
We want to provide an inference API inside the browser that can run offline.
That we can use for all this stuff.
We want to do inside the browser.
But that we also want to surface to the web extension developer.
Because we believe that it shouldn't be just our team building cool stuff on the top of
inference.
It should be something that anyone out there could do.
So that's roughly the goal we have with this new set of API.
So inference.
That happens in Firefox some years ago already.
Do you guys use Firefox?
Everyone.
Yeah.
So yeah.
And I'm pretty sure a lot of you are not speaking like a native English speaker.
So you always get pages in English.
You might want to translate.
And that's where the awesome Firefox translation feature comes in.
It's running fully offline.
And when we started to add last year some new AI feature.
There were a lot of drama in the community.
Wow.
Firefox is doing AI in the browser.
But guess what?
We had that back in 2019 with that project.
It's based on Bergamot, which is also based on Marion MNT.
So NMT.
It's neural machine translation engine.
So it's roughly an inference runtime that's specializing to doing translation.
It's a very neat project.
I don't have QR code.
But this one is easy to write.
Browser.mt and you get all the details.
And it's using RNN models.
It's a trend on language pair.
And it's re-fast and it works really nice.
This is the architecture as of today.
So when Firefox translation when you want to translate the page.
It's going to fork a dedicated inference process.
That doesn't use for process server yet.
But I guess that's one of the candidates for you for you work.
And once the process is forked, it's running Bergamot.
The lib that is compiled as wasom.
And it calls our service that is online called remote settings.
Where it's going to grab the wasom file.
The model that's it's going to use.
In my case, for example, English to French.
Those are 10 to 20 megabyte each.
It puts them inside your browser in XDB and then runs the inference.
And there is also this cool project called gemology.
That is going to take part of the work that's happening to the inference.
To run it's natively on the best back end on your computer.
So if you have a money and you have the latest Apple M3.
It's going to pick the neon i8 mm back end to do with the metrics multiplication.
And it's going to go very fast.
And if you have less money and you have like a smaller CPU,
it's going to look at it and see what it can use out of it.
So this is what we have with far-fruxed translation.
But we don't want to do more stuff with inference in the browser.
So when we started to look at it last year,
we couldn't use Bergamot, which is specialized in translation.
So we started to list all the features.
We thought about experimenting into the browser.
So describing image for this, we need an image to text model.
Recognizing words.
For example, what if when I visit a page,
I can detect that the word is blue cell is in the page.
And that's it's a city.
So that you would use an nomenity recognition model.
What if I could classify all my tabs with the title of each tab.
Some for traveling, some for the best French fries in bullysad,
etc. etc.
That would use a text classification or sentiment ananesis model.
So that's another one.
Semantic search.
Grab your text from the page, create vectors, index them,
and then you can do some cool stuff with this.
etc. etc.
Text to speech, speech to text.
There are a bunch of models we would love to run offline into the browser.
So we looked at all the projects out there,
and we decided to use Transformer JFs.
So Transformer JFs is a project run by having phase.
That is a JavaScript port of hugging phase,
Transformer project that is in Python,
which is one of the most used stuff out there for people
that build machine learning and want to do some stuff,
some training.
And the Transformer JFs project is built on the top of the on-extrand time.
For Microsoft, which is a inference runtime that can be compiled in wasm,
and that also can run against your web GPU.
And by using this stack,
we're enabling, we're able to use over 1,000 models
that are from hugging phase,
and all the task I've described in my previous slide.
So that gives us a super high level API we can use to do inference in JavaScript.
Here is an example of using Transformer JFs.
So if I want to describe an image with cats,
I can pass the URL to the pipeline.
I just say, yeah, I want to use image text model.
I provide the name of the model,
and I get back the description of the image.
And that is super high level.
This is doing a lot of stuff under the hood.
Transformer JFs implements all the classes that do the processing and pre-processing,
because when you get an image, you can't pass the URL like that.
You need to grab the image,
and then you need to convert the image into blocks of areas.
You want to pass the model and etc.
For the model, you need to grab the model from hugging phase,
download it, put it in your cache on disk,
and then run inference engine on the top of that.
So all this is abstracted away by the Transformer JFs project,
and we think it's a very good way for people
that are not machine learning specialists,
to be able to experiment and play with inference.
So we shipped Transformer JFs into Firefox 133.
We added the Onix runtime web,
which is like the wasm runtime from Onix.
As a back end, a long-site barricamote.
So now when you're running an inference process,
if you use the feature that's specific back end,
you pull that from our server, compile it,
put it in the inference process,
that is completely isolated from the web page,
which makes it more robust and secure.
And then we store all the models into index DB
in a way that's cross origin.
So if I download the model A and it's used in several places,
it's not going to download it twice,
it's a single cache of models.
And then it uses the same inference process
than Firefox implementation.
So that's probably what we've shipped into Firefox.
And the first feature that uses it is in PDFJS,
so who uses PDFJS?
All right.
Did you know that you can add images in PDFJS?
Who does that?
All right.
Okay, it might sound like a weird use case,
but there's one use case is when you want to add signatures in documents,
you can put your signature like that.
But for us, it's also a way to start experimenting
with inference in Firefox,
because you can put an image there,
and then we can process stuff on the image.
So the demo here is about that.
Also let me click on there.
So here it's a PDF,
so this is me with my cat.
And I'm taking a longer time to resize it
because the model works in the background.
And yeah, I have a caption.
So this caption was generated by a small,
image-to-text model.
That is quite small.
It's around 200 megabytes,
if I remember the size on disk.
And it's 180 million parameters.
It's a bit encoder for the image,
and a GP2 decoder for the text.
Yeah, so that's the demo for our text.
So that's, I don't know,
like maybe 15 lines of code in the context in the PDFJS side.
So that's cool,
but we're not a lot of people that want to learn.
And we need to help from the community.
So, and yeah, so we want to enable
the music community to build some cool features.
We have not thought about.
Don't zoom on, don't look at people's faces.
This was AI generated,
and it's very creepy.
Very, that's scary.
So web extension AI API.
It's a wrapper on the top of the AI API we use internally,
that we want to provide to the community.
It's available in a nightly.
It's prepped off in Firefox 134.
I would use the nightly version because I've already fixed some bug there.
So if you use the 134, it's not as good.
It traps the runtime,
and it offers a very high level API to influence the browser,
with low friction.
Of course, you could do that with vanilla,
transform a jazz, but you get a bunch of benefits.
It runs in a separated process.
You have a cached, like a list of models
that are cached into your browser,
not in the web cache, but it indexed TV.
So we don't have the same restrictions.
And we're going to iterate on making it faster
by doing some native things in there.
So it's just going to be better.
And it gives us a way to iterate with the community,
because we don't know yet what could be a good API
to do AI in the browser.
So our hope is that through the experiments people can do,
they will be able to help us shape the best API
for the browser for inference.
So that's how it looks.
It's very similar to what you've seen before.
We have under the trial namespace in web extension,
ML namespace, where you can create an engine,
here I'm creating an engine, for a summarization.
I'm trying to get faster when I talk because of this guy.
It's looking at me.
And it's going to pick a model by default for you,
and you can run it, and it's going to do inference.
So I have a demo here.
Sam stuff that in PDF jazz, but in a web extension,
all the code is in web extension.
So here, if I load my temporary extension,
and I grant permission for people to download model from the web,
that comes from another presentation when I talk to a lot about that.
And here, if I write click, it's going to download the model,
the first time you use it.
So that takes a little bit of time because it's big files.
But once they're there, the next run is going to be super fast.
What happened?
Oh, oh, crap.
Sorry.
Oops.
Yeah.
Almost there.
Yeah, running inference, and then it
describes what's in the image.
So that's the three lines you've seen roughly.
And if I run it again here, it doesn't
go down to the model because it's already cached.
That's it.
Thank you.
Thank you, Eric.
So you have to memorize the first link.
It's an invitation.
So just looking on the internet, it's the discord, the musala AI discord.
You can go there.
We can interact.
You can try to build cool web extension.
And we can talk about it.
And the second link is the documentation.
And for the record, every slide is supposed to be available on
pre-tot, which is the first website.
So if you're a speaker here, please upload your presentation to
pre-tot, so people can find them later.
Thank you for joining.
Do you have any questions?
Yes.
I don't want to ask you, is this mozilla browser specific API or
there are discussion to make it?
Can you repeat, sorry, I don't hear well.
Is this mozilla browser specific API?
Or is there discussion to make it a standard for the whole
JS web APIs?
Yeah.
So that's the next question.
So right now we're a musala specific.
It's what we call a trial API.
It's under a trial namespace when you build web extension.
It has to provide a specific permission in your web extension.
And that's mozilla specific.
Chrome, on the other hand, Google, they're working on something at the browser level.
But we think it's a permit rate.
We want to iterate with the community to try to understand what it means
to do inference in the browser.
At this point, we don't really know.
So our strategy there is to try to iterate with this web extension API.
We want to do some cycles.
And at some point, if we feel like there is something
surfacing that we could propose as an official API.
For the browser, we will do it at the W3C level.
But for now, we think it's a bit of a permit rate.
Do we have other questions?
Okay.
Why we're having another question?
People want to sit.
So please pack up.
What's your state use on other inference?
Like who explores and for example, speech to text?
Sorry.
What's the state use for other inference that you want to run?
For other, like other stuff than PDF chess?
No.
On one side, you expose that you want to be able to run stuff like speech to text to speech.
Right.
What's the state use right now?
Okay.
It's still experimenting.
We don't have yet.
So for text to speech, some people have some interest to run models like
Kokoro.
And we're going to enable it.
So if you build a web extension and you want to do text to speech,
it's going to be super simple.
For the other side,
speech to text.
This is something we're look starting to look at to revive what you started some years ago
and trying to build something that works well in the browser.
But yeah, it's coming this year.
Thank you.
More questions?
Thank you very much.
Thank you everyone.
Enjoy your photos.
Thank you.