Hi everyone, my name is Sun, and today I'm going to talk about a fun project that I do like
a weekend project that I stand on.
Start brings Lamar Cp into the web using web assembly.
So my talk will be divided into this point where I first introduce myself.
Why I do this quickly more and then show you how it works and some children's
that I placed on the web and my plan for the future.
So my name is Sun, I'm a software engineer, a talking size, I draw a talking size that I
find myself very new.
And I'm one of the Lama Cp, an active engineer.
Here's my shithap is NGXN, and my slogan is doing AI for fun, not for both of you.
I actually had this way before, I draw an actual hacking phase.
Okay, so some of my work on Lamar Cp is the first, like big things that I did for
project was the chat template things, and then I had a refactorring for the
support for low-rank adaptation.
I'm also one of the core mentioned on Lamar server.
That's now we are able to bring it into something called hardware inference endpoint.
I'm one of my very, very big tasks that I'm actively doing is to reflect the
monthly model, especially the vision part, and it's still going nowhere, but it's a big thing.
And one of the things I want really to do is to add the web shithap you back into
which is your main.
And if you go to shithap, here's me by the way.
Okay, so, but Sun, this is a serious thing.
Why don't we do like this fun thing?
So let me show you what do I mean by fun.
Yeah, so you all know this guy from like 20 years ago.
And now, what if I make it just a little bit smarter?
Yeah.
So hopefully this demo works.
So it's a creepy.
Yeah, nice.
So the fun thing is this one directly on browser.
The model is already downloaded into a local storage.
It's not really a local storage, but yeah, it's being cut in the browser.
And the inference is done using when assembly.
Yeah, so that's the fun thing.
And the high this demo is a project that I made in my free time.
It's gone web lemma.
Yeah.
So this machine has to get started.
It might be interesting.
Yeah, so why I create in the first place, so long story, so long time ago,
when I haven't joined, I haven't faced yet.
I was a GPU for like very poor, not just poor.
And then I also want to push lemma to delete me.
Actually, it's the limit of my hardware that I had at the time.
And so I was very inspired by whisper, CPP.
The web SMD version that you can run directly on browser.
And also it's just so fun.
Not for fun.
To make my voice, not to compete with like production ready for work out there.
Like over internet or web NLM.
Okay.
So what is your goal?
The first goal is firstly to create like a wrapper.
Cheers.
TypeScript libraries.
That you can do a web developer.
You can use this in your project.
Just like running just one command, NPM install.
It's strongly typed.
TypeScript.
And zero dependency.
It's amazing.
So let's show you what that means.
So this little demo here that you show here.
That you see here.
How I made it.
Under the hood is that I import this.
By the way, this reaction is project.
So I import it.
It's a library.
And then I download this model.
I defy here.
I just bought the model from HangiFest.
Then I create a chat completion.
I had a system message that say, hey, you know,
we have a clip here.
Okay.
So that is.
I also have another demo that you can see.
Right on the digital page is here.
And the demo is more functional.
You have like a list of models that you can try.
And then you can chat with it.
Yeah.
This is a demo.
So yeah.
So now the complicated path.
It's a technical, like, deeply technical path.
It might not be very interesting, but yeah, very unique.
Okay.
So that's one thing called EN script.
And it's the thing that allows user to compile with
or any CPP project into WebAssembly.
So that should be simple.
I just take EN script and then compile.
Right.
Turns out not that straightforward.
Then I had many challenges on the way.
But there's that form in challenge.
That I find.
And I wanted to share with you today.
Yeah.
So how it's work is.
Firstly, from perspective, WebAssembly, everything is nice.
Not a string or like a number.
So what I end up doing is the first play.
Is that I add a small wrapper, a team wrapper.
That access JSON from JS from JS work.
And then I translate it into API code, native API code,
CPP, I say API code to the Lama CPP library.
It's work pretty well.
But then I realize that it's a lot of work because I have to,
like, parse the JSON in the JS.
So it's a CPP work.
And it will be extremely slow.
For example, when I call the tokenizer tokenized function,
waste return, bunch of tokens, like thousands of tokens.
It starts to slow.
Slow down.
So I am now moving away from that.
Infliver binary protocol.
Which is inspired by my protocol.
But I invented.
I'm not copying it.
And I am inventing a new step here.
Yeah.
Next thing in my list is something called Defi system.
Yeah.
We don't know Defi for matches you have.
We know when we lose.
The problem is that in the early day.
OK.
So I just load this using the default file system.
Ian Squidon.
It works.
And I'm using too much memory.
Why?
I start to die deeper into the code.
And turns out the MMR function.
That LMSVP will use this function for reason.
Because we don't want to minimize the copy of Defi.
OK.
So when I look into the source code of Ian Squidon,
what is that?
It's exactly opposite of that.
It's first the allocating memory.
That is why.
So what is that here?
It's basically a copy.
So this copy is a chunks of the file.
So why is it not nice?
Because we end up not knows if you load as you.
If you have a 200 megabyte, you end up using 400.
Yeah.
So yeah.
This is how it's looked like.
I draw.
I've re-explained.
But I have just set it in the network.
So it's copied the file into the worker.
Where it's stored temporary into a buffer in JavaScript.
And each time I go and not, it's got to be the buffer back to the
HIP memory of WebAssembly runtime.
OK.
So what is the solution?
I invent my own thing is gone HIP address.
Which it used this stream API of the browser.
And instead of having to temporarily write the file into a buffer
inside the worker, inside JavaScript,
I stream each chunk directly to the HIP memory of the WebAssembly runtime.
So what now?
What about a map function?
So I ended up with patched function to return a pointer to directly to the location
of the file in the HIP memory.
So how it looks is this function.
You don't need to care about the first part.
Just need to care about the last two lines.
Where I return the pointer is the pointer to the file.
HIP pointer to the file plus the position in the file.
So I want to map.
So it's just return a pointer.
Not a copy.
OK.
Next thing is apply storage.
OK.
So we never want to like each time you run the file.
You run it.
You need to read out loads of how one shaker by model or something.
So in the first day, I used something called catch storage,
which is a very nice thing.
It's easy to use.
Then the storage is very limited.
And also it does not support stream.
It's time I read something.
It's in my copy.
First it is load.
If I into browser memory.
OK.
So I turn my attention to something called index dv.
Yes, it's better.
Not the problem is that it's actually, at least on 5th of,
it's actually stored when it's stored to this.
It's stored inside an SP light library.
It lasts not support stream.
Maybe it lasts, but it requires hacking.
The next thing is that it does not have like a hat,
a maximum capacity.
Yeah.
But still, it doesn't have stream.
So it does not benefit to me.
So at the end, I turn my attention to something called OPMS,
or it's in private file system.
So what is there is that it actually, when it's used,
a fun thing is that when you store the file into OPMS,
it actually stores the file as a real file on file system.
On your list.
And I reverse engineering it to know, I can show it to you.
Yeah.
That is a soft stream.
Yes, either an I will show you.
It's very cool.
Okay.
And it also does not have a maximum capacity.
Most browsers just cancel this in my capacity.
Most on the free spy on your list.
Okay.
So how it looks like now is that when I open the file,
the browser gives me a file object.
I think a bit like file is greater, not a copy of the file.
It's like a pointer to the file.
Yeah.
And then I can stream it back to the browser.
Sorry, to the worker to the where I think we won't have.
So there will copy.
Yeah.
So how that is looked like, maybe a little bit more in the,
just look at the function signature.
It returns a promise of the file on in the case.
For example, the file does not exist.
But the most important thing is that this returns this file.
Okay.
So what's a very cool file.
This is that now.
I cannot only just use the,
the file provided by OPS.
But I can also load my own file.
I can also load my own file and I can also load my own file.
Each browser, this button on the file, when you want to upload some file to
Internet.
If I see browser returned a file object with a stream.
Yeah.
Let's go.
But then it's too slow.
Even with.
This flag, this two flag.
The first flag is to enable the support for something,
single instruction manipulator, and the second flag is to activate the
lever trees, lever tree optimization of the compiler, which should allow
something complexurized. So what does it mean by
vectorized? That is it pick up on the loop and it try to use
the same thing on that. But the second flag failed to do that.
The solution is obvious to rely on this function in
Cindy. Sometimes you might already know what I'm going.
Yeah, especially. But then I only have one
thunder and my thunder does not just with the computer.
I also have a wife. Please.
Okay, so. Yeah. I just asked it. And that works just
kind of maybe. But at least two-ex speed, double the speed on
most of the quantization. And almost triple the speed on
the quantization to do it. Which is this?
It's a bit quantization. Okay.
Something I plan to do in the future. Firstly is the
wedge CPU. What is still very early development?
I still have a lot of trouble with this. But I have a very simple
POC pull-up on set for now. I have to revise the kernel
in something complex and please share the language.
I have a fixed-down issue on mobile. For example, most important
is about iOS device. Then I also want it to
compatible with something called WRC, which is where
assembly system interface. We should allow it to just
for example, read the file directly from the file
system retail. Here I'm integrating it to run
with our browser. Yeah. I will also want to
support the lower adapter, which should be
something very simple to do. Just I don't get
as a time. Yeah. It's a weekend for the
better way. It's still here. Okay. And last thing I
want is, it's not the last thing. It's just
last thing here is the list. But the support for
multi-modal is still waiting for my
refactoring on LAMACP. But yeah, I want it to be
very seriously. So, take care. I'm taking my
time on this. So, that is. Thank you for your
attention. Thank you. Thank you very much. We have
time for one question. If somebody has the
question and enough energy to ask it. Okay. Thank
you very much. Thank you. Thank you. Thank you.
Thank you. Thank you. Thank you.