Hello everyone, good evening. Our next speaker is Rex Ha, he is a research scientist at
Homeroom and he is going to talk and let us know more about building local AI with full
tech approach, give a warm round of applause.
Okay, hello, first of them, I'm Rex, a researcher at Homeroom. We are by Zincapar.
Today, I will take you on a journey through an anatomy of thinking machine, exploring how
local interact with us in both hardware and software levels.
A little bit about us, we are building complete environment for open source, local AI.
Our focus is on running AI locally from serving our MS on servers to laptops to even
edge device like mobile robots. We also train a few integrations of a real-time
multi-moder models. We have learnt a few products like
John, Chachibity, Affinity, that's the application that lets you run over any open source
model on your machines. If you prefer to host it yourself, we have Cortex, a local
AI platform. We also release a new model, a family of speech-language models that
can respond to you in a real-time. Additionally, we are also experimenting with robotics simulations
as well. We are basically everything for AI enthusiasts, small businesses and enterprise
need to run a local LM. This brings to the full control for you to easily modify
model knowledge, to adapt with your need and eliminate concern about data privacy.
After focusing on local inference, we wanted to create our own model.
When we look at the foundation of the model, it's too expensive. We can not
complete with mixaxe like meta-open AI or Google, so we think of a different way
that we utilize. How about you train your LM? I saw a few hands, but it's good.
How about you in the multi-model, LM? Oh, I see no. Oh, I see over there are some people.
Well, you guys are like really deep into the field. But then come to the hardware.
Like, have you built your own desktop with Chibiu? Oh, a lot.
And like, you build a Chibiu cluster? Oh,
actually, I haven't built one. Like, like, my other college built it for me to train
model. I know it really know about it. And last one, have you debug X100 cluster for inference
and FI-2 name? Wow, you're rich. You're rich.
Well, I have, I like, that one is like my pen pie of every night. I have to like be
in stuff. Yeah. Okay. So, great. Some of you have, like, I saw a lot. So, you have, like, knowledge,
technically one knowledge, but some dozen so I, I may have everyone to cast up.
So, I will do a quick recap on LM sense of with, like, really high level of restrictions.
So, yeah. Okay. So, what is LM? But attention, guys, researchers out there, please cover your eyes
because I will, like, do a really simple, seriously over simplification for this.
So, please cover your eyes. You already know it. Okay. First, we have a neural network
layers, just for my layers. You see, like, me making up the way you digitalize our version
of our brain. And at a perfect, not really, they're not perfect. They're not really perfect
of, like, me making our brain. But do they work? Yes, probably. But can they handle a lot
of data? Oh, hell, yeah. They, like, they have all of the internet of us. Right.
And then, like, there's an encoding layers, which is, like, a component that can help
to get the model to translate human language into a machine language. And then, we have, like,
the decoder layers, which is translate back the after calculations and stuff. It
will translate the machine language back to the human language. Yeah. But the main focus
of this talk is on how this AI can run on our machine. So, the key thing you will need to know
that these layers, just the stack of components, not let the machine can think. Okay.
So, another piece of knowledge that you may need to know is that what exactly inside of the
model. So, let's say, when you were, say, you were running a mixture of llama, what
does that really mean? Like, there are different versions of mixture. You can see, like, this
one is, like, 4 gigabytes of storage. And at one, it's, like, 24 gigabytes. So, what is the
value difference here? The answer lies in the models parameters. The more parameters
the means, like, the is larger and also is smarter. So, it comes with, like, we need a lot
of, we need a lot more hardware. Another factor is that quantization model is not linearly.
You can see, it's like a logarithmic row. So, scaling up the models and just sustain
approach. But let the smaller models, smarter, is the way. So, and I, and believe me, this graph
which was through in the last year. But this year, smaller models, like, we saw that
two when, seven, be llama, three, they are all really good. And we can do use it for the
basis. So, as expected, with a previous point, bigger models, require more memory. So,
if we want to serve, let's say, eight users in the row. The V-ramlet, it increase even more.
And if we want to run it with, like, flow 16 for the better performance, the GPU needed
is going to be insane, right? Yeah. And this is why, and we are making so much money this
way. And, yeah. And also, that's why, like, multiple companies right now, like Amazon,
Google, or even some startups. They are trying to develop their own hardware to, supposedly,
to influence the language model. So, to make it more practical, you want the most intelligent,
I, I, to serve you at home. So, how many GPUs do you need to run 80, 70 million
model on a consumer cart like, 30, 60 is, like, it cannot. But for, like, even, like, we
100, you need, like, 16, but for, 100, you need, like, a row for, but those are not, like,
simple machine. And you can see on the right here is a picture of my colleagues house
before we move everything into the data center. If you can see, like, when we run
ding a 70, be model, into our internal use, it could hit, like, his house up for a whole
winter. Yeah. And, like, if you, an Apple fan, you might think that you can
start getting max to run the models. Okay. You are stuck in to run the model. Yeah.
Actually, this is, like, you can see, they actually using it to run the model. But,
not just getting, please, no, don't, you do not run your house like that, like, please,
don't, please, don't do this at home. Okay. But, but this view is lush enough, that company
is, like, I said, before, like, start up congrock, they introduced new, new thing, new
hardware, cone LPU, which is the hardware specifically to design for the last
wish model inference. So, this makes the prediction progress insanely fast. So, all
of those just to support a beefy model, that's now even more talkative. Recently, we have
heard a lot about, like, open AI 01. Yesterday is, like, O3 and the tsunami of AI
advancement in the west cone, beefsick. They are own, like, the success, like, lay on
the technique called test time computes. So, which mean, essentially, is, like, when
you let the model model thing longer, or let it, like, talk longer. And, hopefully, it
becomes smarter. But, well, yeah, it's not a way the case. Okay. Okay. But, the
mentality away here is that, we, we all know that techniques, these techniques will
be more viral, and also it consume even more GPU time. So, this brings to another aspect.
Okay. So, GPUs now would, like, not only need more size, like, bigger size,
they also need to calculate things faster. So, you have probably heard of
such a flop on the news when every day, or modes there and GPU, right?
These measure how many floats by operations are up a machine can perform. So,
more importantly, we found that, when we testing on our, when we develop the app,
we found that the memory bandwidth is crucial for faster inference. Inside
of the GPU, a lot of data change flows happen between SRAM and HBM. So, increasing
of memory bandwidth boost data. And, yeah, we have another option, if we don't want to
choose NVIDIA GPU, you don't want NVIDIA to get rich. So, you can switch to
Snapdragon chips, which extend the AMD include NBUs. Okay. So, next up,
people want to move to the desktop workstation. And, the car is more
about, so, which is communicate with GPUs and all the components. Here,
you can put a multiple things into onto the motherboard and how it doesn't
break. These components are typically less and provide more computing
power. And, the next level, we scale up to servers. They make a better
components offering, we can run it 24 hours per day. And, each of the
unit is designed to be modular. So, we have multiple CPU and multiple GPUs
connected together. This could give you, okay. So, at the right side,
could give you a roughly ideal of what needs the server. This is a really old
architecture, up to this day. But, yeah, you can look at it as a reference.
So, when you have a lot of money, you don't want, you don't have to use it.
You can buy multiple server and connect it into a data center
racks. This will connect it through infinity and buildings. Okay. So, those are all
of the intuition I want to give about hardware. So, let's come to the main of
the talk. So, this is the talk is about like the Antonomy of machine
doing inference. Okay. So, I just want to say something. We have learned
over the past 20 months of starting the project. As of this week,
we have been a lot of it over 2,000,000 times. And, well, we have gotten at least
86,000 cross reports. And, this is only count because we enable the crossing
part over the past three months. So, over like, what are the half year, we have
a lot of users across like crazy out there. So, yeah, you can see here,
AI is still in this infancy. We are solely in the game. If you hear,
I'm talking about AI, I'm inferring things today. It's still really early.
And, I met that 30 years from now. AI will be much more seamlessly integrated
into our machine. And, your laptop will feel more human light than ever. So, yeah.
So, let's talk about how machine with single CPU inference requests.
When you ask a model to think, what happened in the hardware's field?
Okay. Let's pretend that I'm a machine. And, I want you to ask me,
what is Jan? Then, if I want to generate the tokens like Jan,
he says something something. The first thing first, I asked a CPU.
Let's make the function code to retrieve the model weight from this,
which is like HDD or RSD. I have to pull another model.
And, let's say, it's like foggy by a storage size of the model.
So, I have pulled it into the CPU.
Yeah. On the other side, I want my simple step diagram.
So, I draw it to simplify everything to for people to understand,
people that are inside of those wire and chips.
Okay. Then, after all of the model weight load from this,
let's assume that we will have a latch in our V-Dram GPU.
So, I want every single layer of the model onto the GPU.
And then, when, and then now it's like, your problem right now is coming through over.
Okay. At first, the prompt will be like tokenize our simple way to translate into the machine language.
And then, it will execute through transformer layers.
Usually, it's in parallel in the GPU and then,
generate a possible predictions.
And the final prediction, the next squad, is met by the CPU,
which, as a look-up table to find the best,
then the most suitable word for the given context.
A response is then returned to the user.
So, well, this is a journey of how a simple question from a user to the machine
through everything, it thinks, and it will respond.
For the next token, it will continue the process again and again,
until it reached the end of the answer.
Well, that was an easy one, right?
But what if you have like a multiple GPUs and one to use a bigger model,
which doesn't fit into the single GPU, right?
Same first step, we also need the model to load into the happiest.
From the happiest into the CPU.
And the CPU wishes for the model weight into the GPUs.
And here is the interesting part.
The model will fit up, and yes, and yes,
it will distribute the layers into GPUs.
So, like the CPU right now is like GPU 1.
Do you got this one?
Do you got this layers, GPU 2?
You got this layer.
Depends on the algorithm, do you want to use these
are like the two main common algorithm?
Which we have like pipeline parallelism, which is like we split the model horizontally
between GPU 1 and GPU 2.
So, it's your like, it's your own half like 16 of layers,
and other one have 16.
And the other one is like TensorFlow parallelism,
which split the model vertically.
So, layer will be distributed in GPU 1 and 2 accordingly.
So, everything works nearly the same as in the single GPU right.
So, the model is now nearly the same.
The model is now encoded into the machine language with some layers
running through the GPU 1 and some like,
when we run through GPU 2.
And after processing the results are combined.
And the final prediction is met.
It's like nearly 100%.
Like, it's 90% of similar to the,
we got a one single GPU.
And finally, when we stack up the GPU stacks together,
we become a multi-note.
Okay.
So, if you have a multi-note scaling of HPC data center.
So, okay.
To be honest, like, as I said before,
I have not, like, fully understanding about what this is.
Because, like, my hackware colleague will do all of this.
But the main thing we want to check here is that all of the other companies
out there, like, open AI, like Google.
It holds, they hold,
charge it with the audio menu on, like,
for our users on this kind of setting.
So, let's come with me on this.
So, let's start, like, at our former beginning,
we always have to load now.
I will talk about more advanced techniques that,
right now, the companies out there,
we've used to serve their models.
First thing first, we have the KVCatch.
Since the code is usually autoregressive,
which means each of the newly predicted tokens
only depends on the previous token.
So, at each generation step,
we are recanculating every attention for the previous tokens.
Which is not the most efficient way, right?
If we calculate it one,
we can cache all of those,
and then just use those to calculate the next token.
That's a really high level of what is KVCatch working.
This is the common text that anthropic open AI
and also Google used to serve the user,
because this technique can reduce, like,
90% of the inference time,
if we end just easy way to enable this one.
Another technique is that speculative decoding,
where we use a last main model with,
like, the best performance we can have,
alongside with a smaller model.
The reason for a smaller model,
a small model, right here,
is that it can handle simple, repetitive tasks,
and quickly generate token inspiration,
and it could significantly reduce the inference time.
Okay, so this is the comparison from open AI,
the speculating decoding.
The right size, you can say, is really fast.
But, like, the left side,
if you only use the big model,
it takes ages to just generate the answer.
Okay, it's finished, thank you.
Okay, so this is, you can see,
like, given Android Compatty,
he's, like, a top lid of,
X, the X, open AI lid.
So, he took about a beauty of this technique.
Okay, so, after, like,
looking down all of the steps,
like, how the normal text base,
generate if AI work within our machine.
Now, we, if we want to add one more step,
like a sci-fi movies,
where we can command the hour machine,
then it will understand with our voice,
and it will respond back.
What would it look like?
So, without in my, we started project isigo.
Oh, I don't think the audio go through.
Uh, can we somehow get the audio hidden?
A little bit of technical help.
Maybe I will just, like,
hey, this is itigo.
The local real-time voice AI,
it can understand human speech and talk back.
And with the latest checkpoint,
you can run this little guy on your device,
and it's fully open soft.
But your scene here is the demo version of it,
and it's running on just one 3090.
So, let's give it a try.
Hey, it'sigo, what's the meaning of strawberry in Japanese?
Strawberry in Japanese is called itigo.
It's a sweet and juiciness that's very popular in Japan.
Here we go.
Okay, so all of this is just like running on a single 3090 machine,
and it reads, respond in real-time.
And like, this is the only, like, a project of three months or less,
is actually thinking around and trying new stuff.
Okay, I will go quickly, go through the intuition of how it works
to speed up the top-congeneration.
So, that's why we can see this general thing is really fast.
And this is an interesting fact,
because like, we train our isigo entirely on text data,
not a single audio file.
Yes, you, you hold it right.
We're training a speech understanding model
without any audio files.
How, how it goes.
It actually, like, we, we did it with the simple model,
like, transform model, transform model,
which ended with, like, the input essay, text English,
Germany, our French.
We input it both, all of them into the model,
and let it predict the speech tokens.
Yeah.
It's just like the beauty of the transform model,
because it's working as a translator between human language
to the speech language.
So, with the spirit of open source,
we met everything available,
even our training code, the data set, the model ways,
and even our discussions,
even some fail attempts and papers.
And so, if you want to see us fail in real time,
you can, like, look into our discussion,
and watch the live stream.
I will be there, and, like,
sometimes I will say something bad over there.
Okay, so, okay, the last thing I want to discuss today
is, like, how to bring the machine into a real world.
We have talked about the live cycle,
AI within a machine, but what if we could put the AI
to interact with a real real environment?
So, before it can do that,
we need to train it on a simulated environment.
So, unlike human baby,
we would take, like, 10 or 12 months to start working.
We can simulate thousands,
or even 100,000 robots in parallel,
to help them to learn the tasks,
like, working in multiple environments.
Okay, so here's not the machine fan
from head up, and video,
but the department said that the simulation
has to write the robots learning curve
by 10,000 times compared to real life.
Yeah, it's insane, right?
So, this is, hopefully,
in new guys can hear it.
This is a really, like, initial attempt
to control a robot using an easy-go model.
So, this is a voice,
we will put it into the model.
The place is not going to.
So, let me just play the video.
So, right here, we used our voice
to talk to the robots.
We said that it's a left 40 degree.
And then we go straight ahead.
I don't know if you guys can hear the voice from,
but it's like, to left 40 degrees,
and then we'll go straight ahead.
So, in the model,
the robot will execute the action.
So, the behind the scene of all of those is that,
the easy-go is just just like,
it will use the voice,
convert it into the JSON format right here.
And now, which is the represent of,
it's to this represent of the action.
So, every that I then stream into
the robot in simulated environment
and the same process will apply
when it moved to the real world later.
So, what's next?
After the simulation comes with the physical,
the seem to real is the term used to just like,
just for everything the robot has learned
from simulated virtual, real world.
And you can see here, like,
at home real,
we just have just bought a real robot to test it out.
So, everything is still ahead.
And we update our process,
publicly even if I fail,
so, if you guys can see it through our discord.
And yeah, that's it.
Another talk.
Hopefully, it gives you a glimpse of
the current state of AI right now.
Thank you for listening.
Thank you.