Thank you.
All right.
So, hi everyone.
Hi everyone and welcome to first
the 2010-06.
My name is Sun and I'm from Huyen Falls.
And today I'm going to talk about my
subject, which is 90-moderate support in
La La.
So, my plan for this talk will be
I will quickly have a self-introduction
and then click both onto the
history of a 90-moderate support.
And then I'll talk about my work,
some future direction.
And lastly, there will be some this for you
if you want to contribute to the project.
And I will try to resolve
like maybe five minutes for you
to have some question.
So, I hope we will have time for
question.
So, let's start.
My name is Sun Sun and I'm
I'm a software engineer at Huyen Falls.
I'm one of the core maintainer
of La Mado CDP.
And you can see my work side,
my sheet hard profile here.
My slogan is that I'm doing a
for fun, not for both of it.
And my fun, I mean the science,
the high machine learning and AI.
And if you are curious about my work,
you can visit my sheet hard profile
I do a bunch of work on La Mado CDP.
So, let's move on to this subject.
So, what gross is that
of multi-moderate support in La Mado CDP?
So, from the very beginning of the project
in 2020, we have initial support
for a model school lover.
And then, we have also had
the support in La Mado CDP.
It was nice.
And then, what happened is,
unfortunately,
we actually should remove it
because it's just too ugly.
So, in the meantime,
there was also new model,
because back then,
the multi-moderate and vision,
especially with the model,
was quite a hot thing,
made me for NOE was a thing.
And then,
so,
the US user experience
was not very good,
because for each new model,
you had to use
its own
called common-line device.
So, in my last year,
I tried to work on this,
I tried to,
as a same time,
improve the user experience
and deliver the experience
by introducing a new thing.
I call this belief empty empty.
And that way,
I was able to bring back
multi-moderate support
to La Mado CDP and La Mado CDP.
So, before my work on
new empty empty,
what is localized
is that we had
kind of this clear
big things in inside La Mado CDP.
We have the new La Mado CDP,
which is the main library.
And then, we had
Clip that CDP,
which is an implementation
of vision transformer.
And then, root La Mado CDP,
it was something specifically
for the La Mado model.
And then, people,
want to enter
a new model,
they tried to hack this library
to add their new model inside.
So, it's not very nice,
it was very costly
to maintain that much
infrastructure.
So, let's go to the empty empty empty.
What is empty empty is
not a model?
So, initially, I propose this
as LITLAVA2.
Then, I realize
I cannot use a model
that is two models,
specific names.
So, let's get rid of that
ring already.
So, the idea is that
now, I will abstract everything
and, like,
and that's where
it is, I want
one LITLABORIC.
So, the core architecture is
that used to have
a clip of CDP, which is
with which content
on the vision model.
So, vision transformer.
And then, on the,
let's say,
pre-processing stop is
doing by
another sub-moder.
So, it's almost,
I will say,
it's almost transparent
to the end user
and the end developer.
So, put more
than just
encapsulates the core library.
So, it's not
that CDP.
But, I also end
to bring
truly multi-moder
support, because we
start with vision.
And now, we have
audio, and then also
video, which is just
image and audio
other things, other things
and by the way.
And so, along the way,
I also provide
sub-moder for
audio input
or audio pre-processing.
And I also end
it to be
very extensible,
because it's now
being more
more
optimistic.
So, here,
here is
an example of
one of the
IP icon
that I add to the project.
So, it's going
empty and be
tokenized.
And the way it's
one is that you can see
I had
an input text.
So, inside
input text,
you can specify
this is
the media, and then you
can head text.
And while
this is
marker.
And then,
we can
knit
knots.
So, you can enter
a bit
map, which is
an image on
audio, or maybe
something else in the future.
And it will be
replaced to the
exact
marker.
So, let's see
a little
little
more
here.
So, I'm not going
to
the
developer
and I will only show you
the
view
the experience here.
So, let's start
with
Lamassianai.
And I'm going to
try
model is
congenitary
four billion
per
mirror.
So, right now
I'm using
the
CLI, the
common light
in the face, and the
high
common light
is the
Z-Lama server.
So, when I
touch an image,
let's
try this image.
Then I ask
what it is.
What is this?
What is that?
Yes,
we don't
speak
more paper.
This one, by the way.
Yeah.
All right.
So, how did I go back?
Yes.
So, that's a
little
quick
the
on the Lama
CDP.
What do you
have?
What in the face of that?
So, let's talk about
future
directions
I plan.
So, so far,
I have
planned to support
these two
big things.
Let's
dive into the
the first thing.
So, not
a model
output.
So, recently, we have
a lot of
image,
generation model,
like
something
image, I don't remember
yet.
But,
the main idea
how is what
is that you have to
meet Lama,
which is the main
library.
That will produce
some
embedding output,
and then we have to
decode that
embedding output
into your
audio or image.
So, the
overall idea looks like this.
It's a little bit complicated
why
because some model
right now, some model
generation
effects and
multimodal
interlip.
So, that's
relatively
complicated
because you can
imagine that
the model actually
firstly
generates some effects
saying that
here's the image
that I want you
generated for you.
Or maybe
it
do some
reasoning steps
and then it makes
some
kind of tokens like
touch generation token.
And then when
I get this token,
I had to switch
into using
the lead empty empty
to generate the
image, for example.
And then
at some point
there will be
another token
that says
stop generation
and I had to switch
like.
So, that is one
of the complicated part
on this
system.
The
second complicated part
is the
actual
implementation
under the hood.
So, for audio
decoder,
there is
multiple choice here.
We can use
a transformer based
which is
a little bit higher
or we can use
a division
by a
which is much higher.
And for vision
decoder, so far
I only know about
diffusion based model
and we also have
something
on diffusion
dot CPP
that is
unfortunately
it's not that easy
to interact
in lambda CPP.
It's not
like
I can just
copy
the code
there's
not how it works
unfortunately.
So, yes
the
initial generation
will be a long way
to go
I think.
Another thing
is
so
I also plan to
have video
in code.
So, video
is just
an audio
like
an animation
audio.
So,
see,
the way
she think about
is that
I can just
extract
bunch of image
from an
audio file.
But what happened
if you had an
audio
so a video
plays
like
one hour long
it would be
a lot of memory.
So,
I'm
inventing
quote and quote
inventing
a streaming
API
in
something
not just a single frame
I want to
process
some frame
other time
it's
matching stuff.
Another way
some of the
new
support
like
a single frame
so I'm inventing this
guy
in
infrastructure
and another
question is
so
which guy
I think
the one she used to
decode
or
bad
because video
code
so
is
kind of
the
long story
but
I think
no
going to be happy
if i
throks
the link here to you to learn more about my plans for the future.
Okay, so now let's talk about some more important things for you to interact with the easy project.
So yes, we are looking for a contributor, mentioned a developer for NTND, and not just NTND,
Lama CP as a whole project.
So if you want to involve in, I have three tips to that you to give to you.
So the first tip is whenever you want to contribute to something new,
that is why interesting to do.
Let's first look around the code by and let's try to reuse what was already there.
So, for example, for most of the multi-models,
most of the vision model or even some audio model,
you have a function is going to be with and VIPs in vision and vision transformers.
So I already abstract most of the vision transformers into one simple function that you can use.
So a lot of model is just like, just be with and this old,
yeah, you see that model, like, quite not something that you just vision transformers inside.
And if you have some, if you need to add something,
there is not just about miles, that you think is going to be a big thing.
So try to make sure it's sorry.
So try to first maybe open a discussion, an issue to discuss with us.
Second tip is, at least for NTND, and maybe I think for positive code by Lama CP.
You can use AI, yeah, of course, to discover a thing.
But I don't think at least for NTND, the code bay is major enough for AI to work out,
because I already saw some here that used AI under the hood to work out the code.
And even some one PR in specific, they tried to re-implement the matrix multiplication inside,
which is not nice at the same time.
Because, yeah, we have a lot of matrix multiplication inside Lama,
not CP, NGM, and for some reasons, there is still need to reinvent their own
method of matrix multiplication.
So to go back to this point, you can, you should use AI to go back to my last,
my first point is to look as a code bay to NTC, what's already there.
And you can also discuss back and forth with the AI, you know it.
What you're going to add could be adding a lot of code or not.
And if we want to be alive with the directional project,
which point should be brought up for discussion.
So AI is very good for that thing.
And last but not least, let's give a simple answer to bits.
So by simple, I mean, the code bay is quite young.
So I'm not trying to add exact support for all of these, all of the model.
I know some of the model, it has somewhat working thing, not exactly the same implementation as,
at an expected level.
Yes, the code bay is young.
So I just want you to have simple things first.
So let's try and, whenever you push a change, let's try to not be,
do you start to use specific or, yeah, to be to model specific.
And let's try not to break our model.
Of course, that's why we don't want you to do.
But yeah, by the way, when you do the AI code,
they tend to break our model, which is not quite nice.
Yeah, so I think that's own for my talk.
And thank you for your listening and this question time for everyone.
APPLAUSE
We've got time for one question.
OK.
I don't see any hands.
Oh, sorry, yeah.
How long, how long until the Lama CPB will have the ability to process the output of the model?
To work on exactly the level of work.
Yeah, so I'll just repeat the job question here,
because you don't have to microwave it.
Can you go there on the table?
OK.
So the job question was how long that is tight in terms of time,
until that we have the first working like,
somewhat working version of multiple of the output.
I don't have a calm lie. I would say maybe I'm planning it for this.
I'm having a demo for tech to fish now right now.
So I think in one or two months, we're going to get it running.
But for emission generation, it's going to take more time.
I don't know.
Maybe another question we have time.
Yes, sorry.
I just wanted to ask you how to test the computer.
Yeah, so the question is how to configure Lama CPB to use multiple users
on only one GPU or one server.
It actually depends, I think, for now.
The best way for you to do is to try to think or ring with the number of parallel requests,
and not end up being requests.
And as a same time, try to balance it with the context layer.
So that's what all I can tell you.
Do you have no time for that?
Yeah.
Oh, sorry.
Yeah.
So the FAP and BAC, it's very famous.
The FAP and BAC library, why I choose this?
Just because it's so famous.
It's nice.
It's top of a list.
Of course, there are some autoconsiduration.
Yeah.
Do you consider to be optional dependency?
Sorry.
Yeah, optional.
So I haven't considered it yet.
I don't have a clear plan on how to do it.
But I try to make it optional.
No more questions, and just people, because man of that will get recorded.
So please do not stand up and walk away.
And until the speaker is done, otherwise the video would be like horrible.
So nobody stands up and leaves until the speaker is done.
There would be like a three minute, five minute break in between the speakers.
When that is allowed, not why all the questions.
Yeah.
Do I have time for one more question here?
Yeah.
So the guy here, I think he's his easy.
So my understanding is that right now, all you're in
support in Lama CBP is already good enough.
So my understanding is that right now, all you're in support in Lama CBP is working and good enough.
What would be the most powerful model right now to get a well effect out of trying all you're in support?
Yeah.
So we have support from Miss Tang.
That's a Miss Tang model.
That's a bottom audio input.
This model is quite big, I think.
There's another model, it's going to be quite a model, quite an audio, which is also very good.
So you can try it.
And by the way, there's quite an army that allows you to input both types and audio as a same path.
Yeah.
Thank you.