Yeah, you just turn it on.
There you go.
All right, everyone.
This is Martin Chang.
He will be speaking about, oh, you asked your mic,
building new GGL GML backends for novel accelerators,
how challenge and opportunities you're ready to go?
All right, take it away, Martin.
All right, I'm sorry for the crazy setup,
but hopefully everything is clear right now.
So let's talk building about new GGL GML backends.
A lot of the hardware companies are starting to come up with
the new new and different hardware,
that supposedly to be vastly vastly faster than GPUs.
Yeah, but most of them have come with their own
programming difficulties and mismatches.
So for example, the hardware may not support a certain feature
that is very common on GPUs or CPUs,
and you have to just have to somehow deal with that.
Yeah, this talk is about the how to integrate
the differences into GGL GML, the challenges
or the program mismatches that you expect that you have to see.
And the opportunities or things that we still have to do.
There's a lot to go through in 20 minutes,
so I'll try my best, but no guarantees that I will be able to do everything.
But first of all, some disclosure,
I'm currently technically sponsored by TensorFlow,
but this work is going to happen with all of your support.
I'm very grateful for their support,
and the engineering is very, very helpful.
And just about two minutes ago,
they just told me if anyone is interested in it
if they're hardware, find them outside the door,
and they will be able to help you gain access to their hardware.
Very good to open source policy.
First, I'm background on my self.
Who am I?
I do sell a lot of C++ and HPC.
I'm a false developer, obviously.
And this is my original fun.
This is a lot of stuff I do besides doing AI,
I maintain the web frameworks,
I maintain the niche search engine for an
niche in her app protocol,
and I also develop some libraries for security.
Beyond that, let's get back to the original topic,
GGML.
Hopefully everyone knows what it is.
It's the back end of LaMASIVP,
which is what a lot of people use right now.
It's very efficient for instance,
and especially for large-enrich models,
it has very strong quantization support.
It has a very good community and very flexible.
And most importantly for us,
it's written in C or C++,
which makes developing very, very easy
to integrate that into a new hardware,
and then you input new capabilities of the hardware.
Could we go through my journey?
I started about 2022,
where there's a new chip called the Arcus V8.
There's an AI core process on there.
I want to really want to locally exist
and with L390 running the summer in Taiwan,
which could be at 40 degrees Celsius,
which is, I don't need another space fee for me.
Either way, that's sort of work.
I'm able to integrate the core process support
into LaMASIVP,
but due to the architectures,
doesn't really work out.
So I tried to side to pivot,
and at the time,
I started to sell their first dev kits.
I got this thinking,
what's the worst could happen,
and that's how it started on this project.
So the last talk,
the people talk about P,
P talking about TensorFlow,
and in D,
how the calorie hardware works.
But in C++,
other people,
this is the very just version of it.
It's a many core processor,
so it has a lot of results
organized into different grids.
These risk fights are connected to different core processors,
so you can do computation
and TensorFlow operations really efficiently.
They call these risk-like cores baby,
because they have really, really small,
and by small,
I mean, like,
undergrad has textbook level of small.
Yeah, it's a grid of cores,
it's a network on ship,
so you can talk to cores,
can talk to other cores.
There's no global cache,
we're going to find memory,
so each core has access to its own memory,
and you can ask a DRM to send something to it,
or send it somewhere else.
There's no cache.
It's listed to help with management,
which will come into you a little bit of a factor on.
And yeah,
and it also scales to a very large array
by just taking more ships,
gather and communicating with each other,
using one of the peripherals.
I stole a slide from there,
caught ship's slide stack for the
an activation processor.
As you can see,
it's a grid of CPUs,
and different CPUs were different purposes.
There's different core types.
There's DRAM,
there's Ethernet,
and they call it N6,
or just the compute.
Compute just do,
just do compute,
does the AI method we actually care about.
The DRM cores is interesting.
The cores can,
the compute core can either just request the MA,
and ask with DRAM to send data from the DRM into,
into it's local address,
and do compute data,
or DRM cores can actively push data from DRM to it,
into other cores,
so when they needed it,
it's there.
There's also the Ethernet core,
which does all the communication,
and that's how it scales beyond one ship,
natively and don't need some crazy peripheral
for that kind of work.
Inside the compute core,
there's really five cores that's working together.
There's two respects that's connected to the knock router.
These knock routers are able to talk to the other cores,
and most importantly DRAM,
so it can access data,
and only these two core can programmatically.
For the compute,
there's three cores that's working together
to perform computation.
A compute side is really a tensor engine
and a background working together.
Unlike a traditional multi-core,
multi-core, multi-process,
architecture, each,
this five cores there are working together
to control the the peripherals
instead of being the main system that's doing the computation.
Because of this architecture,
everything has to be explicitly management managed,
including the data flow.
So instead of just the compute core saying,
hey, I want the data from this DRAM address,
and be able to fetch it.
What has to happen is one of the cores connected
to the knock has to request memory from the DRAM.
If the DRAM has to send the data to its local memory,
these compute cores can then take that data,
that data to the computation,
and put it back into its local memory.
And then the other DRAM,
sorry, knock connected core can then take that data,
and output that to the memory.
That's how it's supposed to work.
And the Ethernet cores are able to do to scale up
to communicate with other chips,
and be scaling out the architecture.
This is the current SDK stack.
There's on the bottom there's Metallium,
which is a C++ OpenCL-like API,
so you can directly program on the device,
on how they build TTNN,
which is their tensor and operator library.
It's also a tensor library on like CUDNN,
which they'll provide,
because they need something called
Tile, which we'll touch on later,
but it's for hardware efficiency.
On top,
they have the MLR stack,
and they're on the fork.
All these software here are all open source,
so it's very hackable.
In my development progress,
I also contributed a lot of features back
into these software stack.
And of course, this plug is about the LOMOS EVP
and GGML support, so that's that.
And that's the current community software.
Yeah, let's take a look at what the SLR waiters is expecting.
So unlike GPUs,
where you can just directly program them to do a ray
or something, something operation,
these devices usually gives you some sort of higher level access.
So, for example, on this code,
like we just open one device,
the created tensor,
we Tile-like it.
This is special operation,
just to make it hardware more efficient.
We can do level one cache,
which does not exist on their hardware.
Well, the console sensors here,
it's really high bandwidth,
because it's directly on chip,
but it's small, it's only 100 megabytes compared
to their 12 gigabytes of DRAM.
Okay, that's just a couple of processor,
and we can start building the GGML back end.
So this is the general flow.
You have to register your back end,
scan or support it off,
so it's how GGML,
what you can support what you don't.
You can accept,
you have accept sensors from GGML,
upload that to device,
execute all the operations,
and GGML can,
sorry,
after you do all the math,
send a result back to GGML.
But here's the problem
that developers have to have to somehow fix.
GGML expected real major tensors,
but GGML has to use Tile tensors
for hardware efficiency.
But TileL doesn't support device memory mapping,
while GGML,
because GPU works,
they expect the CPU can directly access the device memory.
And GGML also does byte allocation,
which doesn't really work on tensoring device,
because allocations are tight.
So what do I mean by that?
That means when I create a 4 gigabyte buffer,
for example,
I have to tell the device that it is holding a 30-degree bit
floating point,
or a 16-degree floating point.
These information are missing
from GGML,
and has to be somewhat there.
Has it be dealt with?
There's also different quantization types,
which is different to GGML,
and somehow.
Yep.
So first of all,
we have to register device with,
sorry,
what's register document with GGML?
So GGML has something called
the GGML background registry,
which is just a big list of
different backgrounds.
You can see,
there's the good up-back end,
metal,
sickle,
walk-in,
etc.
This is a very big list with a lot of
it depth.
So what do you have to do just in
it add your own background here,
and register it with GGML?
Afterwards,
it calls your device registration function.
Here you have to return a back-end handle,
a back-end registry.
Each registry has a registry interface.
This interface is a giant big retable
that has some,
some required and some optional functions
that you have to implement yourself.
And also a context that tells you
back-end,
what itself working on otherwise the back-end
doesn't know what kind of data it has,
or what device this it has.
So it's looking at all the interface.
So the interface, like I said, is a giant B table.
The GGML uses this pattern
internally very, very much.
And some of these interfaces functions
can return more B tables or more interfaces.
So for example,
this one,
the get device will return a device interface.
There's a whole lot of interface
like this in an GGML for device,
memory pools,
buffer management,
et cetera, et cetera, et cetera.
It's a long list,
and I don't have time to get into you.
So you have to populate all interfaces,
and eventually you have to figure out
most of the interface and stop
those start working.
And after you get that working,
you have the instrument,
it tends to have poor management,
but it gets general problems
so you won't crash.
And all the operators
has to really data from
the data to know the actual
instrument set up using the map address.
Right.
Okay, you can allocate
tensors.
You have to get data in and out.
Here's the problem.
Not the old tensoring process
or support old GGML types.
Just take the quantized data.
It's the format is different.
And the tensoring process
are there.
The respect course are very small,
very slow.
They're designed to actually compute
more controllers.
So you cannot expect the CPU course
there to do all the quantization
like we do on the GPU.
So instead, since we're ready
to further the actual location,
what we had to do is
the quantized into F-330 to F-16.
Upload that to a device,
tileize it,
and then cast manually
to whatever type of tensoring
hardware supports on
the device and because
it's the same thing for
go for get tensors,
which is the reverse operation.
And then what you do is
after you're able to do
a tensor upload in download,
everything you do is
tile GGML,
what operating you actually support.
This is a quick,
this is a toy version.
Just tile GGML,
I support a
32 bit floating point tensors.
The operator non is a special case
that tells GGML,
this tensors holds
actual data instead of this
is in operation.
And in this call,
we also tell GGML,
hey, we support add operation
if the dimensions one.
This then GGML will do
operations scheduling
and figure out who runs what.
And then your response
which GGML will give you an array
of data,
of nodes to run and just run it
in iteratively one by one.
One problem you have to deal with
is views.
Views are operations that
create sub,
reads from the data tensor,
but that doesn't really
doesn't do really
computation.
So things like resize,
views and transpose.
Which views
heavily depend on the fact
that tenser data are stored in
raw material out,
which is not the case again.
So initially, the solution was to
just legal and evaluated.
We see a view we executed.
The problem with this approach is
GGML would attempt to write data
back into into our,
into tenser buffers,
which if you agree,
legal and evaluated,
so the realized you can
want to take a back to original tensor.
So the real solution is
to have to do a lazy
evaluation.
GGML is writing
you have to do the right
correct thing to write into original
buffers that of,
run into a coffee, which doesn't work.
So that's,
that's a just of getting back
and working with GGML.
In the development process,
we figured out that there's a lot of
missing features that the
GGML that is very helpful for
A6,
but it's not necessary as helpful for,
for GPUs.
First thing is,
graphery writes.
We,
GGML,
doesn't rehab it.
So like I said before,
there are some
concerns devices,
которые matter of course.
Thank you.
The other speakers.
I'm going to kick
margin out for over there.
So if you want to find the
afterwards,
any questions there.
the next.
Thank you,
we'll be talking
You