And without further ado, Rubin will be talking about expanding GGML hardware for supporting
the Vulcan API.
Take it away, Rubin.
Thank you.
All right.
So basically, we've already heard quite a bit about GGML and I'm here to give you
a different perspective from a developer point of view.
Yeah.
Yes.
As the target says, I contributed the Vulcan backend to the Lama CPU and the GGML project.
And yeah, that was basically my first, my first major open source contribution.
So I quickly wrote the structure, just a little bit about myself and about the project,
or the two projects, a few problems I ran into, I solved them.
And in the end, a little bit about the community itself.
So just very briefly, my name is Humadla and I'm 27 years old from Germany.
I've got a mass science and computer science from the University of Mactaburg.
And ever since I was a software engineer with a focused on C++ and Python.
So the Lama CPU project, I'm not going to say too much about that, we've already seen quite
a bit about it.
But it was started by the already gag enough to run Lama models on the CPU.
And I think you was focused on MacBooks, or aiming at that because they don't have the,
they don't have support for Python.
It should be what previously mostly used for running large language miles.
And it was, yeah, at first it was just for CPU influence.
And the Lama CPU project itself acts as a kind of playground for the GGML development.
Like all of the earlier, all of the experimental stuff happens in the Lama CPU project.
And then that's part of the GGML project.
Synchronize with the GGML project, which is the more basic tensor library for running machine learning,
or machine learning library for running models.
So yeah, that's basically a C++ space.
It's optimized for CPUs, and at this point it also has a lot of support for running other accelerators,
mostly GPUs, but as we've already heard there's work on other hardware as well.
So basically I started, I joined the project at some point, I think, like early 2023.
It was very, very early all of the back end stuff that you heard about didn't exist yet.
And yeah, and we, in the early days, we had a big problem with it.
Like when you, when you run something using PyTorch, as you did previously,
then you had to put the entire model onto the GPU.
Otherwise, as soon as you put a little bit of PyTorch caught onto the CPU, it would slow down terribly.
Lama CPU solved that, but it came with its own problems.
So in this case, it ran on the CPU, it ran very quickly.
But there are two parts to running a large language model to stages, basically, to executing,
to get running a prompt, and that is parsing the prompt, which is called prompt processing.
And in text generation, you can think of this reading and writing, basically.
And the writing part was very quick, even from beginning on the CPU,
because it basically doesn't need that much computer just runs as quickly as it can read the model from RAM.
But the prompt processing part on CPU was very slow in the beginning.
And it's still, if you just run on CPU, it's still not very fast.
So, basically, as someone with at least some development experience,
and some, and the master science, basically, and some experience from university,
and that kind of thing, I felt that I could probably solve in some way,
and the ideas to go back to the thing we just left behind GPUs,
and use that to speed up the slowest operations, which are matrix modifications,
and that's basically what's, or that's the main thing that CPUs slow it,
and that slows down the reading part, the prompt processing.
So, there's a number of APIs you could use for that, like the biggest ones, of course,
the biggest ones, of course, CUDA, by Nvidia.
In the years, an alternative called Rock M, into this kind of,
that's all I'm thinking with the one API, which we've also already seen here.
And I think in the FPGA talk, because it's a few more things,
and then there's a few open variants.
That are OpenCIO sickle, which is the lower-neuer and Vulcan.
So, basically, because I want something that runs on any kind of hardware,
I decided to use Vulcan in the end, but at first I went for OpenCIO,
because it was much easier to implement.
We've also already heard about blast libraries, which are, like,
linear algebra, which can accelerate, or they provide waste,
accelerational duplications.
In this case, for OpenCIO, there was already such a library,
CUDA blast.
And so, in, I think, in March, 2020,
I just replaced the CPU-based blast acceleration that already existed,
and I must say, PPU with a GPU-based CUDA blast implementation.
I think it was, by, like, a week, maybe the first,
the first one to publish that.
But it's also very, yeah, this was simply implement,
but it's very far away from what we now call GPU-back,
and, basically, it just moved the matrix multiplication part
from the CPU to the GPU.
So, you copy the data from the CPU to the GPU memory.
You execute the matrix multiplication on the GPU,
and then you move the data, the result data back to the CPU.
And, of course, you spend a lot of time, a lot of time transferring data with that.
So, the solution to that is, it's a transferring data we have
the data already in VRAM, and we run more of the model on the GPU,
and that's basically what we started working towards.
But at that point, I found out that OpenCIO,
while it's a nice idea, it's a little limited,
like the driver supporters in great, in many cases,
there are specific extensions, which are not supported by some of the vendors,
like, if you use Nvidia GPUs, and you cannot use 16-bit floats,
which are very important for machine learning calculations,
and so on the end, I decided to, basically,
abandon OpenCIO, and try again with something else.
So, I went for Vulkan.
If you've already looked into graphics programming,
and you've already ran to Vulkan, it's a graphics API,
but modern games, modern graphics applications,
also rely a lot on compute shaders, and that's basically general purpose compute.
So, you can use that to run machine learning stuff as well.
It's the support is actually much better than OpenCIO,
and that's because of the gaming part.
So, every vendor wants people to be able to play games on their hardware,
and ship you vendor, and so they support Vulkan.
And some of it's very complex, because it operates very close to hardware,
but much of it can be avoided if you just don't,
since I don't need the graphics part, I don't need to look into
the software chains and image part, image stuff,
and that kind of thing, I can just avoid all of it.
I just need to be able to compute to run computers.
So, yeah, one of the advantages that Vulkan actually has over
Oculus, that they binary set result from it, are relatively small,
like Oculus, Oculus, you always run,
you always end up with a lot of device code,
and this ends up getting, if you support motor devices,
it's more device code, and that kind of thing,
that's a bit very quickly.
So, you end up with binaries that are like hundreds of megabytes large,
or in larger.
I think Q blast itself, which is Qdass blast library,
is more to the gigabyte software of device code.
And this, in Vulkan, you don't immediately have the problem,
because the driver interprets and creates the device code
on the fly.
So, the first obstacle I ran into is that Vulkan is extremely verbose.
So, for any step, like in this case,
like the first thing you need to do,
you always initialize a struct,
fill it with some data, run some operation,
put it into another struct, and so on.
So, in this case, like you initialize the application
for struct, you pick instance extensions,
which can, there are stuff like validation layers,
which basically, at the developer,
when he's done something wrong,
and that kind of thing,
and then you create the next thing,
which in this case would be an instance created,
infrastructure, and you actually create the instance,
the query physically devices,
there's another differentiation there,
but three in physically and virtual devices, and so on.
So, that kind of change just keeps on going,
and so you end up with a lot of code that you have to write,
just to get going basically.
In this case, like, you need to hide all of this behind functions,
so when you use QDA, you already have stuff like QDA,
a name copy, where you can copy data from the CPU to the GPU.
In this case, you don't have to write it yourself.
There are some libraries which help you with that,
but as a little bit of a personal challenge,
I decided to just use the working API itself,
and do everything myself.
So, yeah, that ends up with a lot of code.
A lot of work, just for the boilerplate stuff,
so, for example, like instance initialization,
buffer creation, shader loading, command buffers,
shader invocations and copying,
all of that needs to be hidden away,
and only then can you start actually using
the device for anything useful.
So, the second obstacle was that,
the GISR is shader language that you use to write computer shaders.
It's not immediately, like,
it's not as user-friendly as QDAs.
So, in QDA, you can just write the device code
directly into the C++ file,
and GISR you need to write a separate computer shader
and it's on file compiler to spell V,
and then somehow loaded,
loaded intermediate representation,
and send it to the driver,
and the driver compiles that into something
that it can execute.
One of the big differences,
here is that there are no pointers at least in base.
Here, as well, that means there's no pointer casting,
and since I had to,
there was already a QDA backend that I could orient,
that I could use as orientation,
but since it's based a lot of its operation
on pointer magic, like pointer casting,
and that kind of thing,
I had to translate all of that into something else,
that avoided that, which in times can be very verbose.
But also, in the corners,
you need to take care,
because there's a lot of working,
it's a big chunk of hardware,
but not every piece of hardware supports the same things.
I mentioned, like, 16-bit floats earlier.
They are supported by most working devices,
but not by all of them.
And so, you need to, in the shaders,
where at matters, you need to provide,
like, a 16-bit implementation,
at the 32-bit implementation,
at the only way to do that,
and GSA, is with macros,
and so you end up with,
in some cases, with rather,
with codeless, rather hard to read.
But the advantage of doing all of this in GSA,
and with the work in this support,
actually goes back to end of GSA,
the first end of GSA,
and generation, which was, I think, 2011,
which meets us to maintaining support for,
and with a couple of, which is 600 to use,
I think, and, of course, the integral GPUs,
all of them can, in the end, run the code that you write here.
But the last, or the third,
and the probably largest obstacle,
ran into that there is no plus library for a broken.
So, there was no easy way out of,
out of, like,
out of, doing the matrix multiplication,
myself, the,
in GSA, you can use Q-blast,
and, obviously, I was able to use C-blast,
and here, I didn't.
So, I had to write the matrix multiplication,
I don't myself.
Luckily, there was some work by C-1 Burn,
who spent some time optimizing,
optimizing Q-da,
and matrix multiplication,
can it, to try to reach the speed of Q-blast,
and I was able to pod some of that work to work into GSA,
and use it to,
yeah, to, to do matrix multiplications for Lama C-p-p.
And in this case,
like, it took me around six months of working in the evening,
and on weekends,
besides my job,
to, to get this finished,
to implement all of the operations,
and to create,
a pre-request in Lama C-p-p,
was absolutely massive.
I think there were, like,
7,000 lines of code or something,
and a lot more data that came,
with it,
but at that point,
you could run actual language models using working.
On, on GPUs,
that previously couldn't run any larger language models anymore,
at all,
because they have no support for Q-da,
they have no support for rock M.
So, basically, working,
to last thing they have,
it can kind of like,
OpenCM might work in some cases,
but, yeah, in this case,
like, working, actually,
made it different.
So, last thing here,
so working code is supposed to even like,
not six,
or you write something in runs,
anywhere,
basically, as long as the hardware supports everything you do,
but sadly,
of course, not that easy,
and each implementation has its quacks,
and so,
a lot of time after that,
was spent, like,
deep-lying stuff.
I had to build a server,
with,
with GPUs from every vendor.
So, in this case,
like, the bottom one is into art,
so, radion buffed,
and RTX Nvidia GPU at the top,
just so I can run the code,
I write on every device
and check that it actually works,
or,
if someone reports a bug,
I can debug it.
Yeah,
and there's still a lot of,
a lot of stuff to do.
There's, like,
seven open poll requests right now,
they don't have time right now to work on.
But, yeah,
so, just quickly,
the community,
and I must be,
since there are a lot of developers here,
and,
if someone wants to join it,
there's all of the interactions on GitHub,
you can,
it's just issues and discussions there.
Basically,
you can contribute anything you want in poll requests,
and have a,
in my,
in my experience,
so we just,
discussion with,
there's a lot of nice people that will help you,
and give you good feedback.
Yeah,
there's a base team of maintainers,
some back-end maintainers,
like,
I'm responsible for what I wrote there,
and trying to maintain the work on back-end,
and there's a lot of smaller contributors,
that do their own,
do their own thing in smaller,
like,
chance.
Yeah,
there's a lot of,
like,
recently I got support by,
from Jeff Boyd's,
playing video,
which is very nice,
like,
he has much more experience than me,
writing,
and debugging,
working code,
and was able to speed up a lot of code,
and still trying to understand exactly what he's doing,
learning from it,
how to optimize to be your code.
And also some GCN optimizations,
and quantization methods,
that were previously not supported,
like the IQ ones we heard about earlier.
Yeah,
that's it.
Thank you.
If you have some questions,
I mean,
you can reach me on Netflix discord,
I'll get up.
Yeah.
That's it.
And Robyn should be walking that direction,
so if you have any questions for him,
or if you have any questions for him,