Okay, quite down everyone. The next talk is about to begin, quite, quite down. Okay, good, good, good.
So it is a great pleasure for me to introduce our next speaker. Ruben came to this Devroom last year and
did an absolutely amazing presentation on Vulcan and the use of Vulcan and Lama CPP and
ever since I've been bugging him to keep presenting on that lovely subject and so he agreed
and so he is taken away Ruben. Thank you.
All right, so my name is Wilmotlam. I'm now a very fresh machine learning engineer
from Reddit but this work has, like, my work on Vulcan has mostly happened in my free time,
actually. It's a great match of that. But I want to, like, briefly introduce what's
the Vulcan API. Why should you even care about it? Why is it relevant? Just briefly about
Lama CPP as well, like, then what have we done since last year? There was a ton of work
on the back and on the Lama CPP back and there's a lot that has changed since then.
Some benchmarks actually, that show like the part is how does it actually, like, compare to the
usual suspects for running large language models on GPUs. What are the difficulties that
are struggling with and that you will also struggle with if you try to use working for some
of this and the conclusion about, is it worth using, is it worth putting in the time to use
Vulcan here? So, basically, what is Vulcan? Isn't that a gaming API? So, yes, it's an API
graphics. You can actually, like, it's a successor to OpenGA and the idea there was to get
rid of some of the inefficiencies in OpenGA by making it a lot more abstract. And what they
ended up with is basically, like, a generic interface to GPUs. And so, you can actually use the same
kind of API codes and the same kind of shader or, quote unquote, kind of code to run on all
kinds of GPUs, not just the Nvidia, the usual Nvidia graphics start. And so, my interest
here is mostly about, like, I don't have a huge, like, $200,000 Nvidia servers somewhere.
I don't have data centers. So, I just have, like, some PC somewhere with an old graphics card.
How do I make that run in a large language model so that I can run, so that I can actually
use it for something that I don't want to share with a cloud. So, Vulcan can actually do that.
You don't actually have to use the graphics part of it. You can just from computer
class as a replacement for kernel, it's basically the same thing. And that way, you can actually
use it for machine learning. And what I did was I edited to learn a CPU, like, over two years ago
now, and it has grown a lot since, um, learn a CPU probably, you should be familiar with
with, with also talks already about it. So, I'm not going to go to deep into this, but basically
the idea was that whatever hardware you're playing around somewhere, you should be able
to run an element. And I'm on it. Of course, what kind of, I'm, you can run, so it depends
on, like, how much memory you do you have, and how, like, patient are you, with waiting for
your responses. So, you're a long as you be based on a, like, static graph structure.
It's, uh, not a different from, like, other approaches we've seen. But the cool thing
and that has also grown, like, I think since last years, that, uh, it's basically
abstracted away all of the back end stuff into, uh, something that you can execute on various
patterns. So, you saw the graph, the compute graph that contains all of the different operations,
gets sent to a back end, and it can even be split up and sent to multiple back ends. So,
there's a lot of, like, interesting stuff you can do here. So, there's a lot of, like,
back ends that currently exist, like, the most relevant ones, of course, like CPU,
CUDA metal, and Vulkan. The rock M1 for NDS, basically, on top of the CUDA back ends,
the pre-uses, most of the code from that. And then there's also an open-sear based
one that's, I think, that in that mobile phones, there's a can for, I think, who are
away accelerators. There's Web GPU that's, I'm not sure how usable that is yet, which is also
interesting. And some, like, Blas, as NDS, which is just trying to make the, trying to
offer large matrix modifications to libraries that are more optimized on the CPU and that,
in that example. So, what actually happens since last year, one of, I think, the major,
like, the most important thing we've done is, like, is flash attention. So, probably,
if you've ever looked into, like, attention and the way it's used in large language models,
you've also come across flash attention. It's, like, the paper was hugely influential.
There's multiple versions of it now, like, the usual way to edit, I think, in, in, in,
PyTorch based projects is to actually use the, like, the, the code from, from specific,
GitHub repos. So, we have to custom shader there that was, actually, not, I didn't write
it, that was, just was from a video. There, in last year, I think it, like, the one version of it,
like, that was Nvidia specific already existed. That's the Cooperative Matrix 2 variant.
That's some Cooperative matrices, this boycins abstraction for tensor course.
So, or, or, like, any kind of, like, matrix acceleration. And so, since last year,
we've, like, we've also worked on the, like, Cooperative Matrix 1, which is the
Kronos version, which is not specific to Nvidia. So, that is the one that, for example,
run on modern indie hardware. And, of course, there's also scalar version.
If the GPU doesn't have any kind of hardware acceleration for matrix multiplications,
even, we can still run flash attention. It will give you, it will give you, actually,
a huge increase in performance with, with context that has grown very large.
That's become incredibly important in modern logic initiatives, because the, because the context
that these models can support is extremely big. And so, with, with this, you fuse a lot of,
a lot of operations into one. And so, you do not need huge intermediate buffers.
And you get, and you can run it in one, like, in one kernel core instead of a bunch of operations.
So, with something, like, 128K context, you will see a huge difference from using this.
And so, implementing this was very important. And also, like, making it available to more hardware,
was also a huge step. And, is one of the things that made, that made us, made a huge difference
for performance in, in the Vulcan backend. There's still a lot to do there. Like, I've just,
just over the last few weeks, I've spent time, like, optimizing the AMD, the version running
on modern AMD hardware. There was, like, I got a ton of performance, all of that.
I had some crazy reports of people getting, like, four times faster inference from it.
So, that's, and the same thing, there's probably still a lot of, like, optimization work
that can be done. So, yeah, for anyone else wants to take a look at it.
I would be happy not to have to do all of it. So, yeah.
Another thing that, that's actually, like, one thing, I wanted, like, I worked on, over the last, like, maybe,
half a year ago or so, is, like, using the PFI, or in eight accelerations that's hard
to feature, where you have, like, a dot product of, like, a four, like, packed in eights,
and one, you packed them into one, in 32, you multiply each one, you add that to another
integer, and you get a result, and all of that happens in a single clock cycle.
And that's, that's something that's available on some of the GPUs that are not,
that do not have the hardware to use, like, something that tends, of course, to accelerate
matrix modifications. And so, this is actually something very interesting for, for
me, because we mostly focus on quantized models, and the quantization schemes that we're using,
they make it, they allow you to do a lot of the operations within, like, in eight,
like, multiplications and additions, so you can actually use this, and also get a lot of,
like, a big performance increase. We, the hardware that this most affects is, for example,
Nvidia Pascal, which was the last generation with all tens of course, which does have this
dot product support, and we also have, like, an AMD, the Vega 20 is very interesting,
that was one of the, like, secret, you want to do GPU, you want to do, like, GPU,
acceleration, like, large, for a large range, multi-tone, what are cool accelerators, you
can import from China from data centers, something that was one of those, was the MI50,
that's very, got 20, so that's one of the things that profits a lot from this.
And the inter GPUs also, this is also, like, the fastest, you can,
other, the most, like, relevant acceleration feature on, in the economist.
And so, from writing, like, I just had to write this code, like, I had to add this to the code
once, and it runs on all of these GPUs, and also helped in some other cases,
with, and so, of course, when usable for other reasons.
So, one more thing that's very interesting that, primarily, just both of us, has been working
on this operator fusion. So, in large range, most of you have to turn off, like,
I mean, this, this example is accepted, exaggerated, I just made up some,
times, but basically, for each operation, in the models, you often have the pattern that
you have a big operation, and then a few follow-up, like, small transformations on the same,
on the same, on the same data, on the result of the big operation.
If you, if you do that in the, like, in a regular way, then you get, you get some kind of
dispatch over it, you have to load the memory, you have to do the calculation, you have to
start, and then you have to load it again, and do another transformation on the same
data. And so, if you put all of that into the big operation, basically, you can
save a lot of time by not storing the intermediate result, and by
dispatching, if you are kind of, or computers. So, that's one optimization.
That's quite useful, but it's also very specific, like, we don't have a generic way
of doing this, so you can just, if you add a new model, architecture, and it's
working, it works differently, you won't immediately do this because the, because
the operations don't fit what's already implemented in this sense.
So, the existing fusions won't apply to the new model, and so someone has to go and
look at the new model, figure out where it's potential for fusion, and then
actually implemented. So, I thought some cool ideas about how that could be done in
a more dynamic way, but that's one of the areas that we're still, like, that
would be interesting to look at, but someone has to find a time, of course.
There's much more that has happened, so we also got B416 support, which
wasn't originally in support and working, but got in through extensions.
There was a lot of work on reducing CPU overhead, so in, in the beginning,
we had, like, last year, even still, we had, like, kind of try run.
That way, you had to go through the whole model and figure out how much
memory we actually need in the temporary compute purpose,
allocate that, and then run, and then go through the whole graph again,
and actually run, then actually run the, the computer
uh, kind of. And then, uh,
yeah, we found a way to reduce that by,
by, uh, by doing all of these steps, like, on demand, so you,
actually basically just wait until all operations are done,
and then, then, figure out, like,
resize this purpose, something, and then, uh, continues.
So there, there was some crazy work on, like, fences, like,
about, like, basically fences, something you wait for, so you wait for an operation,
on the GPU to finish, and, uh, someone figured out that,
if you, actually, just wait until the whole, like, graph has been computed,
that takes out some of the, like,
that, uh, that, that for some, uh, the CPU has to wake up again
after sleeping for so long, that it actually takes quite a bit longer,
and has to be, and so, that was solved by adding your fence somewhere
early on the graph, and then busy idling,
naturally, and so the CPU is actually sleeping at that point.
Uh, there were also some stable diffusion operators, which isn't, like,
relevant for large language, uh,
moderate, but there's also, like,
it's very cool to be able to run, uh, like,
very fusion and Vulcan, and, uh,
huge, like, huge number of all that stuff that happened here.
So, uh, I want to show some benchmarks here.
So, uh, on Nvidia, actually, like, uh,
basically what I've done is just run the same kind of, like,
the, the llama bench tool that's in the repo,
and, uh, in this case, on my 3090, and I've run it with a
good aback and, uh, in the, in the, in the, in the
than the same thing with Vulcan, on the Y-axis,
it's, uh, how fast the result was on the X-axis.
It's how, uh, how much, uh, context was in the KV cache.
That's actually, exactly where Flash attention becomes extremely important.
And here you can see that, uh, basically, the Vulcan backend is slower,
and the CUDA backend, but it's not by much.
And, actually, other context, uh, it, uh,
it stays, uh, approximately within that area.
And, uh, so the performance is actually competitive, although,
so, uh, there, so you might ask, like, why, why would I use Vulcan if I
can also just use CUDA, but there are some, some cases
where that would actually make things a lot easier.
Uh, for example, if you have, if you already have a game or something
that you want to integrate AI into, and so, uh, you could now,
like, add, add CUDA into that, but that would be a huge hassle.
Or you just use Vulcan, which you've already using for graphics,
and you can get pretty competitive performance out of that as well.
So this is prompt processing, so prefer that's, also,
where the TensorFlow's could use, um, and, uh,
uh, the same, like, important generation.
There's also some differences, like, the, in GPTOS, as we're still, like,
this optimization left to be done with the lagging behind CUDA there,
but actually there was, like, here on the, on the DeepC2, which is,
that's actually the, uh, DeepC2's, the architect, that's actually the,
and 4.7 flash model of the reason one.
And so for some reason, uh, we actually faster currently in Torpen's generation
on Vulcan, that I could have, which is quite interesting.
Um, so, but more interesting for me, personally, is the, uh,
in the IX-86 years, which is the Strix Halo GPU.
Uh, you can get that with 128 gigabytes of available VRAM,
which makes it very interesting for, um, like, a mixture of export models.
And so here, uh, you, uh, you can see that, uh, on the, on the old Lama,
it be models, it's actually slightly slower than from processing,
but in both the huge GPTOS S120B and the, uh, new, like,
year and 4.7, it's, it outperforms the rock and back and, uh,
in prom processing.
And, uh, talking generation, same thing, even in, in the GPTOS S-like case,
it's actually a big difference there.
It's, uh, it's the Vulcan back and is currently quite a bit faster here.
Uh, on the, yeah, on the, like, year and 4.7 slightly behind, um,
after at longer context, but, uh, faster, lower context, so.
That's work left to be done there.
Uh, there are also cases where there's still a lot of work to be done.
So, uh, there's this, exactly, the very, about 20 thing.
So, uh, there, you can see that the scalar flight retention is actually,
the implementation is actually not optimized for the start where it doesn't run,
where it's, so, really.
So, why we can, can be faster, for example, in the GPTOS S-case.
Uh, at zero context, it drops much faster, so it larger context,
you get much less performance here.
Um, and talking generation, uh, similar, so it drops to faster,
so optimization to be done.
Someone wants to look into it.
I'm happy to arrive.
Otherwise, I'm going to have to do that, I guess.
Um, so actually, uh, another example here is Intel.
And, uh, I wanted to show this because it lighted.
It's an example for driver issues that I'm still running into.
Uh, so you can see that the, that the, like, the resides,
I got done at much sense, like there's something that actually got faster.
At larger context, which isn't great.
And so, uh, basically, this is an example of a driver issue.
Like, the Intel Linux driver is still like, uh, the Vulcan driver is actually not optimized for this kind of thing.
And there's a lot to it.
I'm, I'm having a lot of issues optimizing for it.
And so that leads to examples like this.
Um, I've, I've had issues with all drivers at this point.
Like, I've had, I think I've found bugs in all of them.
So, uh, that's, uh, yeah, this is, uh, one of the things I'm dealing with here.
Uh, the other thing that I'm, uh, still, like, yeah, how do I actually optimize a computer,
and one of the issues I have is for, while Nvidia does provide, um,
a way like with inside graphics to, to get some inside there.
Of, for, for, for AMD, I don't have anything like that.
And for Intel, I don't either, so it's actually a lot of guesswork to optimize a shader here.
You can apply the same techniques as for CUDA, um, but, uh, AMD doesn't behave the same way as Nvidia.
And it's much different, so it's actually, yeah, that's, uh,
I, I had to do like a lot of guessing a lot of trial and error to figure out what is actually fast on what hardware.
So, yeah, in, in conclusion, for working as very interesting, you can, you can get a lot of performance
out of it as you've seen. Actually, you can beat some of you proprietary APIs if you put in another work.
But, uh, the development side is actually much harder than something like CUDA,
because you have to do a lot of the work on the whole side, uh, a lot more work, a lot of boilerplate,
to, uh, to actually get any error with working.
The tooling is limited in comparison, uh, as a lot, like I'm hoping that can be improved in the future.
Uh, it's, uh, like, uh, out there. Um, the hardware compatibility is much much broader than any other, like, uh,
any of the other usual APIs, so that's the big advantage. The binary size is something that's often forgotten.
You actually get much, much smaller binaries, like, uh, if you, if you download PyTorch for CUDA, you get more to
pre-gegar bytes of device code. If you, in theory, if you do the same thing,
if you're working, you would have something very small because the code gets compiled to the device specific code on demand,
during, uh, during the run. And the performance of all you can can actually be very good.
But, uh, there's always something you cannot do with working, uh, so there's the potential slightly lower,
but as you've seen, like, you can get actually pretty close.
Uh, so yeah, so I hope, uh, I've got some interest sitting using working, helping out on the back end,
or maybe integrating it somewhere else. I hope this, uh, that in future, we can, like, uh,
use it more often and get to somewhere where we are not as, uh, as limited to one single,
vendor or one single, like way of, uh, of writing, uh, or where we have to actually write,
completing new kernels to use a different, uh, different GPU, and not use different hardware.
So yeah, thank you.
Thank you.