All right, okay, I don't want to swallow it, all right.
So we are a young startup, like five months out of stealth, I am Vinay, and this is my
colleague, Geon, and yeah, we want to show you what we've been cooking.
So before we get started, let's have a look at the AI game from a special point of view.
We have the world divided in two sections, basically.
We have training, we have the training guy on one hand, and we have the inference guy
on the other hand, and introducing our players.
We have training, training is typically done as a research endeavor.
You have one of something, which means basically you train your next best greatest model
of all time.
More is better, you want bigger models, you want more modalities, and everything, and yeah,
obviously you want to iterate fast, and yeah, you love Python.
On the other hand, we have inference.
Inference is run in production, so this guy operates in production mode, running thousands
and millions of models doing billions of requests.
In this scenario, less is actually better, you want to consume less resources per model,
and smaller containers, and everything, you want predictable latency, and Python there
is, yeah, this is some story, yeah.
And so when you write the framework, any AI framework, you should obviously prioritize
training, right, because when you're training framework, you get inference for free,
and this is what most of the frameworks have been doing, but the experience is often like
this.
So the Python ecosystem is sometimes not the most friendly one to get started running
inference.
So when we talk to potential customers, they are typically AI-flavored backend engineers,
and they have very strong demands, like accelerator, agnosticity.
They don't want to run only on, or it'd be first to only run on Nvidia.
They want compiled models with static typing, cross compiling, runtime cent boxing,
and extra less.
Parallel IO, Async IO, Kubernetes, and all the good stuff.
And this is why we created Cedemail.
So how can we best describe Cedemail?
It's resting on four pillars, ZIG is a programming language, producing MLIR, using open
XLA to compile the models to the accelerators, and yes, we use Basel to orchestrate all of
this.
So we use ZIG as a front end, so you write a model source code, and ZIG will show some
examples of that later, and your ZIG program then produces MLIR, which passes it to open
XLA, which then targets your target platform, and you have a handy Basel interface to do
all that.
Speaking of ZIG, our framework is for inference only.
At the moment, we don't even bother with training because the inference demand is so high,
so it's actually okay to specialize on that.
So as I said, you write your models in ZIG, and what we achieved was zero Python in our
stack.
We can still load PyTorce models, we have a ZIG implementation of that, so that's good.
And the focus is on producing readable, maintainable, modular code, that's statically compiled,
and where it feels more like, if you are a systems engineer, it feels more like you're
doing proper programming, right?
Anything against Python though, so.
Is that a hand over to?
Okay.
So this is like the main dot ZIG file, I'm not sure it's pretty readable anyway, so the main
point of this slide is to show where it's looked like regular ZIG code, so we have a fine
going controller allocation, but we also swing a few modern features that we use as
synchronism, so typically what we do here is that we first open the PyTorce files, we
extract all the shapes, but we don't load the white sets, then we take off the compilation
or the model, paste on the shapes, and yeah, as synchronously we also load the weights on
the device.
But now let's look a bit more about the model code looks like, so the idea is that to make
something which is familiar, if you're coming from PyTorce or other high-level frameworks,
and even though it's still ZIG, statically compiled and so on, we made it so that we
you don't need to under a location, so it feels more very high-level, but we still try
to add a few goodies that's to make you want to write this code because, and what
does that mean is access tagging, what basically what we do is we give names to the different
axes of tensor, and we can propagate it to the different operation, so a matrix multiplication
and so on, and it means in practice it simplifies a bit a lot of the model code because
you need less transposition, especially I want a matrix multiplication, so to be concrete,
like if you have an image tensor, so it's a tensor with three axes, you have the width,
the height, and the channels, so usually you would refer to the width by its offset, so offset
zero, and you have to remember in your code that offset zero is the width, and sometimes it becomes
offset one or offset two, depending on what you do, but if you give it names, then eventually
transpose, you can just use the name to refer to the width and always be consistent
with the other program, also means you don't really want to write transpose one zero two,
just say I want to transpose that the heads and the width, then the channels, and it gives you
that, and for watch multiplication, here we have A and B, which are two matrices, but they are
not in the text book, they are out from watch multiplication, so we cannot choose like the
math rule operator, which usually find it frameworks, so what we do, we give them names,
and this A and B have both axes, namely K, we can say I want to multiply A with B and contract
over the K axis, and you just do what you want, and it scales particularly well to even more
complicated operation like the self-attention, here we just say we want to multiply the queries
with the keys, over the axis name, head dimension, then we want to compute the self-max, over
the other keys, and then we aggregate all the values, weighted by the attention of the weights,
and you can open up this code in the compiler, and you have a lot of transpose everywhere to make
sure you can do the math rules on this one, here it just goes away, if you want to learn more,
you can check out the docs, we have tutorials, builds with the open source tool,
ZIN, and now I'm going to give the mic back to Rene for the rest of the paper.
Thank you, so that was Zig, let's come to OpenXLA, well OpenXLA is a huge ecosystem,
and it's also backed by the who is who in AI, what else can you say?
Yeah, so, in combination with the produced MLR, you can use OpenXLA, or set ML uses,
I'm openXLA to produce highly optimized code for your target, which could be an Nvidia GPU,
it could be a GPU, it could be an AMD GPU, yeah, so it supports the important things,
like kernel fusion, collect memory allocations, it's all highly optimized, auto tuning,
and it produces very mature and stable MLR, so this is not experimental code, this is
like industry grade, MLR, you see a picture of it here, actually you don't see it,
but it will be in the slides, you can download the PDF, it's very colorful and very nice,
and very professional, yeah, and we also add baseline to the mix, it's like the user interface
for you on the command line, and you can do some amazing things, for example cross compiling,
this is something that Zik does out of the box, but Basel also supports it, and we pull in so many
third-party libraries, so the whole cross compilation story is covered by this, and you can do it
from Basel, so that means if for example, here you're on a MacBook, you can compile your
model, cross compile it for a Linux AMD 64 server, and then just copy to the server and it will run.
We do an awful lot of runtime, trimming, and sandboxing, so some of those frameworks like
CUDA, or the runtime, you need for rock M, they can get pretty large, and especially for the AMD
ecosystem, we managed to reduce that by roughly 90%, so we just take out only the required shared
object files and whatever is needed, and bundle it together with your executable, so you can
create self-deployable archives, like I just said, even with cross compiling, you can for example
create a tar archive that you then just from any machine, that you then just copy to your server,
and run the executable, and it will start doing what you want. Obviously, we can produce OCI images
with Basel, and yeah, and ready for Kubernetes deployments, and speaking about CUDA, yeah, all of the
things I just said is probably well illustrated by this mean here. There are many such cases where
you get version incompatibilities, and you have your server or your machine, you installed the
CUDA driver, and then the user's base libraries, your tens of low, your PyTorch state, they
have a conflict, and it's sometimes really, really painful, and this is what we basically
get rid of with the mail by doing our sandboxing, so actually you don't need to do any provisioning
for the mail models, you just need to copy them, so it's just a deploy stage, there's no special
provisioning, no special setup on the server's required, and this is what it looks like, okay,
I'm not sure if you can read it, but it's in the slides anyway, basically we show three
basel command line examples here. The first one just says basel run,
optimize MNIST, and produce an executable that works with the CUDA runtime. The second one is a
bit more involved, we pass to the basel build command that we want to create an archive, which means
a tar archive that supports the rock M platform for AMD, and also the Linux AMD 64
host architecture, and you can run this from any machine, you can run it from an arm or whatever
M3 MacBook, and it will still cross compile everything, and go from your development machine,
just copy the stuff over to your server, and it will work. Coping stuff over, why would you want
to do that if you can just push an image, so that's the third example, you just say basel run,
MNIST push for CUDA, and for TPU, so what you get is a container, you can pull on a machine that has
that has an Nvidia GPU, but you can also pull it on a machine that has a TPU inside, and the
container will auto detect and start up and run just fine, all with just one basel command.
Yeah, so speaking of open source, we are on GitHub, very easy to find, say the mail,
say the mail, if you want to check out the code, please do so.
And something we are quite proud of is, with you, Jan Lecker, who is also often referred to as
the Godfather of AI, thinks that the mail is what it is, impressive, and this man is a genius,
so I won't argue with him. So, what's next? Obviously, we are young, so we want to support
more chips, more models, we just want more, more modalities, more integrations, and we are also
working on an LLM server, we are pretty far with it, we call it LLMD, and it's super fast, super small,
and super cool, and we hope that soon we will also give you a chance to play with it.
And speaking of playing with it, yeah, this was a short intro to the said mail, and this is basically
the least, how you can run models on your machine, and basically that's it, and now we have
a lot of time for questions. So, I would suggest if you have a question and don't mind,
it can close, so I can hear you, and then I can repeat the question or you.
And if you don't have any questions, we'll have a longer break, okay.
Yeah, so I try to repeat a very long question, and it went along the lines of how do we find,
so for example, we showed the amnesty example, and the question was how do we find
this symbolic dimensions, and I'll try to answer the question, maybe I misunderstood it,
but we have enough time. So basically, what we do is, so one thing that's very important,
Siggy's a compiled language, and in the compilation step, you already need to know when you run it,
you need to know the shapes to produce the MLR, right? And so you need to get those shape dimensions
from somewhere. And in this case, like in the emnest case, or in the llama case, we get them
by loading the weights from disk. So if there are safe tensors, they have a meta information,
we just grab the meta information first, compile the code to MLR, then keep loading
asynchronously the rest of the weights, and then it's done. Do you want to add to that, or
typically the batch size is an input to the compilation, so yeah. So we get the actual dimensions
by loading the weights. So we use symbolic. So the follow-up question is, I hand over to
the film, he wants, for seconds lens, typically what you do is you have a max seconds lens,
and then you compile for that. And if you really want more dynamism, you can compile several
batch versions of the kernel. Yeah, that's some approaches, like with project extensions, where
the seconds lens is not that much of a problem, so you can just compile for very long seconds lens,
and you can trade off at one time between batch size and seconds lens, but yeah.
Okay. Yes.
Okay, so the question is, since we are using ZIG, are we not using the build, yeah, the ZIG build system,
the reason for that is that basal is more mature, and some of the dependencies are also using
basal, so typically open XLA, and we compile the part of LVM, so with basal it's easier to do this.
We also generate like the current images and build the content to that for now, so
we need to find a, we are working on a way to make it easier to call or stuff from build.zIG,
but it's not ready yet, so yeah, but it's a common frequent from people using ZIG, but yeah, we have a
working on it. So I'm not sure on the, I think the question is, how do we import
PyTorch models? Yeah, so for PyTorch models, you have two things, you have the weights,
we can load the weights typically used for PyTorch models, so either that, I mean, the PyTorch official
formats are also safe tensor, but then you also need to pause the code, we don't, I mean, yeah, you need to
rewrite the inference code. We tried to make it this easy, it's a bit of work, but usually,
I mean, you probably already have seen like Lama implementation in one file, so it's not that much
work, if you know what you're doing, and also sometimes the inference code is different from the
training code anyway, because you want to optimize for varying set lengths or pre-filling versus
generation, so yeah, so this rewrite is often a bit needed anyway when you go to production.
Thank you very much. We start in a three,
three.