All right, hello everybody, hello everybody, my name is Yolsa and I will be talking
today about generating programmable ampuse with linoch, sorry from linoch with MLR
and circuit. I only have 20 minutes, so I can show you everything, but I'll do my best.
So I'm Yolsa, I'm currently a PhD researcher at Kilo Loven in the Micah's research group
and I've been working on compilers for AI hardware exilators since 2021.
Fun fact, I'm also looking for a job at the end of this year, so if you know anybody who
has work, then send them to me. So let's briefly go over the agenda for this talk, so what
will you learn in the stock? What are ampuse? How do people design ampuse right now or as far
as I'm aware of how people design ampuse right now? And how people should design ampuse in the
future according to me, with my very opinionated opinion, and spoiler is going to be with compilers.
So ampuse are processors that optimize events within your algebra, if you're not quite sure
what density in your algebra is. It's usually a bunch of forloups with some aridmatic
on a bunch of data. So in this case, for example, this is a C code for an element-wise multiplication.
It's quite simple. There's multiplication and there's the forloup. There's other kernels as well,
so a matrix multiplication. There's the forloups and there's a accumulate and multiplication.
And recently there's also the softmax layer that's super popular, for example, in large language
models. So there's a lot of other different operations that are happening here.
I think what is quite important for you to know at this stage of the talk is that in MLAR,
we typically represent this in the LG dialect, and so you'll see that at a much higher level of
abstraction, you will be representing kind of the same thing. So the forloups are represented
as a fine maps. And then the operations themselves are actually represented in the various
dialects that MLAR has. So for example, floating point multiplication in this case for this example.
Now, so what is an MPU? There's many examples of MPUs, so you can see the Google TensorFlow
sorry, TensorFlow Synunit V4, the Apple M1 and all the series process after supposed to
also have this neural engine. Basically, if you're buying a laptop today or a phone or whatever,
chances are that you're having one of these NPUs. So you might be wondering, are they useful?
So according to research, I'm mainly doing research. Our stance on this is of course.
There's this graph on the left that shows you that a lot of things that we have been trying to
do are currently not possible anymore. So we cannot scale up the frequency anymore, scaling up
the single trip performance is also difficult, scaling up power is difficult, and a number of
logical courses is also difficult, but we can keep having more transistors. So there's more area
for specialization, essentially. And then on the right side, there's this plot that shows you
the performance per what's increased when they compared in 2017, the TPU V1 versus various
configurations of CPUs and GPUs, and you can see that for this example, specifically, they get a
200x increase. I should know that this is for a not implemented example, it's still research.
But in reality, are they really useful? Yes, but according to me, it's mostly for marketing,
because in reality, it's quite difficult to use them. These units, they can be too specific
for newer workloads. So maybe they were just nine, two years ago, and two years ago, they were
in the world, that's where they were, I'll work loads, but people have moved on, there's new way
I work loads, and they just don't work on these old accelerators anymore. Also, most compilers cannot
leverage them directly. So if you send the C codes into Clang and you're expected to work on
your Apple neural engine, well, probably it will never end up there. And also, the hardware
firmware and libraries are often close source. So tough luck working on these things.
So in principle, there's multiple interdependent problems, there are new algorithms,
for which we fail to quickly create new hardware. There are new, for the new hardware that
we created takes too long to ship compilers support. And for people to actually use our devices,
there's this close source, those people make these new things. They're like, all this works
really good, and then they want share it with you. So this is kind of annoying.
And so maybe before we think about any solution, it's useful to think about how we ended up
with this problematic situation. So how do people typically create and use? In the beginning,
there's this time where people look at the different algorithms and they start thinking,
like, I would be cool if we can upload this specific part of an algorithm to an accelerator.
Then let's say two months later, they start working on the hardware design. Then six months later,
they're far enough in the hardware design so they can come up with some kind ofization specification.
And then six months later, after it's properly settled, then a library and compile development
can start. Then once a year later, maybe if you're lucky, you can start shipping it to your customers.
But I should note that these are very optimistic estimates. And it's still taking two
plus years to go through this full cycle. And by the time I just said, if someone has come up with
a new algorithm, then all of the work is for nothing, because we're starting from scratch again.
So this is way too slow. The question is, can we do this faster? And the answer in my opinion is,
yes, with compilers, we can do this thing, which I want to call programmable hardware synthesis.
But how, well, we have all of these really cool compilers technologies that we can leverage
to do this. So how do we quickly create a machine learning compiler? We have MLAR for that.
How do we quickly generate hardware? Well, we can use the circuit project, which is a counterpart
of MLAR, the specifically focuses on hardware design. How do we adapt the compiler to the hardware?
Well, if we dynamically register new backends to the created hardware, and how do we quickly
develop this? Well, that is by using a few secret shortcuts, which I will share with you later.
And of course, everything is happening in fully open source, which is cool. So everybody can
hack on this. Now, what are we trying to create? So I think it's useful if we go over the initial
or the typical anatomy of an MPU platform. Typically, what you have is a bunch of memory at the top.
I connected to the memory as a CPU. Usually you use a CPU to also control your MPU. So the
MPU is never really alone. And then to take in, then there's this processing elements. Usually
we use a bunch of them, so you can fully exploit the parallelism of your workload. And then to get
data into the processing elements, you take a lot of load units, or you take a load unit,
it loads in a lot of data. And then there's also typically a store unit that stores back the data
once the processing elements are done with this. And how do we quickly create on this? Well,
what I'm proposing is this programmable hardware synthesis flow in which we technically just create
one giant loop to do the full design of everything. So we start off with the supported operations.
So the things that we want to make an accelerator for, the things that we think are interesting to
accelerate. Still an open question to choose which operations are actually useful. But I hope
by making this quicker that we can find out what is this sooner. Then out of the supported operations,
we generate this PE or the processing element array. And then we stick it together with the
load unit, the store unit, and the CPU. And then we can actually create a simulation of the platform.
And then we have this machine learning compiler, which also ingests the back end information,
which we also get from the supported operations in MLAR. And then we use that to create a binary
which you can run in the simulation on this platform. With this software, all of the issues that we
have. Well, new algorithms, we can just create new PE or at least at the design stage. We can
play around with new multiple PE. So that's not really an issue. For new hardware, there's also
another issue for the compiler support because it's also happening in the loop. And everything is open
source. So everybody can just play around with it, which is a lot of fun. And what I want this
to work in is in two minutes as opposed to two years and two months. So we'll see if we'll get there.
I'm still working on it. And because I can obviously not share everything with you today,
today we'll focus a bit more on this concept of PE every conversion. So to remind you,
those are these prosing element parts of the NPUs. And so what we're going to do, so in more detail,
we have a look of these of these supported operations, which we typically express in Linux in Erics.
So you have Linux in Erics A, Linux in Erics B. Then you convert each of these Linux in to actual
processing elements in a dialect of MLAR that I create myself. Then there is a way to aggregate
these PE's into a PE that supports multiple of these operations. And then you just multiply them
in a way that fits your parallelism. And then you have you end up with an MP with a supports
Linux in Erics A and Linux in Erics B. And if the source file that you use, so if the software
workload that you want to run on your hardware contains the same support operations as your
support operations that you use to generate your hardware, well, then it's guaranteed to work on this MPU.
So let's go over some examples. So the simple example number one, the task is to create a
four-way parallel processing element array that supports elementwise operations, namely addition,
subtraction, multiplication, and bitwise exclusive for. And so we start off with this workloads,
so you can see that there's multiple Linux for multiplication, addition, and subtraction, and
XOR. In the next step, what we do is we generalize all of these operations, so we get a Linux generic
for all of these different operations. And then we will create a PE that is supported for each of these
operations. Then we aggregate all of these PE's into one PE and then we create the four-way array.
And let's now go over some example where we actually deploy the algorithm on this simple accelerator.
So we use, in this case, we use the same software as we used to create the hardware array.
So the first thing we have to do is we have the array that supports all of these operations.
The first operation we encounter, maybe the difficulty in the back, but it's the addition.
So we program it, read as the addition color, obviously. So we program it to run the addition.
Then the next step is to do this subtraction. So we load in the data, we are
querying it to do this subtraction, and we send off the data back.
And so we do this for all the other operations as well.
So if you look at the end, what you had to do was, you load the data four times into your
processing element array, you store data four times back into your memory, and you need to program
the accelerator four times to choose all of these different operations.
But being in MLR, there's a few cool tricks that we can pull off.
So one of these is something that people use sometimes in software as well, which is called
operator fusion. And what is operator fusion essentially, instead of fetching data from memory,
multiple times, you just fetch it once, and you do all of these subsequent operations on it
before you send it back to memory. Obviously, that doesn't always work in each algorithm,
and you might not always want to do it, because it can generate quite complex workloads.
But in some cases, you might, and because we are supported anyway, we can just write.
So in this case, for example, this is the first workload. If you look more closely, you'll see
that actually all of these workloads are connected. So you can fuse them together by
running a few MLR passes. And so now all of these workloads are actually contained in the same
Linux generic. So all that rest is to do is to create this go over this full Linux generic,
convert it into one PE. Then we make another four-way PE array. Then we to deploy the workload,
we just program it to run this thing. And now the performance would be stored data once,
load data once, and program the accelerator once, which obviously helps with performance by a lot.
But obviously, it is leads to a trade-off. So we will create a more specific accelerator
for more performance. Or maybe this one is too specific, and it's not future's proven off.
So currently with this PEHS system, there should be a way to try these out more quick.
So you might be wondering, I told you about a few secret shortcuts. How did we create this?
So I mean, how did we create this? It's not finished yet, we're still working on it. But like,
how did we start working on it? The shortcut number one that we use is we use XDSL for quick
portrait typing. So for people who are not familiar with XDSL, XDSL is an open source MLR, and
does also circuit compatible framework written in Python. So you can very quickly
prototype new MLR dialects and combinations of compiler classes. You don't need to recompile each time you
can just run it immediately, and then do interactively bugging without any issues.
You should totally check it out, it's a really cool project. Then shortcut number two is for the PERA.
The PERA alone is not enough. You need low tuning, you need story unity, memory, you need a CPU.
For that, we use this next platform that was developed at K11 to create an NPU. So that platform is also
fully open source. It uses a wrist-five core to control everything, and all of the search
core for the hardware is totally open source, so you can also check that out.
And we also worked on, well, before we start working on this programmable flow, we also worked
on a flow to just compile the not just to, let's say, hand made accelerators, which we call
the Snacks MLR software compilation to obtain, which is also written in XDSL, and is also fully open source.
And it's also the repository value to find the search code for this approach today.
And in the future, we want to overlay functionality on parts of the PERA, and we're just copying
the PERA, but it could be possible, and maybe interesting to choose different combinations of PERA.
We also want to support multiple data flows. So sometimes you need to do reductions on your data,
or you need to, this was just a simple elementwise example, but you need to do multiple different things.
Sometimes it makes more sense to group together certain operations, and several operations in
different pools. In that case, you would generate different accelerators, which is also something
we're working on, and then also support for multi-cycle data paths for, especially for floating
point supports, not necessarily because we like floating point supports in hardware, it's kind
of a pain, but we just don't like quantization. So that's the main message there.
So with that, I thank you very much for attending this talk, and here are the decoys of this talk.
If you have any questions, shoot.
All right, see, yeah.
What do you think about driving to the stream of various specific model territories,
maybe going from high doors with ports, and I are down to an R, and that too soon?
So the question is, what do I think about making very specific accelerators for models?
I think making very specific accelerators for models is a lot very sensible, because it's
like making an AC or taping out an AC, which is the goal of this work at some point,
is super expensive. So I think just generating hardware for one specific model, I don't think it
makes sense. The question, I think is more interesting is what operations do you need to support
a broader range of models without giving up too much of the specific performance you gain by
going for more specific circuits? But I don't know the answer to the question, that's why I'm
making this tool too. Fair enough. Yes?
So I guess the question is, is there a place for more SRAM on the NPU, because the NPU is usually
required more SRAM? I guess that's a question.
I think it's so the next platform, which is where the memory comes from is quite
parameterizable, so if you have more money, you can make the chip bigger, and you can add more memory.
Yeah, so how programmable is the outcoming hardware? I think currently it's just quite simple,
so it just generates kind of like functional units, and then it will deflect your around it.
But depending on how you combine them together, you can like squeeze things together and load it off.
So and then combine with the load and store units that we have, that actually determines
how much dimensions your input data can have, but those are designed time parameterizable.
So that depends on which supported operations you're given. So the question was how
do you decide on the operator fusion? If you give the operators as few
operations in the supported operations, then it will create a few accelerated out of that.
If you supply both of them, then it should generate and like both the unfused and the few
ones. It should generate an accelerator that supports both unfused and fused, and it should automatically
detect that, that's kind of the goal. Yes?
Can I just pick up that first question, a classroom all the way around? You think you're going to want
specific things? If I go and try and accelerate it now, I might make a convolutional accelerator,
but adjusted to a commutes of multiplication and intuitions well. What's the approach that you do?
If you're coming off a single NIO operator, how do you say, oh, there's these three operators,
like agreement together, and I can handle it so, and then we'll actually accelerate all three.
The scenario costs are just having three. Sorry, so can you repeat the question?
So, you were asking the first question, if you already have a convolutional?
On getting more generic accelerators?
Yeah, so I don't know. You've got three NIO operators.
Yeah. If they're related, you can have a single accelerator.
Yeah. I think that is, so I think the question is, how do you aggregate these accelerators?
Yes. So, it's that you have one accelerator and three different accelerators for each
new operation you add. I think part of that is in the way how we support the various data flows.
So, the load and store units of snacks are quite programmable. So, they will support many different things.
And then, currently, the aggregation is quite limited, but that's currently something I'm
working on to make it possible so you can aggregate more of these, even if they have different data flows.
For example.
Thank you. Thank you.