Hi everyone so after the previous MLIR talk I will continue on MLIR sorry yeah so I will
also be talking about MLIR but also about like a hardware architecture a bit so
bit on the in between the two so it's yeah MLIR based title tiling for
Ryzen AI MPU which is about like like an MPU in a laptop so laptop like this one
we have a CPU like an integrated GPU and an integrated like MPU so I'll be
giving an overview of like what this MPU looks like in these laptops how we
targeted like how we use data packing to describe like copying high-dimensional
copying through the MS and then I'll be showing like how you can map a gem in
different ways to this device and how we use MLIR to describe like this
parallelization and data broadcasting through the MPU so first an overview
the MPU is like an array of cores and memory so we have like four by four array
of cores and then we have like a row with memory tile and then we have another
row with like connections to the outside world to like a DRAM basically and then
at the bottom we have like a microcontroller that like kind of programs and
controls the device there's a different generations of it like you have XDNA1 so AI
engine XDNA imputes like all kind of the same thing XDNA1 is the first generation
that has kind of a capability of 10 tops on the entire array and then you have
XDNA2 which can reach up to like a 50 tops you're easy
stops with like vector processors which can do for example like 512 in 8
Mac on each tile or for XDNA1 like 256 in Macs across the array I'll be
talking about like XDNA1 mostly in these lights like the XDNA2 as like a slightly
different variations but they're like very similar so what you can do it is
as you have an array you can offload some applications to add AI
applications like for example like a video audio some content creation some
LLMs and you can partition is array to do like multiple of these applications
at the same time by allocating like columns for example like two columns to
audio and like four to content creation now how do you program the device
we're basically generating three things initial like boot PDI core code for
all of the tiles and like microclone total code to control like the array in
that boot PDI also have things like routing for example so the most important
part about like this device and why it's like to get like power efficient
computing is basically that it is like a push model in contrast to like a
pull model like typically have a CPU where you'll basically say I need this
data like a pointer difference and get your data and somehow the data will
come into the like registers there's no such thing here like if you want
data on like one of these tiles you'll have to get it through there and that's
where the power efficiency comes from and I think that's why it will see more
and more of these kind of devices in like specialized GPUs and so on as well
because you can like when you control like where the data goes very precisely
you can get a lot of like power benefits like out of it and performance
because of that reason as well so how do you program it like first the boot PDI
like you need to actually program the routes like the routes from the outside
world like truly connections through the like shared memory which is the
mentals on the second row into the AIE course and these routes like can go like this
you can go like vertically like up in some column you can go down you can
have a route between the mentals and like between the outside world and the
mentals and you can figure that in different ways for example you can have
some horizontal routes you're basically here with saying that we have some
data that is shared like a closer role like all these cores on the same like a
row they will have like the same data packet and they can forward it to each other
and this is useful because like in these AIE type of applications like
convolutions and matmos you typically have a lot of data reuse so when you
paralyze it costs multiple tiles they basically a lot of those styles will need
the same data and instead of like loading that data again and again to like from
DDR like shared memory or whatever to those styles you can basically forward it
from like one tile to another so they all at the same data and that's called like
data about costly so you can go horizontally you can go vertically you can go like down
you can go up and you can do things like this for example like you can share across
different columns that are not like next to each other for example here you have
column 1 and column 3 that will basically share data and column 2 and column 4 that
will share like different data and all these kind of routing configurations are part of
like this boot PDI or you can also do it later on but basically it's a configuration that you
all describe to get this behavior where certain tiles will collaborate on like the same
data or partially the same data of course so then another piece of it is like the core
program basically for vector processor and scalar processor that each core has and that's
like a typical like it's a very long instruction board processor and that's a typical like
what would you target with LLVM for example and you just compile an L loaded into each tile
and each tile will start executing that program but it will expect that the data somehow
already is locally on those styles which you can reach by using these connections that we
program so not all these styles need to have the same off like each of those styles could have
like a completely different program and that way like collaborate in fancy ways with each other
and then lastly we have like the microcontroller which can like the two basically program
the data connections into the array and basically say which addresses on the DDR you're getting
like which data from looking at the core we have like a vector processor scalar processor we have
some local memory 64 kilobytes and then around that we have like stream connections so we have
like some horizontal, some vertical streams in this case for example vertically we have six
to retubate a north channels and four to retubate like south channels so basically a little bit more
data bandwidth going up then then going down typically we need more data bandwidth going up
we have important to remember is that this gives us around like 24 gigabytes of bandwidth like
into each column which I will get back to later on we also have like the memory tiles so
the streams are similar we have some north streams some south streams some east some west
but now instead of course we just have like some banks of memory and some DMAs like we have
six DMAs to write into it six DMAs to read from it and those DMAs can do like some fancy high
dimensional reading with a single instruction so basically here we have like some DMA some stream
to memory DMA so it writes into some bank in like a strided fashion so for example here we'll
have like on the blue tiles you see like for contiguous tiles being written then we have a jump
we write another four tiles we have another larger jump and we write another like six eight tiles
and this can be done with like a single instruction and like in this way the making
basically reform at data while we're moving it like while we're reading it or while we're writing it
it can reform at this data into like a packed data layout which is very friendly for the vector
processes so by the time this data gets to the vector processor the vector processor will just
load like a large block of data into a traitors and it doesn't have to do like any V4
mapping because that's already done like by these TMAs so here it's here I show the similar thing
but how we describe it in MLAR so basically you would expect like like some copy kind of
construct that can move data from some memory to another memory from DDR to shared memory
from shared memory to local memory local to local local to shared and so on and that's exactly
what we do but we also need like a transpose kind of operation and so it happens that in
MLAR when we start looking at this there was already this construct which is a pack operation
which is basically like a copy and like a transposition and we use this copy and transposition
to basically describe like these TMA operations that the copy data from like one memory to another
and then we transform those into a like our own dialect which is the same DIAE TMA copy and D
it's basically the same thing where we're saying we have a DMA we have a source we have a destination
and we have a certain way of like addressing on the source and a certain high-dimensional
addressing on the destination now looking at a gemtiling problem so the question is if we have a
gem problem how do we map it to this array and there's different ways that you can do this
for example suppose we have the entire inner dimension that we're going across like for
example we divide the N and the N sorry the two yeah the AB into like slices but we take the
entire inner dimension for each slice to if A1, A2, B1, B2 and we have to compute the products
of those now this we can do by for example mapping this like we can parallelize this by mapping this
to like four cores and like each of those cores will get like an entire slice with like a full
inner dimension of A and a full inner dimension of B so they can compute like the actual output
and then send it send it out and this works if like your entire slice A1, B1 and so on fit into
local memory and if it doesn't then you kind of have to split that up as well so then we get here basically
we split it across the K dimension as well and now these styles will not be computing like the
actual output result but it will be computing like some partial output result and store it locally
potentially send it out or keep it there locally and then you will send like kind of the next
batch of data we can do this another way as well so instead of like keeping it locally
something in the next batch of data you can forward it to the next style so in this case you
some streams which I showed earlier in the slides that will go like down from like one core to another
and basically you can forward your partial result to the next style which will kind of compute like
the complete output result and then you can keep that locally to some other operation on it like
some softmax on adding something to it or you can send it out through the output streams
here showing the outer product again in the rest of the slides I will just be
concentrating on like the outer product not like the inner product where we forward to another
tunnel here basically we'll be forwarding slices and parallelize it across an array of
tiles but no like forwarding like every slice will stay local until and that core will
keep repeating them at mobile on those slices until it has the output result like no sharing
between tiles but that is possible so how we describe that in MLAR is for example like this
we tile the loops and eventually we end up like some inner loop this SCF40 which goes from 0 to 2
so basically we're describing we have two dimensions we're describing an array of a 2 by 2 array
and that is because you here have this core mapping in the wide direction and the x direction it's
basically saying that we have two dimensions to one's y 1x and each is 2 goes from 0 to 2 so basically
2 by 2 array and then we have this DMA copy and these that we saw earlier which describe
copying data from like these mem tiles to these air itals and depending on like this f i
an expression which depends on like the induction variables which are mapped in vertically or horizontally
it will basically save which data slice from the shared memory needs to go to which tile
and this f an expression because it depends on both the induction variables like each tile
will basically have like a different slice like one tile f a 1 b 1 the other a 2 b 2 a 3 b 3
and a 4 b 4 and if you calculate that and see what the bandwidth is the average bandwidth needed
into each column it's around like 16 gigabytes bandwidth at one gigahertz and if we go back
to like the streams that we have available you see that we have 24 gigabytes available going north
with needing 16 but the problem is that we are about only looking like two rows and we have
another two rows on top of that so if we double the bandwidth and for this configuration we need
like turn it to gigabytes of bandwidth which we don't have available we only have 24 so we
can't run like at full speed so we can find the solution for that like we use more data broadcasting
in this example like we again have the same loop the same as f roll the same mapping in the
wind direction but those the m a copy and these they don't depend on like the nf and
expression of both induction variables they basically just depend on the induction variable which
means that like a 1 will be shared across the rows b 1 will be shared across the columns b 2 as well
and so you get this like a full outer product of a 1 a 2 b 1 b 2 and this results in the
rounds like eight gigabytes per second of bandwidth needed into each column if we double that to 16
or well within like the 24 we have available another configuration here where we can for example
if a is small like it's vector and like b is a matrix we will factor matrix multiplication like
you can broadcast a to like all the cores in the 2 by 2 array and send a different b to each core
and combine those we need 10 gigabytes spent with this will still work out if we scale to like
4 rows adding dimensions so if we add another 2 columns to it like we can even more efficiently
like share data uh resulting in like only six gigabytes of bandwidth on average needed into in
the column important is this average because you can actually do the routing you could
proud is in the wrong way where like all the bandwidth goes to like a single column and basically
you end up still in trouble but if you do this well then like on average you only need like six
gigabytes of data um here similar thing are we get to six gigabytes of data bandwidth that we
need but only using like two dimensions so here you saw we add another dimension to achieve
this configuration you can get like a similar bandwidth using two dimension as well by like
permitting the dimensions and now we can add blocks as well so we can add like an address here for
all as well so now we have like nested as here for all loops and this is like how we uh
tile and then eventually map it to like cores and to blocks in in certain directions and here you can
have like yeah we can have like uh two rows for columns but like each of those uh two columns like
two sets of two columns they behave independently um and that's because of like this bulk mapping
so we're basically doing like similar computations and each of those columns um getting to yeah
an average of the eight gigabytes of bandwidth so this works out um and that's useful for example
if you want to do partitioning if you don't know exactly how many columns you might have
available you can like kind of map like these outer dimensions to blocks as well to scale from
like two to four columns to eight columns and and so on and yeah achieve this partitioning on the
across the device yeah so to recap um we use MLA are basically mostly the pack operation
and tile and fuse to get this back into into the loops and then we map this to certain like
a horizontal vertical dimensions to describe all these different configurations and which data
can be shared across this a engineering um and which will result in like different uh bandwidth
which uses different compute and result in different performance uh on this device all of this is
open source so I put the link here if you're interested and you can also reach out to me so then
that's it and I'll take some questions yeah I think you're very much really nice so if you
have these providers different possible mountains with different products in bandwidth etc so my
question is if do you have currently or do you have any plans to build an automatic tool to generate
or to optimize those matrix many based on CGRM offers etc but not only works with makes it
that we multiply but maybe with other kind of code any kind of MLR uh not really sorry yeah so
whether we have any tools available for like doing this optimizing tuning for example these
mappings um and we we currently like don't but there's like a lot of interest from like people because
we have kind of not opened it up all the way like there's a lot of things like routing you can
describe so you can completely parameraturizes and like start tuning for best performance
how we are just currently working at like things that we know what work so we're target specific
configurations but you could totally do that and there's a lot of interest from people to
like start doing that yeah yeah if I start taking my signal processing algorithms and
platform uh what kind of a debugging options I can compare to debugging signal processing
algorithms for the CGRM. Out of the box it depends a lot like here in this project which is
completely open source there there's no not a lot of good debugging tools yet for this there's some
work on basically uh basically if you want to get something out of this core right you have to
uh it has to go through the streams so you basically have to program those streams and say like oh
I will add in like a debug stream and then locally I will put some events some traces
and stream those out and then into some buffer and I'll inspect them there's some work on that
but uh I think a lot more work on that is needed to make it like convenient to work for like any kind of
design. What mechanisms do you have for the type of communication with one another?
What mechanisms for the between the tiles? Yeah so um yeah the question is what
what mechanisms do we have to communicate between the tiles there's different
mechanisms once you have the streams you can put something on the stream rather than another
core and get it from the stream so that's one another way is like some of those styles that
are neighboring tiles they can access each other's like data memory so basically they can just
generate like a DMA instruction to read some memory from like the tile above for example so that's
another way um another way is that you like send some XMM requests to basically you write some data
on some of the other tiles you use some logs to communicate between them as well and you can also
like I mentioned before you can collaborate on like a partial result so there's this like a
cascaded stream where you can put some data onto that stream and it will basically go down
to the next tile and that's like a higher dimension for against 64 bits or 48 bits and you can
specifically collaborate on like these kind of like high dimensional intermediate results like
from Metmus for example or convolutions so there's a bunch of different ways and it depends
kind of the use case like a retrain for collaborating on a mat will on a soft max or different
kind of computation yeah thank you very much thank you