Hello everyone, I think we can start for this next talk because it's something I'm
really looking forward to see. The last word I used when I was talking earlier was
Matmo and that was the perfect presentation for this. So Pete here is going to talk
about how you get a card and how you actually get to zero to Matmo. Thank you very much.
Okay, Mike check. One, two, good. Okay, so hopefully everyone in this room knows what zero is.
And you know what, Matmo's are. And if you are in the room for the previous talk from
John Necker, you'll know what the E.T. is like. One is for the benefit of anyone else. It's a chip.
It's this chip here. You can find it on cute little PSA boards like that one there.
And all the fun stuff is inside the pink. Right angle there. So you've got four D-armed chips
around the edge and then the main chip on that blue. Pete think. So this is a block diagram trying
to show you what's inside that region there. So each pink rectangle is a piece of storage
and each gold square is a little risk-five CPU core. And then the black network connects
everything together. So if you would count them all, you would find a thousand and ninety three
risk-five CPU cores. The five of them are special. So I'm going to kind of ignore those.
32 kind of night get lost for silicon yields. I'm going to print it down on that either. And then
form where it takes a few. So I'm going to pretend it takes lots. And again, it's trying to take 32.
Living with a nice round of 124 cores to play with. So as this is first time, I want to
call out the open source things here. So the surface like is on GitHub, as one we could just
told you. The firmware on GitHub manuals on GitHub, the source system. And later, again, it's on GitHub.
And the article isn't yet there, but I'm told it will be there soon. And that's the man to
badger if you can talk to that happens. So I've got a thousand cores. I'm one of them at a fixed
clock speed in the name of we produce a ability because I don't like to run the same thing
out of not the same measurement. And the matter is at that fixed clock, this hardware should
give me just over 10 turnflops of FP32 compute. So that's going to be my goal for the rest of the
talk. Now, hopefully some of you will have seen this meme previously, which is the how to draw an
element, which is a step one you draw some circles, and step two you draw the rest of the
URL. The joke, of course, in that step two is rather more work than step one. Of course,
you know, AI, we could just need to ask a distribution model to do step two, nice and simple.
You know, we are here to like implement AI, not just use AI. So our equivalent of drawing an
owl is to create your tensile brief, like you know, GGL or tiny guard or whatever else you want
to use. And step one of that is how the map moves work. And then step two is like everything
else. Again, I'm just here to like draw some circles, so I got like a 20 minute slot. So
you know, I'm just drawing the circles, I'm not drawing the whole owl, there's tons more work there.
So to stay within my time, I'm going to say that a matrix is always 512 by 512 at FP32 because
anything else is drawing the rest of the owl. And this function here will take a matrix A and a
matrix B, more flattened together and put the result in matrix C. It is the worst code you will
ever see if you're doing A, not well, but we've got to start somewhere.
Well, what is it? It's the worst code. It runs a trotusy so on my ETIsog one, this is seven
megapop like per second. It's like, you're calculated it could run faster than that. This is
terrible. But you know, I've got 17 minutes to go, I think. So we can try and make this go faster.
So I'm going to do this over like three main acts and it's hard to go out to one because
going parallel because we are going to be go quite parallel here. So if you've written any
code code previously, you will have seen a thread index and block index.
Because in the code model you run thousands of copies of your code in parallel and each
copy has its own distinct value of thread index slash block index. And I'm using a similar
ish model here. So all the code that I'm showing you will get run 2048 times in parallel and each of
those 248 times will have a distinct value of mh ID between 0 and 247. If you've been paying
attention to the equipment, instead of like 1,024 calls, but 248 values mh ID to push that gap,
the harder it is doing effectively, you can have two way hypersodding with two hearts per core.
So I can start out by taking one of my two loops either the outer loop or the inner loop
and use mh ID instead of having a loop, which should give us a nice five-fold
speed up because now we're now doing five and 12 things in parallel. Unfortunately,
only one version of this code gives the right answer, the other one gives the wrong answer,
which is troubling because you look at the code, you like it should be correct,
there aren't any like typos there, like it should be correct. So the reason for this
lies in the memory hierarchy, which I'm trying to show you with this diagram here.
Now we'll come back to later up so if now I only want you to take one thing away from it,
which is that there are a lot of caches on this chip, so every box with a dollar sign, that's a
cache and there are up to 2,465 caches, which is quite a few, and there's no currency between any
of them, which is a slight problem. It's going for GPU kind of background, you'll be like,
yeah, non-caches, totally fine. It's going for a CPU, quite a bunch of weight, that's a problem.
So to see why this causes a problem, we need to talk about cache lines, which on this chip
are 64 bytes wide. So that means that when caches talk to each other, they do so in units of
64 bytes. So to make the example kind of concrete, if we should look at heart number 3, number 4,
as they write to their output c, matrix in, heart 3 writes that bit, heart 4 writes that bit,
but they're both writing to their own local L1, D cache, and then at some point later, the L1 will
flush the entire 64 byte line rather than just a little bit the core wrote, and then one of
those two what write write will over write the other, and the first one's write will get lost,
and the answer's wrong, and you're in a more depend, which is bad. But thankfully, we can choose
on an instruction write instruction, basis which cache to talk to. So if I use standard of this
client, memory instructions, then my cores will talk to the L1, D cache, that is going to be a
per core or per heart cache. I can say to use custom L instructions to talk to the L2 cache,
nearest the issuing core, or I could use custom G instructions, which look more complex,
because I now could have slightly more hours, but if you use them, they actually make things
quite simple. They will give you the appearance of a current memory, because they will root to
a cache based on the address rather than based on the core that's making the request, but this
will come at a latency and throughput cost, so you know, trade us. And as I'm using a standard
risk file compiler, meant for like CPUs, I get a standard risk file, memory instructions,
thing is, so all my writes go to L1. So I could fix things like using G instructions, but for the
particular access pattern you see, L is also fine, so now I can make that change. It's a compiler
and language could have better understood the harder, maybe I would like annotate every point
type with what kind of memory to use, but I've got to make use of the compiler that I have,
so that's the code that I have, have changed. And now both versions are correct, and the first
of the two runs, certainly, if 3.4, give it up for a second, which is still, you know, slow,
but getting a little bit faster. And then the next easy way is to use all two thousand tasks,
rather than only the first 512, which gets us out of the I loop, fully polarized, and then each
heart is one quarter of the inner drive, all the changes are in that ping part there,
the rest of the code is as before, and using four times as many hearts, give me roughly a four
X speed up scenario. Now at 13.1, click pause for a second, which I said, getting a little bit faster,
bit by bit. And then the final easy win is to use all eight vector lanes per heart, and at this
point I'm going to pretend that the second player is a little bit better than it actually is, and
I can kind of go eight wide SIMD, just like changing FB32 to FB32 X8, and fix the up scenario
a few other things. So as I said, this isn't quite possible, but if there are any LVM hackers
that want a little project, please make this slide possible. In the meantime, I will assemble
this code by hand, because I like to write assembly code. And if I do that, then I'll run another 104
give-ups a second, which you know, isn't as bad, and you know, okay, okay. So we've got as parallel
as the hardware will allow us to go, which means it means you act too, which I'm fairly
called being clever. If we've got enough parallel as we can go, so now we have to actually think.
So again, this is the memory hierarchy diagram we saw previously, same diagram as before.
You'll know I've put a star on the L2 cache, the L3 cache and L2 SCP, and that's because all three
of those come from the same pool of SRAM. And you can do a partition that pool of SRAM however you
want to, but these are the kind of default split between the three or that SRAM. And of these three,
L2 STP is really the interesting one here, because any SRAM aside, to it behaves like normal
addressful RAM, but has the latency of the L2 cache, which is nice, like it behaves like memory,
but it's fast, like cache. So every core will have a nearest L2 STP, just because it has a nearest
L2 cache, but any core can access any L2 STP, it'll just require a longer path around the
notch, so get there, suddenly no slower, but it'll still work. And I'm not using that, but that
functionality, but it is kind of nice to have it. So up until now, my matrices of NB were over in DRAM,
which is quite a long way away from the cores. So I can just cheat and put those NB over in every L2
STP. Obviously, if you were doing things properly, you need to stream your data in, as it needs
depth, but I can just cheat and put it all out of to start with, and call it good. And if I do that,
I get up to two hundred and twelve, if it pops for a second, but again, it is a little bit faster.
So at this point, now I'll look at this inner loop of the mammal, and I've gone back one step to
the pre-simdy code, just to keep things a little bit simpler. And this is the rest five assembly
corresponding to that loop. With comments added, if there's a few first language, it is not
assembly, just for the most of you. But if you do have a keynote for assembly, this line will jump out
as being coming out of nowhere and having no apparent purpose. So I won't have dwell on this guy slightly.
So the both I'm using is a zero silicon, and any harder people in the room will be like, yeah,
zero zero silicon always has a few bugs net that require a software worker around them.
And a software people in the room, you're like, what's a zero silicon? And it's basically,
it's the first version that you send to TSMT to be printed. It's like version zero or
your software. It has some bugs. You hopefully it doesn't have any bugs, but there are always a few bugs.
And yeah, this is a workaround for one of those bugs. So now they're over the shock of seeing
some assembly codes. This is back to the SIMD version. And again, it's the inner loop of the
SIMD version that is of interest to me. And again, this is the assembly for that, and notice it's
very SIMD that it's a family you just saw. Just for the these weird dot PS instructions,
rather than 100 ones, otherwise it's all the same. Now it does matter which version we look at
only 40% of the instructions are FMAs, which is rather too low. And we've got a lot of other
instructions, which is kind of too high. And you know, people don't just don't tell the whole story,
but like this number is too low, that number is too high, and we should fix this. And one way
of fixing this is to do more work for outer loop iteration. And it so happens, you know, pulling
a rabbit out of my head here, that doing four times as much work for outer loop iteration is the real
sweet spot. So rather than doing a one by eight piece of C, we double up on both axes and we're
two by 16 piece of C. And then the code now looks like this, you can see this goes pattern
we're doing like four things, we've got like four initializes of X, four because those may,
throughout the eight loads can be four lines of matmoles, four lines of stores. And if we do this,
then we now up to 32% of the instructions are FMAs, and only 30% are others, which, you know,
is going in the right direction. And in terms of performance, we're now above one total per second.
So, you know, life is pretty good. And then we can pull the same trick again. Again, we can
double up on both axes for outer loop iteration. And I'm not going to show you the code for that,
because they're two by 16 and you just fit it on one side, four by 32 is four times longer,
so it certainly went for it on one side. You know, the the percentages I can show you,
and the end things are going in the right direction. And performance gets me at just under three
top of the second, which you know, it's starting to be like, okay, still a lot so I don't know,
like, but you know, kind of, okay. So to get faster, I'd like to do a 100% of my instructions to be
FMAs, of course, will still need like some loads and some other things, I'm going to like name,
four partners at 20% and five percent. I wish when you're going to say that this adds up to 125%
which is clearly not possible. But just to quite as some creative thinking, not cheating,
just creative thinking. So to hit 125% all I need to do is to on average execute 1.25
instructions per core, per clock cycle, which means at least occasionally I need to execute two
instructions per clock cycle, so that my average can hit like 1.25. So which means it to act three,
which I'm calling magic hardware. You know, we've been as clever as we can,
now we have to rely on the hardware being magical. So this is another diagram,
I'm going to show you the ETFs like one, with the main reason in the bottom last being a single
minion CPU core. You'll note, I've found a scalar, you know, inventing it as separate things.
So you might hope like you get like two instructions per cycle by using them both on every cycle.
But I can't do that through the risk file, front end, because it could only issue one
instruction per cycle. There's not enough like front end to two. So that's not going to work.
But as I said, there is some magical hardware here which we could use to cheat. So,
same diagram, but with the additional settings, the hardware that is present on this chip,
which I've drawn is three additional green boxes because just as with the risk file front end,
each of these can issue one instruction per cycle. So now, as I have a fourth of them,
I can enter you hit four instructions per per clock and get my average up and call things good.
Now, we can kind of quickly go to what each of these do. So you can use TensorFlow to load
to load data into L2 SCP, which if you're doing profumant pop-up map models, you would be using it.
But now, I'm not drawing the complete else, so I'm skipping on him.
TensorFlow one load, again, can only load data, but its destination is rather more fun.
So I learned that we've lost a chunk of our L1 decash and turned it into this thing called L1 SCP,
which is a versus file for reusing the results of TensorFlow on loads.
And then there's also this direct pass, in TensorFlow one load to TensorFlow compute.
So for any loads that you only use of the result of once, they can kind of go straight to compute.
This is called A type loads and B type loads, and I put A and B on the diagram, because it's going to be
important in just a bit. And then turns into compute is really like what I'm here to properly use.
I've linked it to that a right unit to show that it can use the right unit at the same time as the
front end using the scale unit, so I can use both of those in parallel. And I've labelled C for the link
across to the vector storage. You also know that all my models have been doing like A times B
equals C, so hopefully you can see where I'm going here, which is that this, my green box,
will take an A type load, multiply it with a B type load, and there's all because they've
had C to be sort in hard zeros, vectors. And then all we need to do is to use this extra
module hardware, and we can make things go faster. And as it is going to take the three sides of code,
this is side one, doing the kind of M-heart's idea stuff. So the only thing we're thinking is that
only heart zero of each core can use tensor compute. So I'm making heart one just to finish immediately
and we'll create it reading the tensor L2 loads, but I'm not drawing the complete L. This is
inside two of three, the main loop of the map model. And you're going to say this reads like
complete mud. Yes, it does read like complete mud. There's not yet particularly nice in
a syntax for making use of these tensor units. So like you can, but it's not pretty. So just like
take on face to like this does work, you know, it's like ugly as anything. Really you can grab
the harder I manuals to cross reference for this to figure out what it is doing. And then we have
side three of three for this slightly faster map model, just doing a store of C of off to memory.
Again, clear as mud, I'm sure you understand everything here. But so it is fast code. It will
run at now 7 to 12 lbs per second, which is starting to approach my goal here. Like I'm aiming
for 10, I've got 7 kb worth. So the code I just showed you is the code is like this left hand
corner. There's side one of calculating i and j in the M-heart idea stuff. And then side two
was just like repeating loop of things and loads and some FMAs. And then side three was doing
a store. Now, it is useful to re-things slightly and peel off the first load and peel off the
last FMA, which gets us to this middle corner. And we have now grouped FMA and then load rather
than load and then FMA. And notice it like this doesn't yet look like a useful change to a
two half made. But if you do this, then this repeating group, now the FMA and the load don't
depend on each other. So I can do them in parallel, which would give me to slide to the white
hand column here, wondering if my FMA is my loads in parallel. And then look at the clear as mud
code for the white hand column. And again, you're going to be like yep, side of code, clear as mud.
So again, this is going to take three sides of code. This is kind of what we saw before,
with the M-heart idea stuff and then the first tens of load. Again, I don't exactly
understand that this, I just want one of you to kind of appreciate that it can be done.
Again, side two or three, it can be done. It's, you know, rearranged some stuff. We've got the
FMA last and the, so the FMA first and the, the, the, the, the last. Again, clear as mud. I'm
sure you get all of that. Slide three is now the final FMA fused with that store.
Wonderful code, moving 50 on. But what I would don't say is that it now runs at just over 10
terms per second. So I think that's close enough to my goal of 10.6. And as I think I'm pretty much
out of time with like two more seconds to go, so an else can draw the rest of this owl. Because I think
I'm done. Any questions? Everybody enters to everything. You explained really well on
one question. And could you say a few words about what you actually based these, these
experiments on, did you manage to get, you know, like some evolved word from Esperanto on eBay
or is there something new from Enico? Can I answer? We gave you one. Actually, if you're nice
to the community and ask us, we have a bunch of one that's right now. So basically it's
not that hard. Yeah. So yeah, you can assign an int that allows to get access as well if you
know, can't actually get a card, but they do have some cards. Yeah, we have about 10 machines in the
lab that are available and we have some cards like about maybe 80 of them left that we can
send to people and actually have some chips left. So somebody is interested in designing a board
with that. That's also an option. One additional question. How long did it took for you to
optimize the whole thing like the whole chain of optimizations? I'm not sure. I'm going to ballpark
a week, baby. Yeah, I don't really measure my time. Yeah. And how long did you take to optimize
an anthropic challenge? Two weeks. Number one. Well,ish. So number one. Yeah, one one last question.
Hi, so all these changes that you did were at the assembly level, this is. So the work
now that you are putting is at the compiler, putting these changes into compiler and upstream
or how is it? Is it only at assembly level somebody would get the benefit or you are
actually trying to get everything into your compiler and upstream into LLVM or GCC? How would it be?
So John Lucas is probably more appropriate to answer that one. But LVM is upstream. The GCC port
is slightly further along for historical reasons, but probably less likely to be upstream. So
I think going forward, upstream LVM is where things are headed or you can jump in? Yeah, I mean,
you're unfortunately correct, but we're not giving up on GCC, so actually. But yeah, so LLVM is
upstream, but you can definitely use GCC today if you only use assembly and line is going to be
fine. The moment you start being a bit too clever with C and using too many floating point
register, you're going to have a bad time. I'm going to call it good for time now.