Can you get off the last talk of the day?
Good afternoon, everybody.
Happy Sunday.
We are finally at the end of Boston, so I'll try not to low you to sleep with the
hardware talk.
Wait, hardware talk.
Or how I manage to sneak a hardware design talk into the HPC room.
So hi, I'm Felix.
Who's this bozo?
I'm a platform engineer at AI Neco, I'm an open source guy, I work on HPC.Social, I'm a
contributor at chips and cheese, we have stickers, they're great, go take them, you
might notice the color scheme on the stickers is a fuchsia and a blue, which is also
the theme of the background, which will hopefully work.
So I've been referred to lovingly as a practical travel maker, so I'm going to try to
not cause too much trouble today.
But the broad topic of what we're looking at me looking at is, what is the base element of
how we do computations?
What is the most basic building block?
And what does it mean to say that a process is accelerated?
When we're talking about hardware acceleration, there's this nebulous concept of, let's
move this slide, what are we discussing?
What defines an accelerator?
What defines the workload that we're trying to accelerate?
What maps that workload be particularly good on a class of accelerator, let alone an
implementation of that concept on an accelerator?
And what is the design space that hardware people talk to software people about?
And how do we go and define the platform that interface and that contract so people can
make realistic usage of that hardware?
What are we not going to be discussing?
I'm not here to tell you that what you bought for your cluster was the wrong choice for
that class of code, that's not what I'm here to say.
I'm not here to tell you your device selection or the device selection of your users isn't
reasonable.
There are plenty of reasons to choose a particular class of accelerator, be that the
interfaces, the ergonomics, the completeness of the software stack, or that it does a very
particular operation that is right for you very, very well.
Classical example of this, everyone hated AMD CPUs back in the bulldozer days.
But what did bulldozer do?
Bulldozer had FMA before anyone on the Intel side did.
So for certain classes of clusters, yeah, my single threaded code is slower, but all of
my FMAs, which for those classes of HPC codes were FMA bound, I'm literally twice as fast
per clock in the same area for less money and the same amount of power.
So implementations matter what we accelerate in our operations matters.
I'm going to contend that everything we use today is an accelerator of some form and we're
going to get to what that means.
But I think you're going to start to understand what we're actually doing here.
So chapter one, the humble, simple CPU, what is the CPU?
The basic building blocks.
A CPU is very simple.
A lot of people here who took one or two electrical engineering courses in school or
who have an interest in this sort of thing will think of a CPU as I have a memory somewhere.
That can be DDR, that can be SRM, it doesn't matter, memory exists.
In that memory is data, I would like to operate on that data.
I need to define a contract for the machine to tell me, for me to tell it, how it operates
on that data.
So you fetch an instruction, that's me telling the hardware what it needs to do.
That hardware needs to interpret that instruction, I decode it, and then I execute it.
You tell me what you want me to do, I figure out how to do it, and then I do it.
And then I give you your results back, that is the interface, that is the question.
And all computers that the baseline are doing this.
This is the only way to implement anything in computer engineering, computer science, and
HPC or anything that is not accelerated is when you have these simple in order fixed function
devices.
The basic building block, we have a register file.
That register file is what we operate on.
We also have an ALU in arithmetic logic unit.
That unit takes the data in the register file, operates on it.
The fuck is an ALU?
Wait, I should probably not do that.
So fundamentally, what are the basic logical operations that anything in everything in
computer science can boil down to?
We need to be able to do logical manipulations, shifts, these sort of ops.
And as we do those, we can actually reconstruct every possible operation that any instruction
or any command could be broken down to those fundamental operations.
But if I had to do shifts and alignment and so on, every time I wanted to do an FMA,
I had a very bad time.
If I went to market with that device, that would also give me a very, very bad time,
because it's going to be slow.
On each instruction I need to check, oh, did it do the correct movement.
Am I aligned?
I need to do the detection for, are my floating point bits aligned?
Am I in the right rounding mode?
Oh, now I need a subroutine that goes through and checks rounding modes, and then changes
the implied state of the rest of the machine.
Having every single portion of that independently and one at a time, that sounds really
slow.
And what do we do when we want something to not be slow, we accelerate it?
And that's the full portion here.
When you can do anything, you want with just integers.
All the math fundamentally breaks down to integers, be they signed or unsigned, one
way or another.
But we don't always want to operate on integers.
Sometimes we do.
Sometimes we don't.
Classes of codes will want to operate on them.
And that's the point.
This isn't easy.
It's also not the European software thing.
This is a different type of easy.
It's not fast.
And we want things to be fast.
We call ourselves hyperformers, computing people after all.
So the first accelerator, if I hadn't laid the foundation easy enough, the first accelerator,
I contend, was the floating point unit.
I'm not going to get the only floating point standard that matters.
I should be simplify for our holy be it's name.
But with that being the case, we're trying to do when we talk about floating point or
other types of operations is I don't want to always be dealing with the minutiae of
the machine.
I want the machine to handle some of that complexity for me.
I want to establish a contract with that machine that so long as I follow the rules
of a format, I follow the rules of the input.
It can make certain classes of operations significantly faster.
At the same time, if I break that contract and give something that's now formed to that
unit, it's going to accelerate garbage, and I'm going to get garbage back, and I will
deserve to get garbage back because I broke the contract.
That's how we operate.
So we trade the generality for fixed function, fixed results.
And as long as the implementer also honors their half of the contract, I get that back.
I can trust the machine to give me what I told it I wanted back.
Is that okay?
Do you care about how the math gets done in the computer for so long as you get the correct
results?
I'm already down five minutes, good Lord.
Of course we do.
We have to care about how the machine operates.
We have to care about what the machine uses back, and we have to care about the performance.
Because the goal is not for me to come here and talk about computer architecture and so
on.
The goal is we have users.
Those users use those machines.
Those users have to operate on the machines, but they don't do it for the sake of doing
it.
They have a goal.
They're doing math, not for the sake of doing math as fun as that is.
They're doing math because they're looking to create a result.
Be that a paper, be that assimilation, be that verifying that a car, when it crumples
on impact, is not going to kill you.
There are honest goals for how we do it.
But the way we approach these problems is we have to give people tooling.
So let's get a little bit more complicated.
Simple three stage.
You fetch, you decode, you execute.
Everything we do is a version of this model.
Just simply how big we allow it to be.
You'll notice I did a little bit to slide a hand.
Now on this little diagram, I added decode, but in the decode fades, now I have two
different options and I have an integer side and I have a floating point side because
I have to.
If I choose to have an accelerator, that accelerator is fixed function.
But the machine has to be general, the machine must be able to operate on everything
I give it.
So that being the case, I have to be able to decode, mocks and send different data around
different devices.
And when that's all in order, that's fine.
But you're not always in order, or at least the machine isn't, it might not be telling
you about that.
So let's get to the not so simple CPU.
The not so basic building blocks.
Super scalar.
What does that mean?
You have scalar units, but you have a superset of them.
It's how I like to think about it.
We have multiple ways of issuing data, of issuing instructions and where they go.
Why do we care, though, and what are the nuances of this machine?
When we think about the basic building blocks, doing a lot of certain functions in hardware,
some are going to take a lot longer than others.
Some require more bits.
Think in FP32 versus an FP64 FMA.
Well, in hardware, in silicon, those required different numbers of gates.
If they require different numbers of gates, and I'm pushing the clock speed up, there
takes time to move electrons, to move current, to move voltage across the machine.
And if the area is bigger, then I have to decide, I'm telling you, as a contract, you
will get the first, the correct answer.
You always get that.
I do not necessarily then have to guarantee when you will get that answer back.
So long as the guarantee is, you will get that answer.
And I do make that promise to you.
If you don't, then it's the halting problem and it's paying.
So not all these instructions take the same amount of time.
So not the not so simple version.
As we go and we build bigger and bigger devices, bigger and bigger CPUs, we've made the
decisions that certain operations must be very fast.
But certain operations must be fast from a concept of latency.
Certain operations must be fast from the context of throughput.
And they don't always have to go together, and they don't all have to happen at the same
time.
It has much like us running HPC centers, running supercomputers, running different workloads.
Not everyone is doing the same math, and it's not always structured to be the same math.
But we're trying to build a machine that can accelerate a large class of problems.
So now the same way that I had fixed function hardware versus general hardware, I have fixed
function and general acceleration.
The general version of the acceleration versus the fixed version of it.
How do I approach that problem?
How do I build hardware and contracts to be able to do that?
And then this is sort of what you look like when you're building a really big CPU these
days.
This is one core.
When you go and you build and you design a single CPU core, that one thread that you see,
they can be doing 8, 10, 12, there's designs trying to go to 16 and 24 in sanity.
But they're parallel decoding all of these instructions, all at once.
And then they look the same way that you would in a compiler and say, what is my dependency
chain?
Because the same way that not everything looks to be happening at the same time, nothing
is happening at the same time.
When I go and I say, I need you to load this, modify it and give it back to me.
I don't know when that memory is going to arrive.
Because on a laptop, my clock speed is changing because I want to save you power so you can
get more work done.
But that means now the clock distance, the effective distance from the core to the memory,
that's changing on me.
So I don't know the fixed time when data is going to arrive.
But I can take advantage of that actually.
Because now on the power side, I know that certain hardware takes longer to wake up.
And I know when I can combine instructions together, I can fuse different dependencies together.
So I can look and say there's a bunch of different vector operations.
And I want to schedule them all, they don't have dependencies.
But I'm going to go wait until I buffer them together.
They cut my vector unit, do all of them at once, then bring it back and shut down the
vector unit.
And now I'm saving you tens of watts all the time.
I'm saving you a ton of power and we're reordering everything at the same time.
So how do you need to think about these CPU cores?
You'll notice on that last slide.
Notice how I have to split the device into two, right?
This is a really, really deep pipeline.
We have front ends and back ends.
They execute different things at different times.
But that still means back to the size of units.
Some units are going to be significantly larger than others.
And we need to move the data.
We take time to do that.
Which means now I have to amortize as much latency as I can, because the user perceives
that it's going to take no time at all and that everything happens in order.
But because of the amount of latency it's going to take to get from the top of the core
all the way to the bottom, now I have to worry about, okay, how can I hide that?
The very classic trick is I try to pre-fetch data and I'm pre-fetching data and I'm putting
that in the branch predictor.
Oh, crazy word there, branch predictor.
So I'm assuming that the machine knows the way I'm going to fork better than I do.
And the truth is, it kind of does.
The basic example of a branch predictor where you get tons of performance and where you can
amortize that latency, amortize those loads.
Is it a loop?
What is a loop?
A loop is saying I'm starting this block of code.
I'm checking a condition.
If that condition hasn't happened, I'll go back to the top and I'll continue.
I'm not going to break out of the loop.
Okay, a very simple mental model then.
How bigger most people's loops?
I'm going to argue that most loops are bigger than two iterations.
And because of that, you can very quickly determine, hereistically.
Oh, because I assume that in the vast majority of cases, I'm not going to break out
of the loop while I'm executing loop iteration one, I can go through in pipeline and do
the fetch for two.
But wait, if I'm already doing that, why don't I pre-fetch data for loop two?
Loop three, loop four.
And now I have this massive branch predicted machine.
But is the code I'm running based on branches?
So now we get into different classes of code, the mental model for how you approach device
selection.
When you're code in the problem and the math and the physics and the chemistry and
the biology, and so many other problems are throughput optimized.
It's the same operation, day in, day out, vertical lanes, no temporal or spatial dependencies.
It's very easy for me to do the mapping of that problem.
To, oh, I don't need all of that crazy branch prediction machinery.
I would rather something that is naively parallel, massively wide, which we're going to get
into in just a second.
I'm going to run out of time and I'm going to be in trouble.
So moving on, the not so simple building box.
When you look at the modern core, not only is it going to memory, but I said pre-fetch.
Pre-fetch means the data needs to be fetched from somewhere and put somewhere else.
But because we're dealing with electrical components, we're dealing with real computers.
The distance from one place to another is going to change.
That distance can change and I don't know where it is.
So when I look at the point of view from the core, my data is in my core or it's a level
above me and a cache.
Why do I cache?
Because I hope that my data will be reused multiple times.
And if that's the case, well, maybe one core needs to reuse data all the time.
Think of constants in the lowest level of loop.
But in other matrices and other problems, you're going to get cases where the core wants
to share that data with other cores.
So it has to have a shared cache.
But now if I'm sharing data, one core can modify something.
And the other core doesn't necessarily know I've modified it.
So now I need to build additional machinery to go and make sure that everything is coherent.
That everyone sees the same place in time, even though we're never at the same place
in time in the machine.
And you do that across levels.
This is just a quick example of what a core looks like at the simplistic level.
All the cores that can be connected in a mesh, and a torus, hypercube, there's 101 different
ways to do this.
And this gets back to when I'm thinking about the use case, what is a CPU determined
for?
A CPU's primary use case is scalar operations that it can be branch predicted.
If that maps really well to your code, then you're in really good shape.
But it turns out that's not always what you're working on.
So let's look at a very simple GPU.
GPU's graphics processing unit, but that's actually very boring.
And that's not what we use them for anymore.
What we use them for is massive vector programs.
When we're distributing tons of data in those examples of horizontal threats, CPU's
we are dealing with cases where data happens in a pipeline within that temporal level,
and you have dependencies within each other.
GPU's are looking at the world of, no, you're just doing the same thing, and that comes
from their origin.
They're looking at pixel programs.
When you think of pixel programs, I'm just doing effectively a x plus b on a single pixel.
I don't need to care what the rest of the screen is doing.
I only need to care about what I'm doing, so I can build very simple, fixed function hardware
for how that's going to operate.
And then, okay, GPU's got more complicated because we started doing very pretty things
like anti-aliasing, anti-aliasing is a form of blurring, well, a blur means I'm doing an
average, I'm doing a local average.
So now I need some form of connection between all the fixed function hardware, but I never
need to worry about the case where the data is global, for the most part, asterix, two
hour talk goes here.
So when you have that type of connection that origin, you're still looking at it as, this
is all vertical and non-dependent data, and I can build hardware for that, and that's
why we do see hardware built for that.
But you can't always trust that that's what your problem space is going to look
to.
And my worry about very high performance branch prediction, where conditions where changes
happen are a problem, or fine, versus the other half of the world, the other extreme
GPU's, where my hardware is so simple, I don't have time to deal with branches.
And my assumption, the contract that I'm building with you, the user of that GPU, is that
you are promising me, that to reach your performance, you will write codes, you will
develop algorithms, such that you minimize the dependencies on different places.
Through put optimized, branchy tree optimized, mental model for how to do that.
This is very simple as a pipeline, you see you have an instruction cache, you're executing
those instructions, and you're also assuming that those instructions are going to be spread
out across the entire machine everywhere.
Great.
Now I have a warp scheduler, and I'm dispatching, same idea as before, with a very simple CPU
core, the warp is just a group of threads together, and I'm saying all those threads
are going to be looking at the same data, that's where the terms SIMT comes from, same
instruction multiple threads, that's how they operate together.
I am assuming that you always have full usage of the full machine, and if you don't, your
performance will be crap, your code will be bad, and you should feel bad.
With that, you have the register file, well it's all local, but because I'm doing these
massive, amortized, wide vectors, it's very easy for me to make a very wide throughput
optimized device.
I'm not worrying about loading a single case, because from the point of view of the thread,
I'm loading a single case, but everything is grouped together, so I can actually turn
that into a very wide load, so now everything comes together at once, and I can pipeline
and match that to the size of the underlying machine.
Similarly, I'm going to run out of time old by a long shot.
So how do you think about CPU cores together versus GPU cores?
When you think of a GPU as being a massive group of connected GPU cores, you're using
those very simple units that we saw at the very beginning, and instead of making the
cores complicated, you're saying, as long as you promise to always give me very simple
code, and a lot of it, I will be able to do that.
But because I don't have the advanced machinery to be looking at different problems all
at once, that also means that when a GPU has to deal with much more complicated machinery,
more complicated controls, flow, especially, it's going to fall flat on its face, and
that's something that you opt into.
What am I accelerating?
I was going to try to do a high level overview of how you do a high performance GPU.
That should have been a 45 minute talk.
We're not going to do that.
I am just going to say at the high level though, when you're looking at the hierarchy,
notice how earlier I was talking about the CPU hierarchy and going through trees.
Well, I'm doing the same thing in a GPU, and yet why does my performance look so different?
It comes out to the assumptions I can make, but why does every hardware designer build
these massive devices with massive amounts of memory, and then these cascading trees
and these cascading hiring hierarchies?
This is a reality of silicon design.
When we go and we implement your chips, we have IO, and we have compute.
These are the two blocks that we effectively have to play with.
But your IO, so that's your memory, your ethernet, your 5O's, your PCI, all of that,
use the perimeter of the chip, and then all the compute, all your ALUs, those SRAMs,
those cashes, those scale with the area of the chip.
You want's here on lithographic design outside of the scope.
What it means though is we start to see similar hierarchies because it's forced upon us
by the reality of the hardware, the reality of the space.
The choice becomes for the same amount of hardware, as we've scaled up hardware, we continue
to have that bandwidth problem, that throughput problem, and the way we design the chip,
has to reflect the realities of the device.
Then choosing that time or choosing the code of the algorithms and so on that you're going
to map to the chip, that's what matters a little bit on the mental side.
I am out of time, we're going to skip FPGA's, I'm going to do this for a second.
I had simple FPGA, that was a lie, no, okay, it is the last thought, I guess.
So what is the machine solving for?
As a program, how do you need to think about those GPU cores?
You need to always be reading into your mental model.
I can do things locally and they're fine, as long as they're within the group.
These days GPUs are organized 95% of them in groups of 32, some GPUs support groups of 64.
This is an implementation question, but the thing I need to be thinking about is, every
time I go to do accesses, every time I go to distribute the data, you'll see groups of
32, 32, 32, why am I doing that, why does that work that way?
There's one, the contract with the hardware is, every time you're going to amortize data,
I'm going to do those in groups of 32, or I'm going to do those in groups of 64.
And that simplifies how the hardware is actually working.
If it simplifies something, then I can throw more hardware, more time, more space, more energy
at the portions of the machine that you actually care about.
Instead of spending time on guarantees that we're not explicit, so you don't as the
user have to intuit how the machine is doing things under the hood.
PGA's, because I put, like an idea, I put that on my foster submission that I would talk
about them, so let's talk about them.
There is no such thing as a simple FPGA, I kind of skipped on that joke earlier, so let's
talk about FPGA as more broadly.
So what is the basic building block of an FPGA?
What is an FPGA?
Actually, that's a more interesting question.
And FPGA, the acronym, because we love those, is a field programmable grid array.
Okay, let's do that backwards.
I have an array, right?
Everyone here knows what an array is.
That array is mapped to a grid, okay.
It's programmable, so you have an array that's, it organizes a grid and it's programmable,
okay, great.
If you think about just those three terms, though, that doesn't sound that interesting,
because you could map that any compute element is going to be a grid of arrays of compute
elements, right?
And obviously, we can program the device.
So what makes FPGA so different and so interesting to so many people?
Well, it's the, it's the field, what it means and what it implies in the contract with
the user is I'm going to give you a massive set of resources and I'm going to make no assumptions
about the type of programs you're going to run.
I'm going to embrace true generality and I'm just going to connect everything together.
And the field portion of that means that in the field, in your data center, in, on your robot,
on your quadcopter, whatever you want, I can change the way devices are connected together,
effectively through moxes, implementation details here, I can change how devices are connected.
That also means though that when I'm designing an algorithm, I have infinite freedom.
I also have infinite complexity, nothing is handled for me.
It also means that as the hardware designer, as I do that contract with you, I can make
no reasonable assumption of what you're working on, which means all the tricks I do in
normal architecture to make the device fast, massive, amortized load stores as part of the
comfortable.
I have to assume that some portions are going to be extremely simple scalar loads, and
some portions are going to be really complicated.
I can make no assumptions of the way the direction everything is going to be.
I can't make an assumption on, are you going to be bigger, little, and all these little
tricks that the hardware people play, we can't use them.
But we wouldn't use them if this entire concept wasn't usable.
So why do we use them and where do they become interesting?
They become interesting when you have a really weird wacky algorithm that represents
a fundamental piece of physics that cannot be cleanly mapped to a GPU because you have massive
dependencies.
But it's also not branchy.
I know the way the data is going to change.
I just can't predict and amortize the way it's going to work, the way it's going to be
used.
And then in this case, the hardware programmer, the software programmer, all in one, the
platform person, you have to worry about how all of these different devices are going to
be connected.
But it's not that simple because one of the things we decided a long time ago was, well,
we don't know how you're going to connect devices together.
We don't know exactly what algorithm you're going to do, but we know you want to do math,
because everything we do is to do the math, just in weird wacky different ways.
So what I can say is, I'll add a little bit of extra complexity and I'll add DSPs, digital
signal processors, an accelerator.
I'll go through and say, broadly speaking, at the scale of the whole device, almost everything
will be look up tables and wires and mixes and so on to connect different portions.
Because what we're worried about is not necessarily doing the math.
But what we're worried about is how do I get the math where it needs to be?
And when I'm doing my math, using tricks like DSPs, these can be much denser.
These are much easier to design and these have a contract, so long as you're doing floats
and not posits and so on, you can go through and pipeline and connect these devices directly
together and flow the data spatially where it needs to be.
But why is everyone doing this?
Because HPC, diversity of workloads, when we're building clusters, we have 101 different
problems on a good day, realistically it's more like a 1,000 and 1, and you look at the
diversity of users.
Well, what do they want?
Are they doing small cases or big cases?
The way I design a small cluster versus a big cluster, what's the topology of the next
work and the cluster that they want to use?
Some people want dragonflies, some people want toruses, some people want hypercubes, factories,
everything in between.
I don't know what you're going to do.
So the NFPJ sounds perfect, I give you the user infinite control, but then I remind you
you have papers to publish and you don't have infinite time.
And I remind you that when I'm looking at the different work, I don't know what's coming.
And I want it's more important to get the work done than it is to get it done perfectly
a lot of the time because the goal is advancing the science, not ending the science.
And we make small iterative approaches to be better.
Some people will scoff at that philosophically and yeah, we can get into it.
So what is the machine solving?
How do you need to think about an FPGA?
How do you need to think about the elements on it?
You're looking at distributed options, distributed pieces of compute, DSPs and connections
and networks that bring the data together.
But because of that extreme generality that's presented to each and every user, I'm going
to take more area, all those wires need to take up space.
I'm going to burn more power.
And it becomes an optimization problem.
Do I have the time and do I have something that is representable in such a way that it
makes more sense to do an implement something on an FPGA, then go and use something bespoke.
At the same time, you run into unique problems, especially saying the defense space, where
I have a very specific application that I need to accelerate.
But I don't have a hundred million dollars to call up Intel or TSMC or global boundaries
or Samsung to go and build a new chip.
So this is a way to implement.
So when across all these classes of problems, you have different hardware that is bespoke
for the use case, different ways of approaching that problem space.
And the truth for what's optional in your use case is somewhere in between.
And yet, our humble CPU, what am I trying to solve for?
I'm trying to solve for making sure that all your users are always taken care of.
And the whole point of this talk is if you know what your users are working on, and you
can map the time that they're going to be doing their computations, the underlying math
to a specific device, you can and you should.
But at the end of the day, you're also going to have a CPU.
You're going to have general use cases that cannot be accelerated.
You're going to have cases where it's going to be a simple pipeline that cannot be predicted.
It has to be scalar.
And for these problems, you're going to always need some sort of base device.
Not every problem can be accelerated, but the level of acceleration changes based on the
contract numbers.
That's the talk.
Thanks.
Thanks a lot, Felix, for deck the key out of the time, but there's no other speakers so
we can take some questions.
Does anyone have questions for Felix?
Out of curiosity, how does hyper-training feed in this picture?
That's an implementation trick.
Yeah, sir.
So where does hyper-threading fit in?
So let's go to our humble CPU.
When I look here, let's look at this one.
Very simple.
What hyper-threading is saying is to the programmer, I can have visibility of two things
happening at once.
But no two things are ever truly happening at once in the same core.
The idea of hyper-threading is when one resources is locked up, the next thread can make
forward progress.
It only works when you're going you have multiple dispatch units, multiple units, multiple
floating point ads and integers and vector units.
So what I'm doing is back to the pre-fetcher discussion.
I'm waiting on data for one thread, while the other threads data may be arriving.
Because that data has arrived, I can make forward progress on that thread.
So you basically, in a lot of cases, there's complexity and visibility problems.
We don't want another spectrum meltdown.
But if you're clever about the way you build the hardware, I basically can double the amount
of threads I have for free, assuming that those threads are not contending for the same
resources all the time.
That's an important nuance here.
Hyper-threading came out in simultaneous multi-processing at 17 and so on.
They all came out of the need of cloud providers, broadly speaking, who are running a lot
of different types of codes.
Web servers, sometimes it's mostly like web servers in different applications, databases,
and so on.
And what that meant is that all these different codes, all these different virtual machines,
the operating system threads, and so on, were all adding contention to different portions
of the core.
They weren't always hammering the exact same unit.
Because if you are hammering the exact same unit, I can do a little bit of extra work,
while waiting for memory to come and go.
But if I'm running a lot of different threads, doing a lot of different things, it's very
easy for me to add an extra thread that handles all of this nonsense for user zero, and
all of this nonsense for user one.
And I can have these two programs effectively going back and forth on the resources, making
sure that you have higher occupancy.
And then it becomes a game of well, I can do more with my core.
When would I personally choose a data flow architecture?
The camera's running.
When I wake up in the morning and I hate myself.
No, I kid, there are very reasonable data flow architectures that exist.
The premise of a data flow architecture is typically what's known as a CGRA, a coarse grain
reconfigurable array.
So we had the idea in FPGA is of mostly I'm looking at where the data is going to be.
And then I go through and I have small DSPs, because it's field programmable.
In this case, we're looking at coarse grain, version of that, so I'm going to have much bigger
DSPs.
And more than anything, I'm looking at how can I flow the different portions of the data
into being reaccumulated.
But notice now, I'm in a weird middle ground.
I'm saying there is a contract, but you don't have to follow it, which also means that
as the programmer and as the user, you don't necessarily know where in the bounds of reality
you can operate.
You don't know exactly where things are going to be at what time and how quickly certain
portions are.
The other thing is, frankly, from a silicon and hardware standpoint.
Now I'm looking at, I don't need as many redundant wires as I do in FPGA, because
in FPGA is like, go and do whatever you want.
A CGRA says, I need to double triple quadruple, maybe 10x the amount of wires I would need
for a day sick design, but I'm only ever going to be using one of those wires, one of
those connections.
So now I've got a lot of extra redundant hardware that I can't use and I can't make
use of.
Those connections aren't being used in the same way that on your motherboard, if not using
all of your ethernet ports and you're leaving them unpopulated or you don't populate
all the dims in your server.
You're missing it on bandwidth and you can't do anything with that, but you still page
for it.
Right?
There's no value there.
Now, at the same point, there's an argument.
Nexilicon is a current architecture in the space that's doing something really interesting
with a CGRA, where they're saying you'll have forward programmability.
The data must always flow in the same direction as a CGRA.
There's a, I can't believe it's not a CGRA.
And that allows me to know at least the directionality when I'm doing my designs.
So I don't have to do copies back and forth.
My designs can be forward feeding instead of rear feeding except for like a backup connection
and I can achieve higher useability that way.
But I'm not solving that problem and it becomes a portus for courses.
There is no correct accelerator.
There is no correct design.
There are use cases in workloads and mapping of those workloads too hardware and if I can get
that hardware and make my users happy, then use that.
Right?
Now, GPUs are at a point where they're finally programmable for general purpose users.
For the longest time, like how many people here remember to do physics for you must draw a triangle.
Right?
That's how we did math on GPUs for the longest time.
That's thankfully no longer the case.
And now we fix those interfaces.
And we express that contract and we refine that contract such that users could make productive
use and get on with solving their actual problems.
Is that the way that we want to move forward to basically have accelerated this for every single
thing that's our program or the company.
But then users on your problems, eventually to make it for it.
Yeah, that's going to be a long way to repeat.
So the question is if everything is an accelerator, how do I present a cohesive front
to my users so that they can make productive use of different types of accelerators?
Resil.
My answer to that is you define the contract.
And I mean that very, very, like I know it's broken record at this point, but you have
to define the contract.
If I tell you that you want to use these, I need to present, I'm going to go off script.
What are the things that's important here is in programming language design, for example.
If you want users to be able to use something, you need to inform them on how they need
to approach that usage.
It's not very useful for Ikea to send me flat pack furniture if I have no concept of
a screwdriver.
But as a person that lives in this world, I know the concept of a screwdriver.
And I have instructions that say this is how you use it to put it together to do something
useful.
However, if I were to just get a bunch of two by force and I said, I wanted someone to build
me a boat that's not exactly useful.
So then the question becomes, okay, I have different tiers of users and different types of
users.
And how do I want to express that complexity?
Because this is a talk I used to give called meeting users where they're at.
Every type of user has a different goal.
I'm an assembly and hardware person.
I live at that interface.
That's where I'm interested, which means I want to define really clean interfaces that
are very ergonomic.
I want to make sure that the first-year master's student who's never even heard of
MPI, let alone open MP and so on can make productive uses of hardware.
At the same time, I need to go and talk to my colleagues and say, well, how wide is that
vector load store really need to be?
And these are different worlds.
So when you're defining the interface, the contract between levels becomes very important.
How do I express it?
And the unfortunate reality is that those contracts break.
Contract here is analogous to a, to a lot of things, to an abstraction.
Because what the abstraction tells you is, I'm going to deal with this for you.
So long as you give me well-formed data, but well-formed does not mean, it means you
are guaranteed how you get the answer at the end.
I will implement exactly what you give me.
It does not mean I will give you the perfect optimal implementation no matter what you
give me and you don't have to think anymore.
It's also why things like, oh, I'm going to jump from Python directly to the size of an
F of the FPU on a device and get infinite performance, they're not going to work in current
problems.
You'll see high-level languages, high-level synthesis, especially where they say any grad student
can give me MATLAB and I will go and program a 10,000 FPGA cluster and I will get 99.99
performance.
No, that's just not how it can happen.
Like, oh, power to you, go try, let me know when you're done.
So that being the case when we're defining those interfaces, keeping a contract between levels
of how do I need to think about and react to it, is how I think about it.
Because yes, the hardware is always heterogeneous.
Some portions are always going to be accelerated, but you don't need to know that.
You need to know I'm going to get the correct answer and if I give it to you in this
particular way, in this particular consistent approach, I will get more out of the hardware.
And then you talk to, well, I'm in assembly now, I'm in a compiler IR, I'm in MLIR, I'm in a
language above that, at different levels of semantics and I can express different things.
It's also why we have different levels of programming languages.
We have your season, your four trends, for people with taste.
And then we go above that and implement different ideas and different concepts, so I'm
getting this some side eye.
We should wrap up here, so we don't get in trouble with the closer people who want that.
All right.
But, actually, it doesn't end here.
They can stick around each piece of social, it's a website, the weekly huddles, like
every week, Felix is often there, so you want to hear him.
Brandt.
I think he's going to connect online as well, but not only with Felix, but everybody
Yeah, so that's actually just before the class, one of the great things about hpc.socials.
We have everything from the people that design the super computers for Azure to first
your undergrad.
It's an open form for anyone and everyone.
The SPAC people are there, the easy build people are there, LVM, GCC people, people that
work on the standards committees for how we actually implement languages, how we implement
MPI.
What are the interfaces between MPI?
Like, what's the ABI standard that, well, what do you do?
Yeah.
Thank you.
There's obviously try amplposts for Felix a vote next.
Count Hpc.