Okay, we're ready to let Felix start his job.
We're using of the 20 of the 36 day road trip.
Coming from Canada to Europe back to the U.S.
And I'm back, Bob.
And he has 47 slides.
Go ahead, Bob.
Everybody, I'm Felix.
We're going to be talking about floating point.
We're going to be skipping some of the nonsense of floating point.
I'm talking a little bit of history.
A little bit of where we got our air today.
Why is the decision that we made when we built hardware?
And when we built hardware for the software hardware contract of how we do math.
And as you can see, we've been computing in general.
So who's this bozo?
I'm Felix.
I'm the need to see it in the air over at 10 o'clock.
We built hardware.
We've opened the source all of our software.
And then I'd like to make this very clear.
Because they're mine.
They're all mine.
They're not anyone else who's mine.
They're long to me.
Don't happen.
So what is floating point?
Very quick version.
You can leave in three minutes.
Both of the point is just a day in the lab.
All we're trying to do.
Sometimes the interviews are great.
The interviews are great.
Most of the time.
Sometimes you have a decimal point.
Because we're a little cool.
And people like to think in.
Fractions are great.
Floating point allows us to do that in a standard way.
There we go.
Yeah.
For the stream, I'm feeling about some HPC.
Those are going to talk about floating.
All kind of.
So, I'd like to point out that little asterisk.
Floating point has some nonsense in it.
This is ominous.
There's a little devil there for a good reason.
But fundamentally, it's simple.
It's, we have a power 2 format.
We have a sign that gets surprised.
Things are positive.
Sometimes things are negative.
You have an order of magnitude.
So scaling up.
And then you have a precision within that order of magnitude.
What that allows you to do is.
And we're going to talk about the fixed point.
A little bit.
I can say, well, I'm this hard, far away from zero.
And then within that range above zero,
I am within this sub range.
But that allows you to do is that you effectively get to scale.
You know, a precision need at all orders of magnitude.
That also works for the opposite when you're going to float.
Or within the interval of zero to one.
You can flip that.
This is me trying again.
Hopefully, I'm.
A lot of all.
Okay.
Well, we got there in the end.
So, that's the short version.
You can all leave now.
There's no nonsense after this.
We're all in good shape.
Except for this part.
The medium length version,
because I was told I have 20 minutes of which I've already used three.
So, why do we care?
The energy isn't fixed point.
So, map is hard.
At least for me.
You guys are smarter than me.
So, probably you guys have less trouble with it.
But fundamentally, making sure you get the right answer is even harder.
And floating point is a way where when we're using computers,
computers are a fixed finite resource.
The way we do bits, the way we do encoding,
is a fixed resource.
So, one of the things that we have to do is decide,
this is how many bits I have.
Do I want to represent numbers up to 20 billion?
Or do I want to be able to represent really tiny numbers
at fractions of millimeters and the density of the nanometers?
Right?
So, the obvious thing is when we're counting on our hands
and we're in elementary school, then integers are totally fine.
But then, let's say I have a number where I have 0.51 of something.
All at millimeters, call integers, whatever you're into.
What happens if you print that value?
Right?
If you print that as an integer, well, you get zero.
Because integers truncate to zero.
Right? You cannot have a decimal point as part of an integer.
Okay, great.
How do I fix that problem?
If they can't store anything past the point?
Well, my domain is a meter,
but sometimes needs the way down to millimeters.
I can move the point in my data, in my format.
So, we'll say, if I need to represent something that's 1,000
of something, then 2 to the 10 is 1024.
So, then I can represent up to 1024 of something below the point.
And then that leaves me with, say, 22 to the 20,
I wrote it down to the 21 amount of states above zero.
And that allows me to scale very precisely within that range.
And if we call this format a fixed point format,
or an fxp, and you print that, what do you get?
Well, you get 0.509 on and on and on.
Right?
So, it's not the correct answer, but it's a lot closer.
And it's better than printing 0 when my value is 0.51.
So, you're getting closer to what you actually want,
and what you actually care about.
And it is close.
And then, for the nerds that want to verify my math,
that's the fixed point pattern.
It's someone always asks.
Now, it's only 0.04% off.
And that's pretty close, and that's pretty good.
But notice that the number I picked, 0.51,
is awfully close to 0 and 1.
So, within that range, when you do a tapered value
format, a fixed point format,
representing values that are very close to other values,
the correct powers of 2, both positive and negative.
Exponents is really easy.
But as you diverge from that, if you try to do,
like an inverse of a prime, you end up being very far away
from those values.
And the amount of distance between those values
ends up growing larger and larger.
So, while numbers very close are about 0.04% away,
as you go further and further, you can easily run
into issues where A's and numbers non-represented,
or B, you're actually say,
3, 5, 10% off from your values.
And if this is an HPC and data science,
I don't know about if every time I read
write a data point.
If suddenly it's 10% off every time I do a time
step, it's a bad idea to have a bad time.
Let's not do that.
So, in the beginning, there was base 10.
It looked like that machine.
This is an idea in 650.
Yeah, it's really cool for it's time.
But one of the problems you have when you're trying
to do a base 10 system with transistors
is transistors are finally base 2.
So, if I want to be able to represent up to 10,
in my encoding, and I want to do it as a fixed format
instead of a partial format, you end up having
to need the amount of bits to represent 16,
which means, hey, now suddenly,
six of my 16 states are actually just wasted garbage.
So, I'm incredibly inefficient from the geth go.
So, that's not good for power, it's not good for
density on a chip, and all the rounds is at that time.
And it's also, when the computers are
doing all of their encoding, all of their math,
fundamentally with base 2,
to represent everything in base 10 means
I'm actually adding overhead every time.
So, I'm better off realizing the computer works this way.
This is how I encode data.
Instead, I can move on and move forward and just do something
at the end where I just print and decimal.
The C standard library is there.
It's a good thing.
It'll help you out.
That's the part that matters.
Avoid decimal times.
But, for the time, that's what we have.
See, yeah, I'm back in good time.
I'm 12 minutes, and I'm less than 50 minutes in,
and I'm 19 slides in, all as well in the world.
So, on to binary floating point.
IEEE 754 is the boat.
It is the greatest part I've ever created,
and I will fight you about that.
What it does is it finally gave us a single
frame of reference.
Everything is IEEE 754.
You know that when I'm running in an arm,
power PC, a VACS, an IBM, an X86 machine.
If I go and I tread my language and my standard
onto IEEE 754 conformance,
the language, the hardware software contract,
the guarantees that I will do this.
This will be when I say a float or a double in C,
or something else.
You guarantee that you are going to do math the same way
every time.
And that's one of the first few things that really
unlock the concept of portability within the
wide world of HPC, is that we could trust
our computations to get the same answer every time.
And what's again, if we're doing science,
that's a fundamentally important and
fundamental building block of it.
So, how did we get there, though,
is that the standard, the Bible,
did not come out of nowhere.
It came out of Intel.
Good friends, loving the bits,
they're having a bad time right now.
We'll be that where it is.
But in the 1980s, Intel had their X86 system,
specifically it was the 8086.
Great chip for the time, little 8 and
slash 16 bits depending on how you look at it,
little micro processor, and work really well.
But once again, sometimes you want to do math
and sometimes that math needs decimal points.
So, Intel came out with the 8087 co-processor in 1980,
and it implemented three data types.
FB32, FB64, and FP80.
FP, in this case, obviously, is floating point.
Now, what happened is you didn't,
these were the formats for the data in memory.
But when you went to actually do a computation,
the computation actually did FP32 data,
extended to FP80 for all of my entries.
Do all the math in FP80, then convert back to FP32.
Yes, that is this curs as you think it is.
Yes, that was horrible.
But it gave us at least something to work with,
where we had a fixed format that was,
a floating point format that was relatively common,
became ubiquitous through the micro computers,
the items of the era, the personal computers.
And what it meant is people could start to build libraries,
and it was also implemented in such a way that
you could do real math on it.
But if you look at the amount of gates,
the amount of multipliers that you need it in silicon,
to actually be able to build the things,
it was relatively inexpensive to build as well.
Right?
Now we're talking to peers in 1980,
so it's still 200 US dollars in that era.
So that like a thousand bucks today,
oh look, that's a cost of a graphics card floating point works.
But it was there, and everyone could start to build on it.
And that was in 1980.
So from there, we could go on and use part of it
to go and say, this is the standard.
So let's go and build upon that standard,
and make it ubiquitous across all machines.
And this is when IEEE 755 was published,
was in 1985.
And fundamentally what 7554 did is that it took all of the formats,
it took the two formats that were reasonable in the 887,
and brought that to a point where it was codified by a standard
by the anyone he used.
And where that gets even better is these are the two formats.
We had a standard, and then the big point is this last line
down here.
The line which is included.
So the two languages that matter,
probably won't be hugely surprising to this
to room in this audience, is we had ISOC in 1989.
Adopted, IEEE 7554, it was its fundamental underlying
mathematical arithmetic standard.
Adopted 7554 for flowing points.
That means, and then the language of the system,
is everything is C, C is how we define the ADI,
the application binary interface.
It's a hardware software contract.
And then when you're looking at doing math, it's scaled.
That's all 4-tran in a lot of ways.
If you look at the actual way that we do math on Windows,
even Windows, if you call a PyTorch call,
actually uses 4-tran Bloss bindings.
The 4-tran Bloss bindings are relevant to F9,
and F9 uses actually 7554 from 1989 to 1989.
It's all the way down.
These are the standards that allows to do the math across
all of our applications.
Now, 7554, I've talked about.
Let's talk about the weird ones.
Because there are certain domains where working on floating
point, you can do some really interesting work
when you're working on domain-specific problems.
So, RGBPact, RGB for anyone who's done game programmers,
shaders, and so on, will make you think of red green and blue.
One of the problems that a lot of the consoles had,
especially in the late 90s, and the early 2000s was,
how do I do a format that has three different data types in it?
That's not a power 2.
That's really ugly.
That's a pain in the butt to deal with.
But also, I know that I'm going to be
outputting to an 8-bit screen,
and I know that I want to work out powers of 2.
So, what the idea was is,
I'm going to go and do two of my formats.
Two of my entries in that packed format
that are 11 bits.
One of them that is 10 bits,
and because I'm dealing with color,
I know that my color will always be positive numbers.
So, what that allows you to do is you two 11-bit formats,
and you fit in that 32-bit format,
which means that for the memory architecture,
the memory subsystems of consoles,
really effective, very fast.
We're kind of following the history here.
Why do we care?
32-bit intervals are great for memory,
fitting three things in hard,
but it's something you can do,
and there is value of doing that.
And sure it's non-standard,
but you do run into the case where,
if you're designing for a console,
you're like,
idea that code is never going to be needed
to run across multiple different types of device classes.
Especially in the late 90s,
early 2000s,
you were designing for maybe one console,
maybe two in certain areas,
and that was probably an idea in PC,
or it was an early knack-to-top,
or it was the Atari's of the other.
Right?
The other thing is that it enabled the idea
for better or for worse,
of domains-specific data types.
And enabling certain industries to say,
this format is great across everyone
when I need to be portable.
Simultaneously,
when I'm in other domains,
when there is a domains-specific reason
to do a custom format,
there is a time in place for it.
Just be careful,
and when you're doing it,
make sure that you specify it.
Because the last thing you want to do is
go and try to use this thing
that, oh, this looks really cool for my build main,
but it's not probably specified,
which in turn means that I can't use it,
the behavior will change across my machine.
And there's one thing that you cannot get
wrong if you're designing hardware and designing APIs,
and that is that you cannot break the trust of the user.
So making sure that if you're going to do something
weird and specific,
make sure you specify it.
So, there are two slides.
I'm flying.
Back to greatness.
I believe I truly still got it.
Still to go.
Still the greatest of all time.
And what happened was we had a lot of formats,
but IEEE 754 was from the 80s.
And we had use cases where, for example,
if you look at mobile phones,
there were times where we needed lower precision
data types for our phones,
because we needed to lower power,
but we still wanted to be able to do the math properly.
Simultaneously,
you talk to say the new physicists,
the particle physicists,
some of the cosmologists,
and they're like FB64 is nowhere near
precise enough of what I need.
So, even if you're not always going to be
implementing the different types and hardware,
I can go through an in software implement FB128,
which means now it's portable.
If you dealt with GCC in the past,
you might be familiar with live GCC quad.
That's a library that's specifically
is a software flow implementation that is fully compliant,
and that works on all GCC compiler implementations.
And also works on your other devices,
such as your GPs that they're running on GCC,
LVM, has something equivalent.
And same for FB126.
Broadly,
you can assume that almost every device
will always support FB32.
Most support FB64,
and anything that's new will support FB16.
FB128 only existed in two devices
that I'm aware of outside of FBGAs.
Those were the NEC vector engines,
as well as the IDM power nine machines.
Everyone else, so you can count on FB32
and FB64 being reasonable.
The other thing is that we got this really new
cool mode called FMA.
FMA is stands for Fused Multiply Add.
So one of the things you run into is
we don't actually,
the reason for the amount of precision
that we have is not because every
we're running against the boundary of
rounding at every single equation.
It's more so when you're running
big HPC workloads, big simulations,
you're actually iterating on the same data point.
Hundreds of thousands of times.
And it's that the air accumulates
every time you go and you touch a piece of data.
So it's not that every single time
step needs more than FB32.
It's that at the accumulation of your simulation,
you will no longer be able to converge for certain
types of simulations above FB32.
You'll see this, for example,
if you're doing computational fluid dynamics
and you're looking at the transsonic regime,
or you have a really weird boundary conditions
of force hitting a mock value.
I digress.
So the point of FMA was,
normally I go through,
and the classic way of doing it without FMA
is let's say I'm doing A is equal to A times X plus B.
What happens is I go and I do A times X,
I do a rounding step.
That's fully compliant.
Then I have to go do my ad.
Then I round again.
The idea of FMA is, hey,
I'm already in my vector register.
I'm already in my floating point, you know.
It'd be really good if I didn't have to round
between because then I could pipeline it
when I'm designing hardware.
I could do some really cool stuff.
But if you do that, then use broken compliance.
And we ran into a situation where the hardware
was working faster than the standards
and faster than software could.
So we had to go through
and standardize this.
And that's what we did in 2008.
We had electrically set by 4.
I'm only going to lightly make fun of Google here.
Google came out with a type called
a brain float 16, which is an AI specific data type.
Essentially it's a truncated FB32.
So IEEE has its own FB64 map.
I have to show the goose.
Because one of the things you run into you very quickly
is when you have all this massive diversity of data types,
all these different ways of doing the math.
Everyone says, oh, I've got 50 key fonts.
I've got 200 tariff fonts.
I've got half an X of font, you know, 60-U machine.
Wow, that sounds great.
And then you realize, well, no,
it's actually a two-bit floating point type.
That is in no way reasonable or comparable
to the other type.
And that's why you'll see a lot of HPC people
get really mad whenever new hardware comes out of them.
Well, no, it's not that great type of flop.
What are you doing?
Stop it.
I'm going to hit you on the nails with the ruler.
So anyway, the idea there was to truncate down
of, I only need these bits, these lower bits.
Because those are important for AI.
When you're looking at AI workloads,
they care a lot of it orders of magnitude.
They don't care as much about precision.
So the other thing that comes with that, though,
is there's a few benefits.
You don't care about the precision all that much.
Prefer the orders of magnitude.
It's always a trade-off.
It's always a balance point.
Multiplyers, which is actually what you use
if you could go back to an electrical engineering design course.
When you're looking at the amount of precision in a format,
that's actually defined by the,
those use multipliers instead of just adders when you're designing them.
And the thing is, when you double the precision,
you actually go up by a factor four on the amount of multipliers you need.
And multipliers are already significantly more expensive
and silicon than adders.
So if I have a type that's more relevant for my domain,
specifically, then I can go through and be more precise for my domain.
And also use a lot less power.
They'd a lot more multipliers, more floating point units,
within my chip, which means that for that domain,
I can drive a lot more performance.
TF32, or why not?
This was an Nvidia specific format,
which I get how they got there, but it's a little bit silly.
So it's called TensorFlow 32.
It has 19 bits.
Interpiece.
But the idea there was, if I'm designing all of these gates for brainflow 16,
simultaneously I'm supporting all of these gates for it.
I triply FP16.
Well, I can just take the long part of one,
the long part of the other, smash them together.
Right?
And what that does is compare to, say, a full FP32.
It's not as good as FP32, but it's better than VF16 and FP16.
And because I'm already going to be supporting both of those data types,
it means that I effectively get TF32 for free.
And once again, it means I can fit a lot of them more of them in hardware,
but both Google and Nvidia have decided that they don't want to have
an official specification of how those data types work.
So if you look at each generation,
TF32 on hopper is different than on ampere,
which is different than on blackwell.
And it's a pain of the butt.
Good luck with your performance.
Have fun.
Yeah.
I spoke about mild players that was more there.
I'll add this part, only minor headaches.
It's why I'm growing my hair out so that it's easier to rip out
when I get angry.
It's a different story.
FP8 is mostly nonsense.
There's 17 current FP8 types.
Some of them are a log.
Some of them have infinity.
Most of them don't.
It's a little bit silly.
There's two of them that matter.
You can ignore them.
They're the OCP ones.
Everyone else is deprecating their support.
And then this was supposed to be a question period.
But in case I don't get to show the slide,
I didn't want to show this one.
It's very quickly.
Questions.
Please be nice.
Thank you.
So different stuff.
And the concept of vendors have decided that sometimes I could be
set that 7.5 words don't set it more than only hands.
So like, give your if you do something like that.
That's finally those thumbs.
Then you get that.
You better close the homies.
So it's how we get to know the stuff.
So the question is fundamentally from the vendor side.
You'll see a lot of implementations that are close to IEEE 754 compliant.
But are not fully compliant.
Compliant matters to a lot of people.
So how do we make sure that vendors are honest about how they implement things?
And the answer is it's not easy.
I am a vendor and I'm fighting for conformance.
But one of the things you quickly run into when you're doing silicon design
and you're down at the big gate level and your libraries is doing things like
the IEEE 754 compliant and 4 trans slash C compliant addition means that I act
to do everything ordered.
So that means that I'm actually linear steps for all that ordering.
Whereas in hardware, you can do different tricks where I can say turn it into a bunch of parallel trees.
And the thing is that parallel tree is a lot to events instead of just something.
I probably can set that exponent.
Which means it's a lot faster.
I only want to do that when I absolutely have to.
So what we typically do is you'll break down the vector unit portion of it.
You'll do compliant on there and you'll have it as an opt-in mode.
There's no good answer there.
It's a given day.
Some of the pilots offer the option to drop activity by a certain type of number.
Do you have to operate and pilot that?
Yeah, I've hunted down most of the compiler engineers that enable that.
They've disappeared off the face there.
I'm covering for you.
Oh sorry.
Sorry.
The question was definitely not a joke.
Not definitely not bait for yours truly.
A lot of compilers allow you to disable or
innately disable floating point compliance from the get go.
What are my thoughts on that?
My thoughts on that is that it's very silly.
Beyond the joke, it's because once again,
a user should only ever have to opt into something on standards.
The standard is there because it forms the fundamentals of the hardware software
contract that you as a vendor, you as a compiler,
developer, you as a user have all together to prove that yes,
I can trust that my math is going to be done the way I expect it to be.
And when you disable that, that is an opt-in feature.
That is fine.
Because that is the user telling everyone else I'm smart.
I know what I'm doing.
I can throw away compliance because this in this specific sub domain of a domain
that is a reasonable thing to do in the name of my end goal.
The problem comes around when you disable it from the get go.
Because then between hardware implementations between different compilers
between different compiler generations,
I no longer can trust on my math will be done correctly.
And for applications that have sensitivity to underlying mathematics,
say a lot of HBC software,
I can no longer trust my results.
Now, at the same time, I kind of want to book in this.
I used to work for the weather service in Canada.
And one of the things that we had is we simultaneously had operational weather models
that those have to be precise though,
but they also have to run like clockwork.
But to validate the models,
one of our problems was the whole bit reproduction problem,
where a numerical result was calculated in the early 70s,
way before 754, before 4x707 was even around.
This was written on punch cards.
And the requirement of the new model is to replicate that exact numerical result.
Now, that is conformance for the sake of conformance that is not reasonable.
And I know that that's very prevalent in the weather world.
I know it's prevalent in some of the banking world,
where you have that exact result problem.
So there's a balance point,
and I don't want to be a zealot for the sake of being a zealot,
and I don't want to be a zealot for the sake of conformance.
It's that zealot not disabled conformance from the user.
Because it's the user that gets to make the decision of knowing
what matters and what doesn't matter.
And it's not reasonable for the hardware to disable that conformance for them.
Okay.
I'll get off my soapbox.
Thanks.
Thank you very much for being here.