You
Thank you. She usually start the reboot but
perhaps, okay, good.
So let's start from the beginning then.
So thank you very much.
So this is about a form of multi-precision or redmetic.
And the problem multi-precision or redmetic
is that it is more cost extensive.
You have a cost overhead.
So that's where the parallel computing comes in.
So multiple double or redmetic allows you
to work more precise, but you still
work with floating point of redmetic.
So we are looking into making the miracle algorithms
give them a more broader range.
Now to make it efficiently and parallel,
I'm trying to vectorize the multiple double redmetic.
And I'm going to explain how you can do that.
There are two types of parallel computations.
So the first one is a high level, very straightforward.
The harder one is actually not really straightforward.
And it will just give some hints to that.
Okay, so I'm teaching also computational geometry this semester.
So the definition of a robust algorithm,
a robust algorithm actually performs in the same way
if you start to tweak the input a little bit.
So typically we work with 64 bit doubles,
but we can extend it.
So the idea to extend 64 bit float,
a redmetic using floating point of redmetic goes back
to the late 60s, early 70s.
When people at some point,
they were still forced to work with 32 bits.
But they could actually figure out how to create more precision.
So I was using the packages QD lip a lot
and there is also the comparative library.
My goal is actually to go to GPU acceleration.
So I'm presenting this on a gaming laptop,
which is actually already quite as powerful as the graphics cards
and the big servers that I was using 10, 12 years ago.
So computers were actually too fast.
For users, it's not a problem.
But if you are a software developer
and you really want to exploit your computer,
it's a big problem.
Okay, what is a multiple double?
It's simply a sequence of several doubles.
So if you work with complex arithmetic,
you have a real part and you have an imaginary part,
two doubles.
Well, double double is actually very much the same.
Now, if your result can be represented exactly in one double,
like you want to compute the number eight here,
then your second double is going to give you
the accuracy at which you compute it.
So with double double, you have about 32 decimal places,
then you double again,
but probably you have 64 decimal places again and again.
So that's the good thing.
So you can still do numerical analysis.
You kind of have an error estimate.
If your result is an exact,
can be exactly represented by 64 double.
The drawback is that you're not going to notice
all that much with double double arithmetic.
Actually, your problems are going to be compute bound.
So your processor is going to work harder.
But when you get to double,
actually, you may want to do some parallels.
And then, of course, you can double it up.
So you see the average number of floating point
operations that you will have to do
to do an addition, a multiplication, and a division.
So here is why I am doing this.
I'm working with Taylor series.
So you may remember Taylor series from Calculus.
So here is a very nice series.
So the exponential starts with the leading coefficient one,
but then the coefficients actually
they go down very, very fast as a factorial.
If you only need eight terms, then double precision will do.
But if you need more, so for example, if you need 32,
theory turns in your Taylor series,
to represent the exponential correctly.
So also having these tiny components,
you actually will need, quote, double arithmetic.
And actually 32 is a good number if you think about the warp size
in a GPU.
So typically, we would like to work with 64 or 128.
OK, so here is the idea.
So what happens is that when you do an operation with a double
double, it's going to do the operation,
and then it's going to renormalize it again.
So if you do a quad double, it's going to also renormalize.
Make sure that all the doubles that represent your quad of double
are actually going to be known overlapping.
So that is going to have a lot of tests ifs and things like that.
The idea is if you vectorize, you actually
would like to postpone the renormalization.
So here is if you want to add a sequence,
so you have 52 bits of precision in your fraction,
you simply cut your 52 in half and you insert 0s.
So these 0s are going to fill up.
So you could also think about if you want
to do exact integer arithmetic, you
would split your 64 bit into 30 bits,
and you would add 0s in upfront.
And these numbers will actually fill up.
OK, so double, double arithmetic is also
referred to to error free operations.
I want to do inner products.
So I want to multiply matrices, for example.
The reference code is going to compute just
a sequence of sums of products.
So for products, if you multiply 2, say 13 bits,
you can have a 16-bit 26-bit number.
So that's why a double double, which consists of a high double
and a low double, it's going to be split into 8 parts.
The 8 parts are all going to have 32 non-zero bits,
and going to be filled up with 0s.
So it sounds very complicated.
So what we're doing is that here with one inner product
and double double arithmetic is going to be replaced
by 8 inner products, but the 8 inner products
are going to happen in double precision arithmetic.
So computers are very good at doing matrix matrix multiplication,
with double precision arithmetic.
Multiple double precision in a product are replaced
by inner products in double arithmetic.
So that's what I mean by factoring, factoring.
So here is the setup.
It's important to still float in point,
and the numbers can still float.
You have to prevent this by kind of re-normalizing
intermittently.
So sometimes people say with a 64-bit double,
you have 53 bits in precision,
because you had that normalized bit.
Now when we split a double in four pieces,
we also have to make sure that these exponents
are actually 0, negative 13, negative 26, negative 39.
OK, so here is the computational result.
And in some sense, this is already a parallel computation,
because inside the computer, there are pipelines happening.
If you do a double precision floating point operation,
there's going to be the denormalization,
making sure that the exponents are the same,
adding the fractions, and then again storing.
So by rewriting so that first big inner product,
as many inner products of doubles,
we actually get a lot more speed up,
and a lot more gradual progression.
So if you see if you go from, and the left column there,
so the ordinary, what I call the ordinary normal,
double double and ticked,
you have a immediately penalty hit of 13,
and then if you go to double, you have again 12 factor.
But if you do it vector, it goes way, way more gradual.
And this was, by the way, done on a very recent compute,
with a recent chip, compiled with a full optimization.
OK, now two levels of parallelism.
So the big point is that going from, say 12 milliseconds to nine,
seconds you're going to notice.
So that's actually the whole big problem
of using multiple precision arithmetic.
Things that start to take milliseconds,
suddenly become seconds, second becomes minutes,
and minutes become hours.
So the nine seconds, you can actually,
if you do this in a very straightforward,
so I had to do 1,000 in a product.
I can actually reduce this with using all the course,
and that CPU, I can reduce it from 300,
from 293 to 300, going from quad-double four times more.
I like speed up, but we also like quality up.
So in some sense, if you use all the course in that computer,
you actually can do the same computation,
four times more accurate within the same time.
OK, so that's the nice type of parallelism here
as a reference.
So this is what I call the work crew model.
You just every threat picks up an in a product,
and they work through a job queue.
And the only thing you need to be careful about
is that when you increment your job counter,
so every threat, when it's done, it asks for the next job,
that they can only do this.
They have to do this in a critical section.
And this can be very nicely implemented.
So I think this is classic.
Ada, OK, now comes the hard part.
So with GPUs, you can actually do the multiple double arithmetic,
that's fine, but then there are the tenser course.
So the computer that I mentioned actually
it has a high-end on pair graphics coordinates.
And what this experiment shows is that these convolutions,
they can actually be reformulated as matrix-matrix products.
So this is what we call data staging.
So we have to make sure that the data is all properly
arranged in vectors, and that we have very carefully
we know where the beginning is, and where the end,
and where are we going to write.
So GPU code is typically very short.
You just smash some kernel, but the preparation
for the code takes a very long time.
And actually what I've presented to you here
is kind of a first step in making the tenser course,
making them useful for higher precision,
multi-matrix multiplications.
Thank you for your attention.
On time.
Thank you.
So questions and answers.
I have gone, you said we were using a 96-core CPU.
Yes.
Then you mentioned that we had only a six-time speed-up or something
like that.
Yeah, so these speed-ups are actually only done on one core.
This is, okay, I'm sorry.
So the question is, I'm using 96 cores, and how come these are
the speed-ups here?
In this slide, I'm actually only using one core, and I'm using the 96 cores
to reduce the 9 seconds over there.
So this is kind of here the highest time, and I want to reduce this,
comparing to the 318.
So that's about with quadruples, about 10 times slower.
What I can do here is if I have 1,000 in a product to do,
and they are distributed among the 9-d6 cores in that computer.
And that reduces the time to 200 to 318, I'm sorry, to 298 milliseconds.
Yeah, so the question is, I'm using some of fours here, have I considered using atomic
variables, and I think I do.
So there are these types that I can use, yes.
Yes?
Yes.
So the question is, are the exponents useful?
So the exponents, so in some sense, so here is how you can see it.
So that's also drawback actually of multiple double arithmetic.
You still remain 11 bits of in your exponent.
So at some point you're going to have overflow or underflow.
So you have to do still numerical analysis.
So the question is if these exponents are useful, here they are not, if you are just normalizing,
because they all stay fixed.
And that's actually what you want.
So in some sense for your numbers, you're only going to use, so here I generated all constant
numbers in a very nice range, but it's actually this exponent that is going to matter.
And all the other numbers they are going to somehow carefully need to be rebalanced.
Yes?
Yes.
So it comes up with that theorem, and that impacts so much of the temperature of the process,
so that it has a better effect of the right, and then put it in like this range, so the
load, I don't think of the fact.
So your question is about the load balancing?
Yes.
When they are currently driving a lot, yes.
When they impact over the temperature and the control by a high temperature of how the
process is going to be.
Yes.
So the question is about the temperature, and when the, so I don't know, so that the machine
starts to get a lot more noisy when I run it, and the fans are really starting to blow
very hard.
So that's how I see that it is working or I hear that it's working.
But yeah, I have no sensors, I mean, there could be sensors there, but I don't know how
the temperature.
Okay.
That the HLFI sort of made a close and the cancels, you know, that to reduce the temperature
because the family is able to add a risk, it's max speed for the heating of the
process, and all this particular process, it's great for this into the ignition of the
temperature.
No, I never used that.
Yeah.
Okay, thank you.
Thank you.