It costs anything from $700,000 to $700 million to make a new chip because you want to make
your code smaller or faster. I actually made the code for a particular chip, 7% smaller
in 24 hours by additional application of benchmarking. And that's why I'm going to tell
you about today. My name is Jeremy Bennett. I'm chief executive of Embecosm. Welcome
to benchmarking in the rest five devroom. I hope at the end of this you'll have a good
insight into how much open source benchmarking technology there is around that you can take
advantage of and how it can help you improve your software, your hardware designs. So first
let's think about benchmarking. How do we benchmark? And for good open source benchmark, the
number of criteria, this is just the main ones. benchmark on real code, not synthetic made up
someone thought this looks like a typical program, programs that real people have run. And make
it open source. There's an idea of code source benchmark, trust me, it benchmarks your program, but
you're not allowed to see it. Doesn't float my boat. Make sure you can benchmark everything you need
to. Certainly speed, code size, embedded people worry more about code size than code speed.
And indeed, we've benchmarked things like energy consumption. It's quite important. How you
write your code can affect how much energy it uses. Keep it simple. How good is this system
program according to your benchmark? Just give a number. Don't give a large paragraph of
explanation. And lastly, the technology moves to keep up with the benchmarks. So change your
benchmarks regularly. Update them regularly so that you don't have a compiler that understands
your benchmarks and just magically does them really well. So I'm going to focus today on two benchmarks.
One is spec CPU, excellent benchmarks. It's marginal coming in here because
spec itself you have to pay for. It's not open source. But all the underlying benchmarks are open source.
So we'll live with what we get from that. And I'm mostly going to talk about the underlying
benchmarks, not the spec CPU infrastructure. And Mbench, Mbench, IoT, which is Mbench, is actually a
family of benchmarks, but it's best known for its very small benchmarks, IoT. We'll use that for
Michael control of course. And this will be, I think the first time I've actually spoken about
Mbench 2.0. It's actually not officially released yet, but it's in final release candidate. So
this is doing the regularly updated. This is the second version of Mbench, IoT. So what
am I going to use as my benchmarking targets? Well, the focus is RIS5, so lots of them.
We heard mentioned in the earlier talk. I was involved with about running these the Core 5,
CB30E40BV2 on FPGA boards. I've used that for a lot of the data here. I've used a high five,
I'm matched as a bigger machine. I've got milk five pioneer, which is a more modern
server-class RIS5 platform. I've also used one of these, but not on a pie. This has got RIS5
vector, version 1.0 on it. And I've used some QAMU, because sometimes you don't have a chip,
you can use QAMU to at least give your rough inside. For arm, this is the go-to board. That's the
just an SD discovery board. It's been around for years. It's cheap. We use it as the baseline
reference platform for Mbench, precisely because anyone can get a hold of it for a few dollars.
I've also used an Apple one, and I've also used QAMU and use a note a bit.
And finally, we have looked at X86 in one or two places. And for that, I've used a thread Ripper.
It's an old thread Ripper. This is about seven years old, I think. But the focus here is not
particularly on X86. If anything, it's RIS5. If anything, RIS5 against armors,
being closest to like to like. You may wonder, what about Drystone and Benchmark?
How Drystone and QAMU? We don't use those. It doesn't meet the criterion of being
a set of programs, and it's not regularly updated. Drystone actually tells you to turn off
all the compiler optimizations, because otherwise it goes away completely in everything runs
internationally fast. Coremark is so well understood by the compilers. It's not meaning for any
more, and it's a thin, thin, thin benchmark. And you can see this in this graph here. This is
looking at the impact of a number of RIS5, Icer extensions, the basic ones, IIMC, adding in a
whole load of string, a bit minute ones, which are the ones on the milk five. And then some of the
Core five special extensions, the multiply QML8, and the extended memory addressing
instructions. And you can see with MBench, rich set of programs, you can see variability across
all of them. But actually Coremark and Drystone are so tied into the baseline. You can't tell
whether those last four extensions are any good at all, because they'll give the same result.
So we're going to ignore them. So first up, I'm going to talk about application cost processes,
so these are the big RV64 machines. I've quoted one of two of them that we're using here.
And we're going to focus on Speck CPU 2016 2017. I know 2017's the up to date one. We have one
customer that was very keen to produce 2006 figures, so I've got a lot of those. So first of all,
let's have a look at architecture. So we thought how does a modern RV64 big machine
application class machine compare again, X86 and against AR64? Now these all run at different
clockers as I've scaled the results according to says they were all at one gigahertz. And what
you can see here is RV64 has still some way to go. I'm not, you know, this is was a relatively early
stage of design, so that RV64 has improved since then. But generally whenever we benchmark RV64,
we find at the application class process that there's a way to go before it catches up with
even X86. And actually the figures for the AR64 and an Apple M1, which is a phenomenal machine,
it's in a different league. So we were a long way before we start being replacing the M1 with a
risk 5. And this talk is going to be mostly just graphs, and I will take questions at the end,
but if I say something particularly agree, just wave your hands around a lot, and I will address the issue.
A GCC versus Ella, which compiler should I use? I run a compiler company, we get asked that question.
The answer is they're both good compilers. The quality of the compiler, you get will not depend
on which of those you choose. It'll depend on the quality of the compiler team that you get to do the
work. A good compiler team can make an X compiler out of work. Both in general, this is just looking
at instruction count. I didn't have the time I did this, I didn't have machines compare. So just
looking at the dynamic instruction count as counted by QANU. In general, LLVM generates more instructions.
That doesn't necessarily mean it's slower, it may be generating longer sequences
that are more efficient. With the peculiar example of 462-lib quantum, there's a reason why that's
not in the newer version of SpexyPU. It is the most unpredictable benchmark. You've made a
slide to change, and it doubles in speed, halves, and speed. It's a bit sensitive to precise
little compiler configuration. But broadly, there's some interesting stuff around 456 hammer,
and I'll come back to that later. But broadly, there's not a huge difference and absent actual hardware
to run these on. We're not sure that there's much to be alarmed about hammer and lip quantum
of the two that interest there. What about LTO and PGO? So link time optimization and profile
guided optimizations, too. The most important compiler optimizations are last few decades. And
what we've got here is we've got a baseline, which is we measured the performance of SpexyPU 2006,
and then this is only speckint. And then we turned on LTO, and we turned on PGO, and we turned on the
both on. And in one or two cases, it does make a big difference, a hammer being an example. But in most
of the cases, it doesn't help, and quite often makes things significantly worse. And that's a bit
worrying, optimizations that make things slower and not really popular. And when you look at X86,
you see X86 does a much better idea. We understand a bit what's going on with PGO. Profile
guided optimisation tries to straighten out code flow so you don't take branches as often by
profiling and working out what the preferred flow is. The problem is with risk five, as the PGO is set
up at the moment, that tends to make things get further apart and you switch from using short
branches to long branches, and then you've actually got more branching activity than you had before.
And we are working slowly on fixes to that. So next thing is risk five vector, and at the time I did this,
we didn't have RVV1 hardware. This is why this board is so important. I had hoped
to actually have hardware figures for this, but I haven't. What this is based on QRMU instruction
counts. And what we found is that other time I did this, for most programs, whether you have vector
instructions, doesn't make much difference. But there are a couple of benchmarks notably the X264
benchmark, which does the sort of operations where vectors really imply. Now, actually we found
the number of instructions carved there. So if you execute these RVV, you execute half as many
instructions as you do if you don't have RVV. The data point I'd hope to have by now, but I don't
have, but I have heard from others who have done this, is this, as of crude, it's too crude.
Vector instructions are so different to scalar instructions. Instruction count really isn't
helping you terribly much. And actually what I've heard people say is when you come to real RVV
architectures on X264, the number instructions is hard, but the execution time is actually twice as long,
because those vector instructions take so long to run. I had hope to have that proved out on the
banana pie, but I haven't managed yet that data yet. So I say the vector, the jury is still out
on RISP 5 vector, but the data we start to get in as the first real chip supply suggests that
thinking about when you generate RISP 5 vector instructions needs to be thought about very carefully.
And sometimes even though it's a smaller number of instructions, it might not be the right
thing to do. So that's if you like the application class processes, and then what I really
want to focus on is microcontrollers. I do more work on microcontrollers and it is fully open source
embedtriety. So one of the questions I wanted to look at was compilers over time. Are the compilers
getting better, and I'm a compiler engineer, benchmarking, yes, it's about your hardware,
but it's about your software tools and your libraries as well. And this is code size,
and for the embedded world, and this is microcontrollers, remember that matters. The good news is
that we GCC, looking back the last seven or eight years, code size has been improving.
If we compare against the obvious competitors, the ARM Cortex, in this case of Cortex M4,
we see that's also got better. But the M-bench score for ARM is improved by about 4%,
whereas the M-bench score for RISP 5 is improved by about 7%. So they are getting closer,
and that's good news. When we look at speed, and of course you want to go faster, that's why
the graph goes up, we see that GCC's generally getting better, though it did get worse at one
stage. You'll notice that GCC 10 was lot worse than GCC 9. But generally it's got better, and over the
last seven or eight years the scores gone from 0.91 to 1.03. So we're looking at about 12, 13% difference.
The thing is so as ARM, and I think if your comparing against ARM, ARM has also been getting
better, though interestingly the RISP 5, and this was done with the CB3040P v2 on a FPGA board with
the normalizing out for differences in clock speed. The RISP 5 is actually slightly higher,
it's slightly better than the ARM Cortex M4, which is a nice result. But there's nothing to be
taken for granted. We do need to keep on developing the compilers if we want those trends to continue.
Comparing compilers, do I use GCC CRL over here, and that's the question I asked with
specs, if you have application class, when you look at this class, you can see here the scores
for all the different marks that make up. This is actually a major one point in order to do this
to the world. And what you see here is that for individual programs, one compiler is better,
or the others compiler. But if you look at the average, there is little in it, as a tiny bit,
I think one's 0.98, the other's 1.01. They're very similar in performance, and LLVM
18 is slightly better than GCC 14. Actually, historically, that's a change recently, because when I
did this a few years ago, GCC was the quicker one. There is something important out here. This is where
benchmarking is more than just looking at the numbers. For compilers engineers, the LLVM engineers
can go and look at where GCC does much better and ask, what have they got that we haven't got?
On the GCC engineers do the same with where LLVM and what we got, are both car pilots improve.
So, actually, this all compare, and we do a lot of this of comparative analysis, try two
different compilers, what's the difference, is why the difference can we make it better?
I want to do another look at comparing architectures. This is Mbench 2.0 again.
And what I've done is I've taken all the benchmark, I've taken the, I've looked at the code size,
and this is a little bit code size, matters for embedded the embedded world. And I've compared
risk 5 against arm. And basically, this is looking at arm code size, remember small is good.
And if you're down the bottom, you're smaller, and if at the top, you're higher. So, what we see
is, and then I've ordered them by how well they win. So, we see the very best benchmark is nearly
40% smaller if you compile it on arm, whereas if you go to the other end, state mate,
the code on arm is 20% bigger than risk 5. But if we look at the averages there, we start to see
that actually on average on code size arm is about 9% more compact than risk 5. That's
slightly worrying. It doesn't actually, that's where you end up with, but that's something to be
worked on. In fact, of ice or extensions, this is looking at what's the benefit of the multiplication,
the compression extensions, and then we've looked at a couple of the core 5 extensions as well.
And I think when it comes to code size, what you see is that multiplication makes code a bit smaller,
because you're actually able to just use multiplication instructions rather than have to, you know,
all the emulation of multiplication. Compression, of course, makes a huge difference. You give the big
drop down as compression, and actually multiply accumulate, and then the extended memory addressing
instructions also further improve it from 45. When we look at speed, they're not surprisingly
having hardware multiplication makes a huge difference. So we got a big job as turn on M. Compression
instructions slightly slow things down on the implementation I have, and then the to the Mac and
the memory extensions from the core 5 slightly improved things further. So you can use benchmarking
to measure the impact of the extensions, is it a good extension? And I encourage people to do this
pre-silicon to check that as part of the extension design process. It's not the only thing
we benchmark, what about the compiler community? Is the compiler community healthy? So I've got the
statistics here. In terms of commits, this is looking at 14.1, and there were 158 risk-5 specific
commits for GCC-14.1, 600 left from. That is partly a huge contribution from G-Jong,
but the name right, in China, and his colleague, Ping Li, who contributed the risk-5 vector
initial implementation. That was a huge contribution in that release, so that made a big difference.
In terms of number of committers, 45 for risk-5, 43 from arm, the biggest committer, because of that
risk-5, 363 commits from the most prolific committer for risk-5, only 173 for arm. The number of
committers that accounted for 90% of the commits, 15 from risk-5, 13 from arm, similar communities.
And the number of companies committing, this is a bit strange actually. There's a lot of companies
committing to arm, 12, even though it's only a single company, where you might expect something
from Qualcomm and some from Apple. But you think risk-5 is used by all sorts of people. The lifting
is being done by just 16 of those 4,000-odd risk-5 members, okay? And that is an alarm bell for me
on the risk-5 community. Now this is GCC-C instantly. There are, you can do the same analysis for LLVM,
but I don't think it tells you anything to see anything different. And this is the graph
looking at how the contributions, so what percentage does each contribution is ordered by a
country? And you can see the first two contributors for risk-5, make a huge contribution. That's also
true of the first two contributors from arm. And then you've got a body of contributors
of contributing regularly. And I think I talked in my, the talk I was giving over the GCC rule.
There is a small corpus of committas that commit more than once a week, and they're responsible
for the majority of the code in the compiler. Okay? So I think I would say here, the community
is reasonably healthy, but it's not big. And I am worried that 4,000 members of the risk-5
community only 16 of them are actually doing any work to make sure the compiler works. And that's
a red flag up there. So I just want to finish about one example of using benchmarks. In my case,
I'm a compiler engineer looking at improving the compiler. And I want to introduce you to a technique
called iterative and combined elimination, iterative combinations, the base technique,
combined elimination is an optimized version. And what we did is we took the embeds benchmark
and we compiled it. Now GCC, we compiled it with GCC and we measured its code size. And we got an
embeds score of 1.0 for something. And GCC has something like 300 flags control which optimization
passes you turn on. And actually we started off by turning pretty much all of them on. And then
we did a 1,000 runs and there's a whole load of parameters to control them. So you've got about
1,000 flags you can change. And we tried changing each flag in turn, turning one optimisation off.
And we ran it all again. And those 1,000 runs we saw that some of these optimisations were making
things worse. Okay? And we found the one optimisation was making things worse than anything else.
And we disabled that. Okay? And then we went through and did the same thing with the 999 remaining flags
and find the next worst one. And we went on going through turning off flags until none of the
flags were making anything bad. And that took us about 35 runs. Okay? And it took 24 hours on my laptop.
This wasn't a big server. And the result of that was that we made the code 7% smaller.
Now for something that was automated run over 24 hours, that's quite a big win for no effort.
And I didn't change the compiler at all. Did take away a lot of useful information to
what was going wrong that led to ideas that might further improve the compiler in the future
from those flags we've got rid of. But it's a very simple technique. That's iterative compilation.
That was about 100,000 compilations I had to do over a 24 hour adventure small compiles in
second or so. The problem comes is we've done this with much bigger systems. We've done this
optimising for the performance of spec CPU. And a single spec CPU benchmark even with its test
data set takes a long time to run compile and run. We're talking minutes maybe hours for the
longest running ones. So you can't do 100,000 runs as you turn all the flags off. And that's where
combined elimination comes in. Because combined elimination is a set of ureistics for guessing
what's not worth bothering about. And we use that with one of our customers who's developing a
massively powerful RIS564 multi multi multi core architecture. And that took about three months to
do all those runs. But we gave them a 15% improvement in execution speed. And that's not bad
for something that well it was fit three months. But it wasn't three months of paying someone
it was three months of waiting while your boards burned electricity FPJs. So that was a pretty
good result. And we've gone round and we've gone back again because they've learnt from that
and improved their architecture and we're going to go over a second work. And then they're going
to add vector instructions in and we're going to do it again for vector. So it's all done on
the back of benchmarking whether it's n-bench or whether it's core 5 and we've done it even for
energy efficiency. And for those of you don't realize compiled code makes a difference to energy
contribution. And it's not a huge difference, 5 or 10%. Again the right optimisation flags and it's
worth bearing in mind that likes of Google and Amazon burn of the order of a gigawatt
of electricity. And a useful thing is a gigawatt costs a gigawatt dollar a year. So if I say
if you 5% or 10% of your gigawatt dollar that's a decent win to have. So I leave you that as my
fine thing is benchmarking isn't just about measuring you can use it to improve your systems.
And I'd just like to say thank you to those here. Dave Patterson who leads the
M-bench initiative. I'd say it's an entirely open community. We meet on the second Monday
of every second or third Monday of every month. I just sign up to the mailing list, sign up to
join. And we're working on the next M-bench for DSP class processes. Come and join that.
The open hardware foundation a lot of this was done in support of the open hardware foundation
with core pressure. There are colleagues there. The GCCNLO, the M community is tremendous
amount of input and back and forth with them. And I know particularly on the GCC, there's a lot
of specs for you working done by some of the companies working on a RISFI vector machines.
And lastly all my colleagues at M-COSM because I get come up and talk to you but much of the work
here has done be done by other people and I'm just showing off their work. So thank you all very
much and I'll take any questions you may have.
Hello. Thank you for your presentation.
Can you speak up? I mean it's not. Yeah. I'll repeat the question up this end.
I have to understand the choice between the four arms. It's 64 wide, the upper and one. It seems
like it's under different category. Yes, the RISFI've got a test at the moment.
It was being compared against a RISFI've core that's intended to be in a thousand core
high-performance computing scenario. So it was against a big core. I can't tell you which one
because it's secret. That bit isn't open source. But no, the Apple M. We wanted a D-SAN
AR-64 and I've got an Apple M1 in the office. Sometimes your choices are governed by
simple things like I've got one in the office. Yeah and the question was, were we comparing
light for light when we were comparing Apple M1 against RISFI've 64?
Okay, shall I have it?
So, AR, we're going to be, you mentioned light BGO makes things worse. Yeah. Why does LGO makes things worse?
So, the question is, I explained why the EGO made things worse and not why LTO made things worse.
The answer is still open question. We haven't actually had the time or
unalcomership compile a company. People haven't paid me to have the time to investigate that further.
But it is a cause of deep frustration because LTO can sometimes be the best optimization you have available
with some applications. I do not understand why it doesn't do a good job with RISFI and
if someone would like to say, please could you investigate this? I'm happy to take on the business.
Can you just, you said about the AR-64 with the extension that sometimes we need to
pull but we need to add it, can you explain you're making a bit more? Yes. So, when you compile
for the RISFI vector, you may well replace a whole load of scalar instructions by a single vector
instruction. That's the whole point of it. Auto vectorization works well. Both GCC and LTO, the M-support
are the EV. But RISFI vector instructions are not necessary as quick. Okay, so you may
end up replacing a few quick scalar instructions by a vector instruction that takes rather longer.
It's hugely dependent on the implementation. And certainly the initial analysis suggests that
might be happening certainly with some of these early RISFI vector implementations.
On the QMU, yes, shows it uses fewer instructions. But if I've got five instructions that
run in one nanosecond, and I replace it by one instruction that runs in 10 nanoseconds of loss.
That's the problem. And we don't know the answer. I had hoped to have the answer. I have
got a spreadsheet here, which has actually, I got finally got working at two o'clock this afternoon.
And I have not managed to get the comparison to look to see, it's only M-bench. It's not
specs CPU. Really to see does actually RISFI vector go plus or minus on that. Yeah, I had hoped
to have that by today, but I just run out of the hours.
We're getting the comparison. So there are now some new instructions that came up for
more compressed. There is one of them is now a part of the RISFI vector degree.
Yeah. Yeah. So did you also think those in Guadaldo, just the base C?
Hi, so question is, what about the ZC star new generation of compression extensions?
I didn't measure those, mostly because the work I was doing was with a particular
implementation that didn't have ZC star in it. I have done some working at code size, because
I don't actually have to run the stuff to do the code size for M-bench. And I actually benchmark
the code size effect of all the different, then combine them. And I think that came to the conclusion
across M-bench 1.0, you got between 3 and 4% improvement in code size.
Over standard C, okay, so that which is not as much as being be predicted, but I think
it's still worth having. Okay, and that was with the first generation of compiler
in May improve with future version of compiler.
Yes, a lot of work. I mean, working with various colleagues on actually the QMU side of
Nathan's been working with me on that. And actually the length of the vector is hugely
influential. And so, yeah, I can't say here is a RISFI vector result. I've got to say what length
of vector, this one is 256 length. So definitely makes a big difference. Yeah, question.
In addition to X, for GCCUS, go is in a little bit of a result. So even sometimes crashes.
So as you have the occasion to discuss with the front combination of flags, because if you combine
those, you do always add the exact same results, and you'll crush this or the front or the later.
So I confess that I was only doing code size, I didn't try and run the programs. Okay, so
one of the things, MBench 1.0 took a decision that when it did its code size optimization,
it would do it with dummy libraries that were always the same size. So we could get rid of the
library overhead. That meant, of course, we had nowhere of checking if they were real. That was an
MBench 1.0, I think I did there. Okay, one of the good things that comes with MBench 2.0 is
the stupidity of that decision has been made abundantly clear. So MBench 2.0 does do code size,
but it does make sure you can also execute it so you check you get the right result.
I haven't done, I know with the stuff we did with spec CPU, we did have a few cases where
the combined elimination blew up because a compilation failed, or an execution failed.
Now execution failed, it might have been because it's a architecture under development,
it could have been a bug in the architecture, or it could have been a bug in the code generation.
But over a half that, you know, I've got some a compiler developer, half the time is spent
the compiler tester, because that's where all of that happened, I guess. Yes.
Yeah, so the question is, are we exercising things like atomic and synchronization in MBench?
MBench is aimed at the very smallest of chips. It is defined to require no more than 64
kilobytes of memory. It's a sort of thing that's in your electronic key lock on your hotel room.
It's not of any significance. They're generally single threaded, they're not doing
complex synchronized, they're not running an operating system, they're bare metal, so it's not particularly
relevant. When we come to things like MBench DSP, which is just coming out now, that's going to be
the sort of thing that they will have to worry about on that. Let's go question.
So the question is, about using handwritten assembly optimizations, I'll make two comments about
that, yeah, you're right, spec doesn't allow you to go around hacking the source code if it's
a valid spec score. You do optimize, you turn on whatever optimisation flags you want. I just wanted
to qualify one thing. I get this all my customers. Oh, I've improved the code, because I hand
optimised this by writing a sembler. Why is it going 100 times slower? To which the answer is,
don't try and second guess a more than compiler. It will generally do it better than you. You've
just wrecked the data flow analysis and the optimisation's gone out the window. So that wasn't
the question you asked, but I use them, excuse to make that comment anytime I get it. So thank
you all very much for out of time. I'm around all weekend. Please don't find any questions.