So, yeah, I want to start a small talk about the Linux kernel, because actually in this
Devroom, it's quite rare that we talk about the Linux kernel at all.
So first of all, I'm not a kernel developer, I'm not even a software developer in C, I'm
just a Czadmin of an HPC cluster here in the university, so it's a perspective I'm going
to give you.
And as I said, we put a lot of care in this the HPC community in user space, like we compile
the software of our researchers with a lot of care so that he uses all the features of the hardware
underneath, all the best instruction set of the CPU, we also use their two things with modern
compilers, modern libraries so that we can squeeze all the performance out of the software of the
researcher, but on the other side we have the kernel and we don't even look at it, we just
install usually the kernel that comes from your Linux distribution of choice, and there's
a reason for that, and it's because you want stability and compatibility in the kernel space,
the Linux kernel that is shipped by the Linux distributions has a lot of features that we like,
so the people in the Linux distribution, the Linux distribution and they put a lot of effort
to make sure that the kernel works, is stable, is secure, they fix vulnerabilities, you will get
updates, and you have a kind of standardized environment where for instance vendors can provide you
with kernel modules, they will work, and there's a good understanding of what you can expect from your
system, but we would also like to see what happens if you can improve the performance of your
cluster by optimizing the kernel, because the kernel is shipped in a very stable fashion,
it's very portable as well, but that means that the instruction set that the uses is not optimized
for the hardware of your system, so it uses a generic instruction set for X86 CPUs so that it goes
to null 8, X86 CPUs, but if your CPU has other features that are specific to that model,
they will not be used by the kernel, and we want to do that also without losing all the stability
and compatibility I talk about, so we don't want to break any users of what it is already running
in the system, we don't want to change any of the features of the kernel itself, we want that
all the drivers that we have from vendors still work, so basically what we want is a drop in replacement
of the kernel, that magically can be make the system go faster, so no patching of the source code
and no changing the config of the kernel at all, and also use the same package manager of your
Linux distribution, in our case that's RPM, because we use our Red Hat based Linux distribution,
so we'll use RPM built to actually play with the kernel, so what we did is we device some
benchmarks, typical HPC benchmarks, and we used easy build and reframe, and those are the
the benchmarks that we systematically run on our cluster, those we have data historical data,
and we know very well, and I also pick some benchmarks from the phonics test suite, which is very
commonly used by the press, and it has a suite of tests that is targeting HPC tools as well,
so what we, what I did is run, it's very simple, run some the benchmark with the
digital Linux kernel, also recompiled the kernel from the distribution by myself without making
any changes, and then start playing with configurations and optimizations of the kernel,
and say what happens with all these benchmarks. The system also is a production system in our
cluster, so it's a Skyleg Note, and it has ABX 512, so that's something that we want to actually
be test if it's useful or not, and we are currently using a Rockley in Obsator 10,
which is not the newest, I know, but that's what we have on production, and my goal is to not create
exotic system to play, at the end of the day what I want is to actually apply
any improvements that I might get into the cluster that we have today, so I'm going to try to
test as much as possible, close to the system that we have in production. So what kind of
optimizations are we talking about then? Basically, what we can do is recompile the kernel with the
same source code of the Linux distribution, with the same configuration, and then you can play with
optimizations at the compiler level, so the compiler, as I said, it can generate assembly with an
instruction set that is more or less adapted to the features of your CPU, like for instance,
using SSC instructions or AVX for the transition, we can also enable optimizations of the
assembly code, because the compiler can actually do a lot of magic with your code, so for instance,
it can inline functions, you might have multiple functions, but if the compiler
evaluates that those are more efficiently run and bet it one into the other, what it will do
is just copy paste the code of the function into the caller function, so that there's no calling
between functions that might be lying very far away, and this is what we tested first, so here you
have the results for the different CPU instruction set, in blue is AVX 2, in red is AVX 512,
and what you see here is a percentage, the performance gain, and for those tests that the performance
gain is again in time, so that the less is the best, I inverted that, so positive is good,
negative is bad for all the tests, and as you can see here is messy, so in some cases you gain,
so for instance we have FFTW, so the first Fourier transform with floats and SSC,
that actually benefits from 512 with max sense, because it has floats and SSC,
but on the other hand, the same test with stock settings, it doesn't gain that much, and other
applications total you are affected negatively, but by this change, the overall result is
negligible, so we get in on average 0 to something percent of improvement,
and what is also interesting is that all the tests that you see here, the first ones are those
that we run with easy built and reframe, and those are software, modules that are very well optimized,
and very well adapted to the hardware already of our cluster, and those see even less change
than those from the Forenix test suite, so it seems kind of not really interesting,
we also compare using GCC and LLVM to see if there's any change, fortunately it's even less,
which is expected, there shouldn't be much change between GCC and LLVM with default settings,
not making any other changes, and then we compare O3 between O2 versus O3,
so by default the kernel is compiled with O2, and then we try to O3, with the enables,
that kind of magic I talk about, the compiler inlining functions and on rolling loops and all the
kind of stuff, and it's kind of the same story, it's messy, there's no general trend, you might
improve some applications, but others you might have a negative impact in them, so what we also
one is that we don't deploy something, then one of the user's, one of the research groups,
and finds out the day after that their jobs are running half speed than before, so and on average
the improvement is totally negligible, so there's not much to be said here, but then there's the other
as a SQL level is what is called the profile guided optimizations that you can do in the kernel,
and those works in a different way, so you can have your kernel compiled with some instrumentation,
which is GCCov, which is a coverage tool that will analyze and generate a profile
of which parts of the kernel are most more used or less used while your software is running,
so you can compile your code with this instrumentation, you reboot your system into this special
kernel with the instrumentation, you run whatever workflow you want to run, it will generate this data,
and then you can recompile again a second time your kernel, using this data this profile,
and it will annotate the source code with the probabilities of the different branches,
and it can also do estimates of the values of expression and then compile the code
to the assembly based on that information, so it generates a binary that is more optimized for
that specific workflow, so we test that with NumPy, and we saw on in red you have the kernel that
it's optimized for a test of NumPy, and you can see that NumPy here is performed 3% better than
the rest, and if this is the best value that we ever got in these all these tests that I've run
with all 3, 0, 2, and all that stuff, and what we also did is to run the same with profiling
for all the tests, so we took all these 5 tests that is here, and profile of them, and you can
you can see the NumPy then performs worse, but what is nice also in that in both cases you get
on average more or less the same improvement, and this is just 1% 3% in the case of NumPy which is
very small, but in this case it's the profile guide optimization that I did is the basic one,
just branch probabilities, there are newer ones that are coming in the kernel 6 to 13,
which was released just on January, and there's also both developed by meta, which will be
available now, but they are very, very narrow, so these needs more work, so for the conclusions,
this is a little bit of improvement, and also it might be very complex to deploy, because as I said
you need to target some specific workflow, and in a rich HPC environments that may not be even
applicable, because if you have users doing very different things it's going to be complicated to
actually have a surge of demisitions deployed on your poster, so let's do it
all right for one question, before we start moving, let's do the question first,
can you go up slide 9, so there you have FFTW flow does SSC, what do you mean SSC,
so is that SSC then you run it with AVX5 flow, what does that mean, well SSC is because
these codes will use SSC instructions, so we'll use this SIMD, the single instruction,
multiple data instructions, those are not, not enabled by default in the district kernel,
because that's generic, compacted with the generic flag, so there's no SSC in there, with AVX12,
well I actually use native, native is AVX512 plus SSC4, plus some other stuff, because this is an
skylic CPU, so it has all these, these instructions are supported by the hardware of the CPU,
so now this code can actually use hardware paths, transistors in the CPU that were not used
by the Linux kernel version, so very used as C by default, and then you use the
S512 and the others, you don't use a C at all, the other code that the stock code doesn't
use in Seattle, even if you enable it, because the source code of this test does not have those
instructions in it, all right, thank you Alex, my pleasure,
you're all good, you're all good, you're all good, you're all good, you're all good, you're all good,