OK. Good morning everybody. I'll give you an introduction to the easy project, which is
our way of trying to keep the P in HPC. How many people here know what HPC stands for?
Everybody. OK. I can skip one slide. So I'm in HPC, is that mean? It's super computing here in Belgium
and in the university when doing open source for a while. Like other things beyond open source as well.
Usually if you bring me a beer, I'm happy. It's a bit too early for that for them. I've been coming
to force them for a while. I've been co-organizing the HPC with their room, which is just two doors down.
Since 2014, and I'm the lead developer of the EasyBook project and also a core contributor to
Easy and there's a strong relation between Eastern. So HPC, everybody knows it. I can skip it.
Large computers, parallel computing, expensive servers, lots of CPU cores, maybe some GPUs as well.
Fast interconnect storage. It's a complex D, right? And it's not that easy to use as well.
So anything we can do to simplify this is actually helpful. What is important that in super computers,
you often see if every recent CPUs, like the often the vendors will first deliver a first batch
of new CPUs to the super computer. So you're dealing with the latest stuff. Modern micro-pressants
are support vector instructions. Things like AVX, AVX, AVX, 12, you've hopefully heard about these things.
On R, we have the scalable vector extensions. And using these are quite important for performance.
Like, if you're not using the vector instructions or if the binaries you're running are not using
vector instructions, it's like you're not using half your PC or half your CPU. You could be running
on a Pentium 4, mind as well. So these things are important because some parts of the hardware are
really dedicated to running the parts of the binary that use these vector instructions. So if you're
not using them, you're leaving a lot of performance on the table. You're not using a big part
of what the CPU supports. So you will be impacted and potentially heavily impacted,
especially with scientific software running on super computers. There's a flip side as well.
If your binary is used vector instructions and you're running it on the CPU that's a bit older,
that doesn't support them, it will just crash. You'll get illegal instruction and it will not work at all.
So there's this balance between both. But we do in hyperformance computing, we care a lot about
the performance, so we go through the pain of not using these generic bindaries that work everywhere.
So we want to keep the P and HPC. That means we need to optimize our software for the hardware
that's going to run on. So we won't take around the binary from the internet and say,
this is probably be fine. Now we're going to build our software from source on the host,
where it's actually going to run. And we're going to tell the compiler, optimize for this CPU that you're
seeing right here. So this can be quite important, especially for scientific software. So how important?
There's a good example. Gromax is a molecular dynamics simulation software, so you've similar
like molecules and atoms, and that's all I know about Gromax, don't ask me more. But I know it
consumes a lot of compute. There's some numbers that don't have statistics. It consumes about
5% of the super computing power in Europe. That's a lot, right? 5% is not small, but this is one
software package, consuming 5% of everything that's there in Europe. So this is huge.
Now with Gromax, you have to be very careful. What the graph shows here is we're on one system,
a fairly recent system in AMD Epics and for, that has 192 cores in total. So we're stuck to a
single computer. We're running a recent version of Gromax with a particular input. The details
are there, but they're not really too important. And the different graphs show you different
Gromax binaries. So exact same source code, exact same Gromax version, exact same system.
All we did is compile the binary differently. Higher is better, so higher means higher,
better performance. If we're using a generic binary on the left, we're essentially using
two instructions, which I think are pending for 20 years ago already supported. So it makes
zero sense. And then you get about eight nanoseconds of simulated time per day. So we're metric,
but yeah, it gives you some idea. If we're doing a bit better job, if you say build Gromax
for a modern, more modern CPU, let's say the last decade, anything that supports ATX 2 instructions,
we already gain a bit. We get to let's say 11 and a half. And if we do, if we go full out and
really optimize for the CPU, we're running on, we get another bump. Now, the difference here is 78%
going from left to right. So if you're not careful and you happen to be on the left,
you're losing a lot of performance and you're spending just way more time running your simulations.
And you're not using half the harder, that was quite expensive to buy. So that's a bad idea.
Don't want to do that. You want to make sure you're on the right side of the graph as much as
as possible. Now, very good. Why don't we always do that? Well, we try to. But there's a lot of
scientific software that get run on supercomputers. And getting worse and worse, it's really expanding
to other fields. By one of our metrics is now very compute hungry as well. They have a lot of data.
They can start running on GPUs, so supercomputing is very interesting for them. And I don't have to
explain you about the AI boom that's happening. So that's bringing lots of new software,
to supercomputers as well. There's an increased interest in the cloud. So cloud is very flexible.
It just pin up VMs. You run something, you tore down VMs again. And you only pay for what you
use essentially. And that's, yeah, supercomputing centers cannot beat that. So it is very interesting.
We see other micro-archities as other CPU families coming beyond Intel and AMD. So for the
very long time, it was like everything was Intel. There was nothing else. AMD came back in the
game that was too late. It's exactly six. It was close enough that it was still easy. But now we
have our, like the biggest supercomputing in supercomputing in Europe, Jupiter in Germany, just across
the border with Belgium, near Archen. It's full arm. Nvidia grace CPU, so it's all arm. If you're
software cannot run on arm or it can compile on arm, you're not running on the biggest supercomputer
in Europe, so that's a shame. So that's already interesting. And then there's the risk 5 CPU
family that you, there was a death room, I think, yesterday on risk 5 as well. That's coming,
so that's the third CPU family, which is totally different from exit and six in arm. So if your
software was built for Intel and AMD, it will not run on arm, at least not without emulation
and other 30 tricks, which only slow things down. Same thing for risk 5, so you get this huge
mess of things and it's only getting worse and worse. So we're getting more software,
we're getting other platforms like cloud, we're getting bigger diversity in CPUs, and it was
already pretty bad that we had to build everything from source. On the flip side, the people that
help the researchers using the supercomputers, so the support teams, which I am in, they sort of
have a cap in the amount of people that can be there. There's only a certain amount of money for staff,
but they get a lot more work, a lot more questions and they need to help a lot more people.
So this gives some idea. This is the number of software requests we've been seeing
at Ganton University over the years since 2018, and it just line goes up, which is bad,
because more requests is more work for us as researchers waiting longer because they can't figure
out how to install the software themselves, so that's bad news, just more and more software,
more and more requests come. Yeah, so this essentially has doubled in about 5 years, which is pretty bad.
So we're constantly busy, we have ideas to improve things, but we just don't have the time to
implement the ideas because we're busy handling all the requests. What if we could change that?
So what if there's a way that you don't have to install the software anymore that you want to use,
even on your laptop, on the supercomputer, in the cloud, what if it was just there and what if
that could be possible without compromising on performance? So that would be interesting,
and that's what we're doing with the easy project. So that's exactly the goal we have
to really change the situation for the better. So how are we doing this? So easy is a bit of a
game to acronym. It's the European environment for scientific software installations very long.
European actually helps because it helps us get in funding for doing this. You can pronounce it as
easy, which is funny and titles of talks and stuff. So what it really is is a shared repository
of installations of scientific software. And I say installations that's important. It's not a
package manager that you can do RPM install. No, it gives you the installation itself. So if you
get access to easy, the software is there, you can start running it. You don't need to install anything else.
I were really doing this to work together, so we want to avoid a lot of duplicate work,
people were installing software again and again in their own silos, and maybe exchanging ideas
were working together on tools like easy builds, but they were still doing the actual work of
getting things to install and compiled and build in their silos. You also want this to be a uniform
way of giving software to the users. So whether you're on your laptop or whether you're on a
super computer or in the cloud, you should be doing the exact same thing, and it should be working fine.
It should work on any Linux operating system or still geared towards super computers, which are
100% Linux, but you can do virtual machines, I can run it actually on my Mac here as well,
and the Linux virtual machines, so that's good enough. So we want to support laptops, work stations,
super computers, clouds, Raspberry Pi, as well, why not support different CPUs. We're also
take care that pay attention to the network and the super computers. We want to support GPUs to
make it even more interesting next to all the CPU families we already have, and performances are
number one focus point, but we want to automate things, there's lots of software, so we need to
automate things as much as possible. We're very serious about testing as well, and of course collaboration
is everything we do. Some of this is funded through a Euro HPC center of excellence, so the
European does really help for getting money to build this out. So yeah, some of this is repetition,
uniform software stack essentially means you have the same software environment everywhere. You
go somewhere either easy as already there, if not, you can easily install it, and if it is there,
you know what's going on, you know how to use it. If you don't want to sacrifice performance,
you often see this with containers and come down somebody gives you container image and
everything is there, trust me, and it works everywhere. It's like, that means you're running
binaries that work everywhere, so they're not optimized for my host CPUs, so performance is
going to be not as good as it could be, I get a ability of compute when I'm sacrificing performance,
and that's not what we want to do. To some extent, condos the same thing, they build one package
for x86, they don't build for every possible x86 CPU, we do. We're avoiding duplicate work by
working together, we use tools that automate the software installation process, like easy build,
but like I already said, you're used them in silos, you're not really working together on the
installations themselves, and that's something we want to change. And it also helps, and then
it actually opens up additional use cases, so it helps with training. You can give people a temporary
super-computer in the cloud, there's tools for spinning up slurn clusters in the cloud,
you give people access, you give them easy to get access to the software that they want to run
on that cluster, you train them on them for like a week. After that week, you throw the cluster away,
you literally burn it down all the VMs are gone, but they can still get the same software
environment, easy somewhere else, on the laptop, on all the super-computers, so you train them in
an environment that they will be familiar with and that they can translate to other platforms as well.
So a bit more graphically, what does this all mean? Well, we have a whole bunch of software,
like Gromax, but many other things as well, or Python packages, bioinformatics,
PyTorch for AI, whatever you can think of, as long as we have source code, and it's open source
software, we can install it in easy. Today that includes 650 different software projects,
and then I'm not counting things like NumPy or R libraries, we have, it says over a thousand,
it's actually over two thousand additional things like those, if you're counting those,
that software then it's close to 3,000 unique software projects available today,
work on any Linux system, I haven't seen any exceptions there, whether you're running
Fedora or Ubuntu, I don't care, they work. We support 14 different CPU targets, so today
across essentially two CPU families, so AMD Intel is one arm as another, and we essentially support
multiple generations of CPUs in there. So if there's an AMD's and two, it's in three, it's in four,
we support all of those, and we build the binaries for each of these generations. And then we also
already have support for an Nvidia GPUs, also there are three generations of GPUs.
Rescribe is coming, we're playing with that already, exploring that support for AMD GPUs,
so for the Rokom ecosystem is a work on progress, so we're expanding our scope as well,
in terms of how we support. What's really interesting, I'll show this hands-on,
is that the software is three and then on demand. So once you have access to easy, you can see what
is there, Gromax, which versions, whatever, when we need to start running Gromax, it's actually
pulling things in behind your back, a bit like you go to Netflix, you pick a movie, the movie is not
on your system or on your TV yet, when you start watching it, it's like being pulled in as you
watch it. This is the same thing happens with the software here, so things are pulled in on demand,
so you're not pulling in a 10 gigabyte container image and then using a 100 megabytes of it,
now you're only downloading the 100 megabytes, they're actually going to use.
And this is now being embedded in the Euro HPC Federation platform, so they're becoming
rather bicting in Europe across all the European super computers. All of this is powered by
open source software, so all of this is done in public, anyone can access easy, there's no limitations,
you don't have to tell us that you're using it, you just do what's in the documentation,
you install and configure certain human-fests and you get access to everything.
The top layer of easy, which we call the software layers, where the actual scientific software
is in, so that installations installed with easy builds exposed through the module system
that we use L mod for, and there's a small thing that does CPU detection, so you don't have to
tell us which you have, we detect it for you and we do the right things on the need of the
covers. We use Gen2 Linux to build a middle layer to make things compatible with essentially any Linux
system, so we don't get about your Linux distribution or work fine, and you certainly am
a fast to distribute the software and get this on demand's streaming behavior. Refrain for software
testing, magic hassle to spin up clusters in the cloud to do demos or to actually do the builds
themselves, we do a lot of the builds in AWS, which are sponsoring easy, they just give us
free access to whatever CPUs we need to have access to, so everything essentially it's one big
puzzle, and at some point all the components were there to actually make the puzzle and build
out this bigger thing fully built with open source software. So same thing, software goes into
easy, easy pops up on local supercomputers, like we have in flounders, the flamish supercomputing
center, it's actually one very close here in Zellek, a new one Sophia at the University of
the University of Brussels or the Flamish peaking University of Brussels. They do get along,
so you can use it on your laptop, on your PC, as long as you have Linux in there somewhere
in the VM or directly, you can use it in the cloud, commercial clouds, but also this
European open science cloud, if you haven't heard about this and you'll research that
sometimes need like a small VM to try things, you can get it for free there, every citizen in
Europe can get a small Linux VM or whatever, where you can do some stuff on, you have like a
monthly budget or you can run things. It works on Raspberry Pi, we have done demos and this
it also works on risk five boards, if you're a bit more careful, if you know what you need to do,
and it is on the biggest supercomputers all across Europe, thanks to the Federation platform,
so we're essentially everywhere. So how does this work? Like I already explained, software is
streamed in only a month, as it is being used, thank you, sir, the MFS for that, we built the software
and we built the binary so that it's independent of the host operating system, that's where the
gentle Linux comes in, this middle layer there on the top, we call this sometimes containers without
the containing, because containers do essentially the same thing, they build a smaller operating system
in the container image, and then they put you in a box to protect you from the outside world,
we actually don't want to be in a box, we want to access the GPU, we want to access the inter
connect, we want to get access to all the fancy hardware, so we're taking the idea of being
independent of the host, but we're not containing, because we don't actually want to be contained,
and we're optimizing for specific CPUs and GPUs as well, we'll alter the tech what you have,
so you don't need to care whether you're on an Intel or an ARM system, that's fine, it works fine.
As an end user, and this is something we can still improve, we're working on this, once we get
access to easy and I'll try to do a demo with the Wi-Fi holds up, as an end user, you source
and in its script, this is something I really want to improve, we're working on this,
but it does the detection of your CPU, it sets up your environment, and from that point forward,
you can just load modules to activate additional software and run the software, and everything will
work fine, so once you have access to easy, you can stop installing software, everything is essentially
there, if the software that you need is included in easy, of course. How does this work?
Well, you're this nice person working on her computer, you ask easy, do you have the
Chrome Expinery already, it checks the cache, if it is, then it just runs it, if it's not there,
it will download the binary from the middle of server or one of the middle servers that we have,
puts it in the cache, and then runs it, and the middle server synchronizes with the central server,
so you have like a network of servers, essentially, serving this thing. All of this is done by
SerbianFS, we get this for free, essentially. So yeah, the streaming thing is interesting.
This gives a bit of a better view of the SerbianFS network, so in the sort of middle of the slide,
the darker part is the central server, this is actually sitting in groaning in the Netherlands,
currently one of the partners in the easy project, let me have a couple of middle servers around,
there's one in AWS, again sponsored, thank you, AWS for that, we still have one in Azure,
but that one is probably going to die soon, it's sitting in US East, which is not that interesting
for people in Europe anyway, there's some people in the UK that built or introduced another middle
server because they like the idea of easy, so they want to help the network, and then there's a separate
one on the top right that we use to synchronize local members, so that's interesting, if you have an
HPC cluster, you actually want to have all the software close to the cluster, so you can actually
set up your own middle server, synchronize with hours, and then you have everything close and you're
essentially in full control, you can disconnect the network and everything nicely works.
All right, so demo, I'll skip this because I'll do it hands-on, I'll skip this too,
essentially it's the same thing, you source, and in it's script, you load modules,
download an input file for Gromax in this case, you file a Gromax and it works fine,
and you can do the same thing, so this script that I'm showing you can copy paste this, put it in a file
called easy-dash demo.sh, and if you use Lima, it's a tool for starting Linux containers,
or Linux VMs, I guess, on macOS, so you set up Lima to be aware of easy to provide the easy
repository and slash tv and fast software, and then you can run the script, so this script that I showed
you works on the laptop, works exactly the same way on a supercomputer, you're logging to the
supercomputer, you use commands to submit the script as a job, without touching it, without making any
changes, works fine, so you get some performance numbers after running Gromax, you're looking
to the cloud where you may have a slurm cluster, here I'm actually, it doesn't show it, but it does show
it, so the P equals says I'm submitting this to an arm partition, again I can run the same script,
no changes needed, I just s-batch the script and it comes back with the results, and the same thing
on one of the supercomputers in Europe, I can do the same thing, so wherever you are, it's fine,
if you're a bit smart about writing the script, so if you download essentially the input files that you need,
it works fine, the software essentially follows you around wherever you go.
So before I do the demo, how do you get access to this, how do you get this tvMFS software
easy-ion thing on there, all you need to do is what's on the slide, so this is for a redhead-based
system, you're essentially installing sort of your MFS, so just a young installer, the NFN
stall from Europe, you can find this in the documentation, so this is for the repository where you can
get sort of your MFS, so then you can install sort of your MFS, you configure sort of your MFS
for this guy, in this case a single VM or single client, which is just giving it direct connection
to the mirror server, if you're a single client that's okay, and give me 10 gigabytes of
disk space at max to download software when I'm running it, so you can limit it as well, and you
complete this setup, run this on a redhead-based system, an RPM-based system, you get access to
easy and it works, now you may not believe me, so the best way of showing you is actually doing this,
so here I am in Amazon, I will spin up a VM, I'll make it an R1 to make it a bit more interesting,
we'll have eight cores, 50 gigabytes of disk which is way too big, I will launch this,
I should take a seat, so I'll launch this VM, this should be quite quick, it will go as well,
this gives me an IP address, I can SSH into that IP address, and I do fast, yes, there it goes,
this is an entirely new VM, uptime zero minutes, so it was really spun up right now, this has
like no software, it's a bare bones, redhead-9 to a realm-9 operating system, there's not even
them here, so how can I survive without them? No, no, we don't do it, so what I'm going to do,
I'll just copy-based here, the installation instructions for getting access to easy,
just literally copy-based for this, the same as was on the slide, I entered this, and now this
is going to take hopefully only like a minute, but essentially all this is doing is the biggest
part of the VM installed SerbianFS, now the nice thing is we don't need to do anything extra,
there's actually a second step here as well, which is installing a very small RPM that has
the configuration for SerbianFS that knows about easy, you can actually skip this one because
easy is included in the default configuration for SerbianFS, so the SerbianFS people like us enough
that we actually get into their default configuration, this is not relying on the Wi-Fi at all
to do the installation, so this should be quick, it's literally just installing SerbianFS and it's
dependencies, so you won't see Go Max opened from TensorFlow, none of this is actually being installed,
so hopefully this completes quite soon, yes complete, alright, so now we have SerbianFS available,
if I remember the command name, so this is installed that's actually all we installed, so now you
think, oh yeah, we can access easy, right, this is so this CPMFS part is now suddenly there,
but this looks empty, like what's put it I miss, well you didn't miss anything, this works like
web pages, if you know the URL, you go to it, it's auto mounts the repository and suddenly it's
there, and now of course if I do that, let's in here I do see it, alright, so we have access to easy,
excellent, now I'm thinking what the best way is to actually do the demo, well let's copy paste
the script from the slides, like I promised you, this should work out of the box, so I'll
literally put this in a file called easy demo, and of course I don't have the, I can use fii though,
that's okay, put this in the script, make it executable, yeah, alright, so just to remind you what
the script will do, it will set up our environments for doing CPU detection, what are we running
on, please update my environment, so I can start using software, will load a gromax module,
and I will just download the file to run gromax and actually do the do the launch,
so this fits out a whole bunch of output, also again something I'm not a big fan of right now,
the key part here is it has detected that we have an ARM CPU, and an ARM Neo1stv1 CPU,
it supports certain instructions, that's the binary we're going to use, it didn't find any GPUs,
that's correct, we don't have any, it does all that, it download the input file for gromax
and it just runs gromax, I don't need to do anything else, so that's excellent, I'll cancel it,
this will take a little while, to bring this home a bit more, and since I'm doing quite well on time,
I'll download some of the demo scripts as well that we have, okay,
do I need this, no, actually I don't, but I can, I mean, get this in easy, so I don't need to
install it anymore, I can just do this, I set up easy to update my current shell to make
modules available, so it does the same detection, and now I do have get, I didn't have to install
it at all, so now I can do this good clone, go for demo scripts, I've demo scripted here for
the bunch of software, I have bio-conductor espresso gromax by torch open foam, quantum espresso
tensorflow, somebody pick one, quantum espresso, okay, that's the dangerous one I think, let's see
what happens, so this looks wow, science, I have no idea what's going on, but essentially this
will just load modules for gromax and then science, so let's run it and see what happens,
so it sometimes takes a while to react, that's by, because it's firing up the gromax,
the binary, or trying it, in terms of your emphasis, oh wait, I'll actually get it for you,
and then you can run it, right, but it's all transparent, so you don't really notice this,
so I don't know what this means, but it's something scientific, it's some kind of plot, I guess,
or yeah, so it does work, okay, so hopefully that was a bit convincing and the life demo worked, yes,
so that's really all the rest of it, there's no hidden tricks here, you can essentially do it,
but I did, if you have access to any Linux system, we have installation instructions for you,
Ubuntu as well, if you get really advantageous, if you're on the risk of five boards,
you need to install sort of being a fast from source, it's not that difficult, it takes a while,
but then yeah, so the steps may be a little bit different depending on their own system,
now this is a software performance devroo, right, I already explained, you need to make sure
you run binary that are optimized for your hardware, that's very important, so build for your CPUs,
don't sacrifice performance for mobility of compute, like people typically do with containers,
now you can actually build containers that are optimized for specific CPUs as well and run those,
there's actually benefits to that as well, but that's just not what people typically do,
they build a container image once and they drag it along everywhere and they assume everything is magic and fine,
it's not, so you do have to be careful, like I showed you in the demo, another performance aspect,
it takes minutes to get access, so in a hard beat, it works fine, whatever you're running,
it's trimmed in on the amount, so don't need to wait long for it to actually start
because you're only downloading what you're using, again, different from containers,
if you have a container image, that's big, because it has a lot of software, that's good,
you have to wait until the whole container is downloaded before you can start running it, so that's a bit annoying,
so we may first use lots of caching as well, so if you download something once,
it has like a local cache that was that 10 gigabytes I mentioned,
so it doesn't download again and again every time you run it, of course, and there's multiple levels of caching
in this network, it improves startup performance as well of software, on parallel file systems,
and that's maybe the last big thing I want to talk about, I have more slides on this,
and then humans waste less time as well, they collaborate with essentially install the software
once and easy, and everybody can use them, so the startup performance thing is I added these slides
in 10 minutes ago, but so starting software on super computers can be slow, like what the fact,
it's a super computer, why would it be slow? Well it's because very often you only have a parallel
file system on the super computer to like really big things, huge data files, and a hundred computers
working together on single files and writing things, parallel LIO, really nice, really bad for software,
you don't want to install your software on a parallel file system, if you go to the documentation
of like Luster and GPS, it actually tells you don't install your software on Luster, it's just
it's a bad fit for what we do, but very often it's the only option you have and like there's nothing else,
so software installations are typically small files, binaries, Python files, whatever,
and quite a lot of them, potentially, and it has a pretty weird access pattern compared to regular
data files as well, you can solve this by putting the software in the container image and you have one
big file, very good for a parallel file system, but you still want to build your software from source
and then put it in the container, which is like extra work again, and if you're doing a central
software stack, so you want to get something rich for your users, doing that with containers is a mess,
because you either have very big containers or you have a hundred of them and people have to pick
and get very messy again, so one very concrete example I'll power through this quite quickly,
start up performance, how do we test it, how do we benchmark it, let's try import TensorFlow,
it should be fast, right? There's not a lot going on, it's a single import, we're not actually using
TensorFlow, well, this triggers six and a half thousand open goals, and it opens over 500
directories for some reason, it actually requires almost three and a half thousand of those files that
includes over two thousand files from TensorFlow itself, then almost a thousand files from dependencies
and other stuff, and then a little bit of things from essentially the operating system and
this middle layer, this compatibility layer, and totally it's about one gigabyte of data,
that's not a lot, but because it's scattered because three and a half thousand files,
it's very bad for parallel files systems. To show this, some benchmarks, this is like the optimal
case, crazy mode, where you install TensorFlow in memory, like you actually built TensorFlow from
source and you put it in the RAM disk, which is bad because you reboot your computer, it's gone,
and you can start again, but this is the best you could do, there's no faster files system than
so if you do that, it takes about two seconds to do the import TensorFlow, not bad, that's
worth waiting for, you can probably optimize this, but this is like the ground truth, we cannot really
do better than this, more realistic is lost, so if you do this on the left, it's called
situations, let's say a fresh reboot, lost or hasn't cashed anything yet, you're the first
poor person to do import TensorFlow, you're waiting for over a minute, for the import
TensorFlow to complete, and this is like normal for a ruster, if you talk to a luster people,
it's like yeah, it's software, we told you not to do it, if it's hot kernel cash or everything is
cashed already, you can get close to the optimal because then essentially everything is already
in memory, so luster also cashs things and it's not actually always that bad as it is on the left,
luster is open source, parallel, file system, very good, GPA vases commercial software,
so there you get like 40 seconds and if it's cashed or not, it doesn't really matter too much,
so it's always like half a minute to do an import TensorFlow, very annoying, so bad fit for software,
certainly my face is a little better, but you have to be careful how you set it up,
so with certainly my face we get here like in a cold situation, so the cashes are clean,
you get around 9 seconds, if you have the software very close to your system, so we'll
private proxy or a private mirror server, if your mirror server is quite far away, so if you're
in the Netherlands and your mirror server is in Dublin, then you're suffering about a minute,
but that has bad as DPSS, okay, I'm actually downloading from half across Europe,
that's still doing fine, if your mirror server is across Atlantic, then it's very bad,
so you have to make sure your network is set up well, and then there's two additional situations,
there's hot cash, so everything is already in memory, so ground truth, or very close to that,
if it's not in the kernel cash yet, so not in memory yet, but it isn't a private cash of
serve me my face, then you get the stuff on the right, but then we're still talking 4-5 seconds,
so nothing even close to the 30 seconds or the minutes that we were seeing with the parallel fases,
that happened because here with me my face was built for software, right, so it knows the
access patterns, it can handle small files quite well, it does local cashing on the client,
multiple cashing levels, so way better than GPFS or LASTA do for this type of stuff, yeah,
more information, time is up, okay, I won't go through this, I covered most of this, lots of
software is already there, if software is missing, we actually this is a community project,
you can send us a full request, please add this, we will check it, if it's okay, we will build it,
we will ingested in easy and is there for everybody, if you want to learn more about this,
come on and talk to me, but this is a community project, easy is already available, all across Europe
in the cloud, on the big European systems, there's a list in our documentation,
and in this part of the EuroHPC Federation platform, so really essentially, so can
university is being paid to integrate it into this new platform, which doesn't exist yet,
it's coming end of March, and it's essentially a one-stop shop, one central point to access
European supercomputers, easy is going to be part of this, more information on the webinar,
and thanks to our sponsors, European Union, European project, and lots of links to click on
and get more information.
I guess we have time for a question.
So, if I listen correctly, the main goal of the easy project is to make infrastructure,
so it's easy for the scientists to access the HPC, right?
Yeah, to use the software that they want to use, not necessarily HPC, but.
Okay, so then you mentioned that one issue that you have is making that software performance
for specific CPUs, right?
Yeah, like R, GPUs is the one, so there's another player now coming to the industry, which is the
QPUs, right, the quantum process units.
Yeah. So, my question is, is there an initiative in your project,
where I'd taken to account that or how is that being handled?
Because this is a hot topic now in the quantum computing industry.
How can we integrate those things, right? How can you do share the load between a GPU or
a CPU and so on? It would be very nice to have our ads, some open search there in this field,
because there are a lot of private. So, summarized could easy help with the situation where you
are getting quantum computers and you want to optimize software for those, very good question.
I don't have good answers there. What I can say is that easy is being integrated into this,
which is not only the supercomputers, but also the quantum computers. So, in some sense,
we'll have to provide an answer to that. Now, to me, the libraries that you would use to access
the quantum computer is just another software package. I don't care too much. And for me, it's the
job of the libraries to make sure that they are able to use the quantum computer very well.
Or maybe the compiler that's using, it's not really programming languages in quantum computers,
it's very different. But it's, I'm more like on that side and we will just, we install all the
software that you need to make good use of the quantum computers. But it's not our job, I think,
to optimize for, I don't know how the quantum computer works. I'm happy to install thousands of
software packages. So, as long as it's open source, then we'll support it. Okay? Thanks a lot.
Yeah, you have to wrap up if there's anyone has questions, I'll be outside so you can meet me there.
Thanks a lot. Thank you.