I have to go and find a broomstick.
Oh, I can't.
Okay.
So, good afternoon, everybody.
We have to really nice and professional presentations.
But I have 40's light in 20 minutes.
So, I would say to people,
I'm a Spanish. I can't talk really fast if I propose to them.
So, my name is Elena Vella.
I work for Dwighty now, but that doesn't really matter right now.
I'm here to represent easy,
and I'm going to make you an status update
on the European environment for scientific software installations.
Who?
Status update since when?
Well, we presented it for the very first time here at the foster five years ago
by a work colleague, Bob,
and he presented easy one scientific software stack
to rule the mall,
and I hope to keep you convincing about that after all this presentation.
So, what do we want with easy calibration test?
How many of you actually work in each PC?
Okay.
I don't have to over explain.
What we want with easy is to provide
a truly uniform software stack,
and that means that you will find the exact same software everywhere
without sacrificing performance,
because we are doing hyperformance computing,
and I want to think that the work performance actually means something.
And you know that if you compile a binary for everything,
it usually doesn't work that well in performance-wise.
And to avoid duplicate work,
because we are all in these together,
we are researchers,
we are users,
we are user supports,
we are system administrators,
that we have to install the exact same thing everywhere in Europe,
and it's just not practical.
So, we have to,
like easy wheel and spark that help us doing that,
but at some point it's not enough,
because you can share your easy wheel recipe with somebody,
and it just won't work for them.
So, we are not just sharing how to build them,
we are sharing directly the binaries,
and we are working all together into a shared software stack.
And this way we facilitate HPC training,
the development of scientific software about modern data.
And with this,
I go to the demo,
I hope to.
Maybe can it help?
Okay, so this is my laptop.
As we are at the first them,
I feel obligated to say that Ubuntu is not my preferred choice.
It's a company policy.
So, you can see it's an Intel machine,
nothing fancy here,
if it's an Ubuntu, right?
So, nothing super fancy,
if I know too,
oh, also,
if I do a modular wheel,
I have a lot installed,
because it's a dependency of easy wheel,
but not because I'm actually installing things here.
So, if I do now a source,
this is script,
which is the easy in-evaliation script,
magic will happen if adorone allows us.
Okay, this is expected,
because my installation is a bit broken,
but if I do modular load,
this thing,
magic happens,
and now we have all of these software stack available for me.
And you can actually load things,
and use them,
like Romox,
if I know the memory access help,
it just works, right?
And the same thing,
if this is a development environment,
we have in our company,
also again,
in-term machine,
but this is a,
this is a rocky Linux,
so better choice of operating system there,
if I know again,
okay,
if I do modular wheel,
not even almoded here installed,
and if I now do,
the source of the script,
okay,
it actually picks more things here.
I don't know what's happening with my local installation,
but okay,
you will see that it detects the architecture,
that it's an inter-machine,
and also the micro architecture.
In my now-do modular wheel,
we have the same software stack in both of the machines,
and now I come back here,
and thank you, can it?
Okay, this is the backup in case it didn't work,
but it worked, so I'm going up.
It looks like magic,
but it works,
and you have seen it,
and this is the way it works.
So, it's like a layer design,
you have your machine,
architecture underneath,
it can be anything,
from IntelliMD,
RM and Rix5,
yes,
it's that possible, it works too.
And on top of that,
you have the host operating system,
which is the one providing the,
the drivers,
that can be GPU,
network,
whatever,
also the resource manager,
usually is the norm,
and then on top of that,
you have the file system,
the year,
which is Serenbe MFS,
which is another open source project,
we have developers right here,
who do us something to them.
This is the infrastructure,
to distribute the binary,
not the binary themselves.
And then on top of that,
we have the compatibility layer,
I hope that with this audience,
I don't have to introduce what is it can do,
but basically,
you call Instality in a particular prefix,
and so it's like you have a container
without being a container,
and now,
all the software installations,
we do in the software layer,
these are the binaries,
they all point to this compatibility layer,
so you make sure that all the GLFC,
is the same one,
and that's how we make it to work for every operating system,
and machine in the universe,
probably.
And we build this software with easy build,
of course,
L mod,
and then there's ARX spec,
which is this library that helps us detect what's underneath.
And we test all of these thoroughly through refame.
This is the infrastructure we have,
to distribute the vanities,
it's all hosted in the University of Ukraine,
and then it's replicated through various servers,
in Europe, US, East, and US,
and anyone in the UK, I didn't know about it since this morning.
And basically, everybody who has easy install,
goes to fetch that there onto our top proxy.
So you can see everything is in Europe,
everything is nice, and trustworthy.
Okay, this is not going to be an actual demo,
just to show you that,
you can install it.
With this very simple, easy commands,
the thing here is that,
obviously, you need to be the administrator of the machine
to be able to install it,
which if it's on my laptop,
it's not a problem,
but if it's in HPC cluster,
you will probably have to fight a lot of people
to get this installed.
So, what if we don't have this privilege,
there are many, many ways to install,
well, to access easy,
you can do it through a container,
you can do it through,
to be my physics,
which is provided by the same file system,
and some other tools,
I could be and I were talking about them,
but you have the link in our documentation.
So there are ways,
there are ways.
Even if you have access to a cluster,
but maybe the computer knows
don't have internet of things like that.
So now, the actual new releases and developments,
we have been doing.
So, when we presented the project,
it was a pilot,
the pilot of this year,
your repository died already,
and the production ones are the software,
the DC.io,
and the default one has been in 2023,
the one we released then,
and it has more than 1,000 software packages installed,
that's per target,
that means that the total it's 16,000,
but it's a bit already,
the software that's in there,
it's compiled with GGC up to 12.4.
So now we're working on the new release,
the 2025 one,
it's already available,
it has already 500 software installed,
that means 7,000 in total,
and it's everything compiled with GGC 13 up.
And then we have the dev.easy.io,
that's the developer one,
but I will talk more about it later,
and we have the risk five one,
which it was for the risk five software,
but it kind of died,
it was sort of a developer one,
so more about the later.
This is the evolution of the software window,
you can see we have been doing quite a steady progress,
this is for the old stack,
and this is for the new one,
so I hope we can reach soon the number of installation,
we had in the default one.
And this is a nice good thing with it,
like I worked with the software we have installed in easy,
you can see we will love easy build,
we hide the software we have more versions of,
but it features other superstars of scientific computing,
like MPI, Python,
I want to say,
but it's here,
you know,
all these tiny kind of packages,
because it's not only big scientific software installations,
but it also has a lot of panels
like Python,
now we feature much more modern and nicer architectures.
And it's really important that we support as many of them
possible because the plan is to support all the machines
already available and in the future in UHPC.
And who knows about UHPC a whole lot of you.
Because then a work from our sponsor is that UHPC
it's like the father project that gives money to all of us.
And especially to countries to put their own super computers
and also to the projects we work.
This being said, whatever the GPUs,
well, the support is in there.
But it's a bit complicated, you know,
because Kuda has a license that doesn't allow us
to redistribute it, obviously.
So we had put two helper scripts
to help you like fetch the Kuda from the system
and inject it into easy so we can find it.
And we have actually Kuda enabled software in easy
like Kuda samples, Gromax, Express, blah, blah, blah, blah.
And it doesn't tend to flow, I'm quite sure they are there, right?
No, okay.
But I've seen issues.
So, okay.
So this is complicated, but we are working on them.
And about AMD, it's also working progress.
And it's nice because you can have combinations
of AMD CPUs with Nvidia GPUs.
And it does not only support a wide range of CPUs,
architectures, but also like the interconnect
through the machine, like in feedback.
And as I said, the goal is to support all of the UREG PC systems,
present and future, because for example,
Jupiter, it's not in production yet,
but we build all the software for the Nvidia graces.
So easy will be there with a less of a ready for one.
The machine is in production.
Something that also got a lot of love and attention
and it's the build test deploy bot.
And this is the whole CI infrastructure.
We have four adding software to easy,
like UREG, PPR, too easy.
And this bot will start, well, we command the bot
to start to build for all the targets we have.
The bot will build them, it will test them,
because it's important to us.
And if everything is okay, the bot will also deploy it
into the repository.
And it's a lot of runners running into a lot of machines
that can be each basic clusters that can be cloud environment.
Also, yeah, sorry, I got the tracking mentally.
But it's important to note that the bot
doesn't obey to everybody.
It's really important to be secure so it only.
So if you open a PR, you cannot do this.
We need to do it.
OK, so I said software testing is really important,
part of easy.
So easy will, but it does its own sanity checks,
but we also do a lot of regression testing
through the test suite.
And that's a lot of different tests running constantly
in different infrastructures.
It can be like, it's basic clusters we have access to,
but it can also be like VMs or cloud machines.
And it's better to show you also, but I can manage.
I can manage, I can manage.
This is dashboard that is easy.
I know you can select the benchmark you want to run,
but that you want to see the data off.
And you have this bulk kind of painting here.
But so you can see like, this is for grammar,
but maybe for something else, like TensorFlow,
we can see better if it loads.
Well, anyway, any point, it's like the result
of that benchmark for that day in that machine
in particular, and you can see how much it took.
And this is not actually a scalability thing.
This is like a performance thing.
So the flatter the better that means that all your machines
is running fine, you have no surprises.
No, no, no, no, your notes, it's sheety or dying.
So it's really nice.
And you can obviously filter out of this mess,
so you can see it properly.
And going back, OK.
This is what I just said.
And it's also, we also accept new systems.
So if you are interested in two days,
constant benchmarking, please let us know.
Also, of course, it's an open-source thing.
You can open a PR in case you want to see
some software added to it.
We have a contributing policy, though,
so we ensure some quality.
And what if you want to build on top of this,
but you don't want to contribute to it,
that's also pretty valid.
You can do it with easy build or throwing yourself
into the void and risking to compile it manually.
So with easy build, it's really easy.
You just do module load easy extent.
And the environment will be set to you
to just start building with easy build.
And if not, you can load the built-in module
and that with load air-path wrappers
and all the weird flags you have to set
like black, black, blah, blah, blah, blah, blah.
But we didn't test this too much, OK.
So in case somebody, if you want to try it
and give us feedback, then it's also really valuable for us.
And what about if I want to build to use a spark
on top of easy, that goes later?
Now, that is easy, that is the development repository.
And it's really nice, because probably,
most of you are developers, you have to test your code
in a lot of your HPC machines, or maybe just different machines.
You can start front-commit.
You can start patches.
It's not meant to be in production.
You can also skip CPU targets if you're not really interested
into testing there.
And it has just some different folders structure,
so you can put many versions.
So it's really, really nice, because you want to test your software.
You just push it into easy.
And in some minutes, I was whatever.
You will have it already available in a wide range
of machines to test it.
And for the moment, it's only available with the code.
We have within the multi-scale project,
which is the one that gives money to us
to develop all of this.
But maybe in the future, we accept more.
Also, further integrations, and I will speed run these
because I have only five minutes.
This is what I said before.
The aim is to have them in all the year-to-year machine.
We have them in the Caliomare, Nostromio,
Nardo, disco, Verde, Carolina, Vega, and Jupyterie
will be available since then, one, and then in Lumi,
and Meloxin and the working progress.
So a good job for the moment.
But thankfully, to the European Federation platform,
which I don't have time to explain,
is it will be the official software stack
for the Eurich PC machines.
If you want to know more, there's a series of webinars
starting next week.
You can attend them for free, just register,
or ask me later about what it is.
Also, is it has been integrated in the European Open Science Cloud
that's another kind of computer source?
As you can have if you're a searcher in Europe,
is it just there?
Actually, don't know much about this one,
because I haven't used it.
And then, how we have easy now inside open on demand,
which is a web desktop-based application,
it makes HPC access really easy for new ways.
And super, super nice.
I could be also now, we're talking about how great
open on the money, but we don't have the time.
So if you want to know more, just ask me.
Later, and now, Spark.
So not only running kali every time I give this presentation,
somebody asked me about Spark,
so at some point we started working about building
with Spark on top of easy.
And it kind of works.
Now, we have a proof of concept in our blog.
You can visit it yourselves and read through it,
because this is also a talk worth an hour.
But there's the working progress.
It's coming numbers.
And our training is in this administration.
We always find that people doesn't know us very much.
So we have been asking our asses to talk about easy
and spread the word everywhere.
And one really nice thing you can attend
is the happy hours.
It's every Monday from 2, 2, 3.
It's be a Zoom.
We just give some, we talk about easy,
some late, some topics, some updates.
And then we just have an open discussion.
They're also bloated in YouTube,
if you want to see them for more sessions, I mean.
Also, webinar series, we have done this one last year,
and we plan to do another one this year.
So you can deep dive into the topics of easy.
There are quite some of them.
And they are really like long videos.
They are also in YouTube.
Also, we have, as I said, I have,
we have traveled the world and the seven seas
to spread the word of easy.
This is just some of the dissemination
if we have done, and probably more to come.
And of course, winning the HBCYR to the readers' choice
of words in my brain, it's last year,
but it's actually too used to go.
Super super nice, especially because there's
the one that people vote, not decided by the magazine.
And if you have any problems or whatever,
take a picture to this, you can open a GitHub issue.
And it doesn't actually have to be a problem.
You can also sense the gestions,
report problems, request things.
We will be there to help you, thanks to the multi-scale COE,
that gives money to have a super team.
And okay, this is the one you actually need to take a picture of.
You can join this, like you can look at the documentation.
You can look at the block.
You can have the paper, and you can look at the YouTube channel.
We have a lot of resources there.
And well, this is our test socials for the multi-scale project.
So I will leave this one.
So because I see phones taking pictures.
So I think that's it.
And we have time for questions.
This kind of speed will not only work with Spanish people,
but in a picture of the questions for Helena.
Go.
Can you go back to the slide with the card?
Sorry, which slide?
Is the slide in the card?
Ah, yeah.
The question is if I can find it.
This one.
So there is a measure mentioned there.
Which is the software is actually given by Nazan.
Because my colleague's so partner with the list
about anything that we see next time to this.
So the question, if you want to torture yourself,
if we see make you can do it.
And you're like, let you choose.
But it has to be careful how you can configure it,
because sometimes you can configure it one way to see make.
And then if you use Meson, you lose that.
Sometimes it's a bit picking, as well, what makes sense.
But we see it more common.
It looks like people are moving away from,
see make for some reason.
But good.
Some unknown reason.
OK, go.
What does the open and the monification?
It's really, really.
Oh, sorry.
What does open and the mandu?
It's a web-based interface.
So basically.
So what does the easy open on the mandu?
It opens a desktop, which you can, OK,
there's like this easy general up.
And if you can see at the module list,
those are modules from easy.
So it just opens a desktop that's
interactive to use graphical desktop.
So it's really user, like beginner friendly.
And you have like all the materials from easy to have a loop.
OK, some materials.
No?
OK, go.
I have a piece of software that I've got to add to it.
But why not you open the desktop, and the software hasn't
been updated for multiple years?
OK, I have a piece of software that I've got to add to it.
OK, so the question is, if we can basically
put very outdated software into easy.
I mean, why not?
If you have the need to write not, but yeah, if,
I mean, it can mean that why not, just why not?
OK.
I would say the biggest issue is if it builds
with a recent compiler, or if you can make it build with
the recent compiler, fine.
If somebody still needs it, they want to use all junk
from 10 years ago.
I mean, maybe it's important to reproduce papers,
or what, but if you can get it to work,
and if it's open source, then sure.
So, we need a question.
How do you determine what software it reads for you?
Even suppose are two or three users,
would you do it, or what are criteria?
The question is, what's the criteria to reduce the good software?
I think the main software here is the licensing thing.
Like, if it's open source, whatever, just why not?
If it's close, then you won't be able to do it, basically.
And the work environment for like, at least,
so much interest, or at least so many users?
No, not on this one.
No.
OK.
What's the question from the user to my documentation?
Is there a question from the user to the user?
So, the question, what's the process to add?
The new microarchitecture?
I actually don't know this one.
So, painful as part of the answer,
because everything we've installed up until now,
if we introduce a new CPU architecture
and do optimized builds for that, we do everything again.
Now, thanks to EasyBuild and the fact that we know EasyBuild
well enough, we have scripts that help us to harvest things
and build things in the right order again
and make sure it all works out.
There are hiccups, though, when you're targeting a new CPU target
and you run a test sheet of say, NumPy,
there's often surprises in there.
And we do what we can to figure out the failing tests.
Is it the problem with the test?
Or is it really an upstream bug?
You need a patch to fix that issue.
So, there's a lot of these hurdles that we have to overcome.
We've now done multiple times.
We'll do a new V2 soon.
Actually, new V2 is a bit complex
because what you actually want is you want
to like, grab it on four.
And as it Google Axial, as any of V2 and then Grace,
you actually need several builds for each three of them.
For each of them.
It's not that easy to come up a new V2
that if you fully optimize it and use all the instructions
that you can, you're like missing things out.
So, you have to build, be already built for Grace, separately.
But you can not use N1, University 1, University 2.
There's an issue that I can point you to.
It's a lot more complex than that.
So, it is a painful thing where, set, grab it on four and five
is two that we want to do quite soon.
The other one is N5.
So, the AMD Turin and it's just work.
We build again from the bottom up and we power it.