Hello, welcome everyone, so thank you to the organizers, well, for organizing the
Devroom first and for having me to, it's a great to be here again, it's not the first
time, like three, four times, five times, I don't remember.
Anyway, so my name is Ludovic Cortez, I'm a research software in January at Teneria,
which is French Research Institute in Computer Science, and I've been doing stuff with
package management and with HPC for a while now, in particular I've been a contributor
of two gigs for a long time, yeah, a long time, so today I want to talk about package management
and putting it in the hands of users, meaning in particular in the hands of users on HPC
clusters, right? So, you know, I've not always been an HPC person, I don't, I don't think
I'm actually an HPC person, I don't know. But, you know, when I first got introduced to HPC,
like probably many people in this audience, I discovered modules, you know, environment
modules, and they are great, and I'm not saying these just before Xavier's previous
speaker, maintainer of environment modules. I mean, it's actually a very clever hack, it's
more than hack even, because it gives you a lot of flexibility on your development environment
and everything, and so everybody who uses an HPC cluster these days and even a long time ago
uses modules, right? This is great. Actually, did the modules paper, the original papers from 1991,
I think I checked it before, so it's been around for a while, and it's sort of timeless,
people are still using it in 2026, and even there's a talk about EC, I think it's called
afterwards, which is the European environment for scientific software initiative. It's still,
software installation, sorry. Anyway, and it's based on modules too, so it's really,
it does a great job in sit-around. But it also has some shortcomings, so in particular, if you go
from one HPC cluster to another HPC cluster, well, maybe you'll have modules with the same name,
like OpenMPI, but maybe they'll have a different name, and maybe you'll get different versions,
or maybe they're compiled differently, you know, so you never know, or maybe you would just not
find those modules on that other HPC cluster, and that's kind of annoying. And also, if there's
software that's not available as a module, you'll have to go ahead and compile it by yourself on
the machine, and, you know, I mean, if you're working on a complex software stack with C++,
with scocals, for example, you know, things like that, you know, bidding stuff by hand on the
cluster is kind of annoying, right? So, the way I saw it, as soon, pretty much as I discovered modules,
that well, maybe, I mean, why don't we have it too, like, you know, like the typical free software
Linux version, I was used to APT from Debian, you know, or you know, things like that.
And I was like, why don't we have something like APT on clusters, right? That would be ideal.
So, I mean, of course, there are a few differences, so in particular, we need more flexibility,
because in HPC people just don't take, you know, software packages, other shelves, sometimes they
want to customize how they build packages, right? You know, sometimes in its specific versions,
etc., etc., and APT typically doesn't provide that level of flexibility. And in portability,
so ideally, you know, I would like to be able to go from one cluster to another and run the same
package manager command and get, you know, the same software, right? And yeah, reproducibility in
the sense of being able to run the same software, you know, on different machines, and also at
different points in time, because with modules, for example, if I even if I just choose one
single cluster, all the time for my experiments, maybe I come back to the cluster six months later,
and the modules I use are gone, or maybe there's been a great idea or reconfigured, you know,
you never know, so I might find myself unable to rerun my computational experiments, and that's
kind of a problem. So I thought, why don't we have a package manager for that? And well,
it turns out that I had been working on Giggs as a hobby for a while, and Giggs is a package manager,
and within the Giggs community, there are quite a few people actually contributing to Giggs,
but also doing HBC stuff for a living, right? So with my friend Ricardo Vermous, a contributor to Giggs,
at some point we thought, well, look, we have a tool, Giggs, and it actually has good properties for
HBC, right, for using it on an HBC cluster, and we thought, well, maybe we should use it and promote it,
as a tool for HBC clusters, and that's how we came up with this paper about having reproducible
and user controls of their environments. With a package manager, that people would actually use
directly, right? It turns out many people had the same idea at around the same time, right? So you've
probably seen this project before a source pack is a bit counter, there are package managers,
app owners, it's not a package manager, it's a container runtime, but anyway, many people
felt the need to have package managers in place on HBC clusters at the time.
I get back to this a bit later. So for now, I'm going to talk about Giggs specifically in HBC
and what we did, what we managed to do so far. So in terms of availability on clusters,
it's available in a bunch of clusters, probably not all of you don't know about this cluster,
did I relatively small medium sized clusters in France and in other places in Europe, in the US too,
but we have Giggs directly installed on these machines, so people can actually use Giggs directly
on these machines. You might notice that there are no big clusters here, right? The T-RUN, T-Z rock
clusters, meaning national scale, or European scale clusters, we don't have any of these on
this list, I get back to this later. But anyway, so quick short of France, who knows Giggs already?
Like the entire room, okay, I'm going to skip quickly, that's great and believe it will.
So on these machines, as a user, you can go ahead, you log in and you can directly run Giggs
shell, some package into LMPI Binchmarks, for example, and then you run, you know, you use Slurram
and you run your command. And this particular command comes from the environment set up by Giggs
shell, right? So basically, instead of doing module load, you use Giggs shell and it's kind of
similar, you know, it sets up the environment and you run the command directly from there. And
this is wonderful because from, you can go to any of these clusters and you go ahead, you run
the same command, you get the same software environment basically. So it works pretty well.
And because Giggs is a package manager, well, it knows about the packages and it knows how to
rebuild those packages and to modify them in all sorts of ways. So we have what we call package
transformation options. And for example, I can tell Giggs, well, please take that into LMPI
Binchmarks package, between place, everywhere in the graph, up an LMPI by MPH. And it will rebuild
it, if needed, and if it's already there, we'll find. And from there, I can run my benchmark, you know.
Well, then we have manifest, we can store, you know, you can describe basically in a find the
software environments that we want. And the cool feature is basically if you need to
remember something about Giggs is that we store files and two commands, you can have a fully
reproducible environment. So the Giggs is quite common here captures exactly the precise
revision of Giggs you're using. And the Giggs time machine command takes that
that information and reproduces the exact same Giggs, from which we're using shell to reproduce
the exact same environment. All right, what about performance? We have a good tool, but if
performance is not here, that's kind of a problem. And people were rightfully, you know, wonder about
this in HPC circles. So first thing is MPI, we did quite a lot of work over these years, you know,
about MPI packaging, because that sort of the one of the key things you want to get right. And so
typically, I experienced details, but we have an open MPI package that can, that is built to support
all these kinds of networks, that we actually encounter on this diversity of medium sized clusters
I mentioned before. And our MPI package is able to, you know, at the time select the right
back end. So if you run a slingshot machine, it's going to use that. If you run a normally path
machine, it's going to use the right back end. And this is not, this is something open MPI
doors for us, right? We just package it right essentially. And speaking of slingshot, well,
we do, you know, there is this misconception that you're never going to get the right performance
if you don't use the vendor provided MPI like create and pitch for example. And this is something
we run and not asked for. This is a micro benchmark, you know, just ping pong. And we get the same
performance, because there's no reason it should be different. We also run not just by micro benchmark,
but also actual application. And it just work, you know, we get the expected performance
because we're using, well, leapsiex side in this case for slingshot. And by the way, I'm glad we
contributed to having leapsiex side available as free software, which was in the case before.
So the story here regarding MPI is that I think we can bring our own MPI on the machine,
right? It actually works. That's also something container people wonder about. Can we bring our
MPI? Is it really going to work? Do we are we going to have the performance? Well, yes,
and it works and there is no surprise, you know, you don't need to do some guest work regarding
ABI compatibility, things like that. You have a world-tested software stack, you just bring it to the
machine and it works as expected. That's a story. So that's one aspect of performance, but then
there's also, you know, micro architecture. So we have this package transformation,
dash dash tune that we can use to say, well, please build this Agen benchmark package specifically
for Sky Lake. For example, that's on my laptop. And then, well, it gets rebuilt with the, you
know, the dash MRH flag. And you get the expected, you know, level of performance. And likewise,
now we're doing this for AMD GPUs. This is brand new. Actually, the pull request was merged like a
week ago. And this is very nice. So you can say, well, I want this package, but I want it to be
specifically built for this and that AMD GPUs. So the name are not really human-miniq for, right?
But there's, there's documentation. It's like, am I instinct 250 something? Anyway,
and this is pretty nice. So I did mention before that Giggs is not available on, on the big
supercomputers, right? Like, national level European level. So how do we do? Well, we can actually
use this Giggs pack command to bundle up our software stack, you know, create a bundle. So in this
case, it's going to be a table, right? Because we know this software stack is well tested. So we
created a table that contains Intel MPI benchmarks and all the dependencies. And then we can basically
SCP it to the to the target supercomputer. We ship it entirely. We extract it and form there.
We can directly run our stuff. So in this case, running the Intel MPI benchmark,
you know, using all the software stack that was packaged in Giggs. So including Open MPI,
including OLEDs, dependency, all these dependencies, all the way down to Leipzig. And it actually works.
So in short, what we are doing now is that we're using Giggs. You can use it on your laptop.
You can also use it on those tier two clusters. Giggs is already installed. But if you're
targeting one of these bigers, one of these bigers, sorry, supercomputers, while you can use Giggs
pack to create a table, an obtainer image, or a Docker image, and ship it directly to the machine.
So it's pretty cool, but it would be even cooler if these machines were running Giggs directly.
All right, there are demos and tutorials. You can look at if you're interested in how this
actually works in practice. And people are actually using Giggs, I should say, in a number of different
fields, you know, it's not just a geeky thing, right? People are doing research using these things.
All right, so this was sort of the dream part as I see it as some of a seed because, you know,
using the package manager directly on the machine. But what about reality?
Well, reality is we are still using modules, as I said before, right? So if you go to the documentation
of all these major supercomputers, they say that the first thing you have to do is use modules
to get the officially supported software packages, right? And I was like, why is it still the case in
2020-26, you know? Because SPAC, for example, supports and geeks actually supports and privilege
user installs, you know, you're supposed to be able to bring SPAC or easy build on the super
computer and to run directly, you know, SPAC install, etc, and get your packages there.
And there is a story I keep hearing about, I note, quotas, so maybe it brings a belt to some of you.
Well, if you start using SPAC install in the supercomputer, for example, then you're likely to
run out of vinyl because they're the quota, right? And if you build a lot of software packages,
well, maybe you're going to use a lot of this space and nine notes. And this is from a user
community report of one of the French national supercomputers. So maybe that's one reason.
And this is a solution to this, as I understand it, and there's going to be a SPAC talk right after
me, so maybe we'll learn more about this. But anyway, one thing, a supercomputer, and it means
to is that they provide, for example, SPAC or is built or both pre-configured and also in some
cases, the SPAC module, they provide because it's still a module, right, is changed to the
two software that the admins build with SPAC on the machine. So what that means, as I understand
it is that you're not going to have to rebuild everything because this SPAC that you're going to use
is changed with, you know, step that was already built. But you're still stuck with one particular
version of SPAC or is built that's provided generously to you by the admins, right? You cannot
just, you know, pick the version that you want. And I did a quick survey here, and, you know,
some of the, so the first ones are the Euro-HPC clusters, some of them, some of them provide
both, some provide just one, some provide none of them, but the typically provide
obtainer or something equivalent. And the last ones are the French clusters, which are maybe
a bit more conservative here. The quick show of hands, who regularly uses a super computer in
this audience, right, the vast majority, who uses a package manager on super computers.
Yeah, that's, that's still a lot of people. Okay, because I was curious about this,
who I asked people operating a dust ray in France, so people at CNS, and they got me this data
about SPAC usage. So it's, they provide two versions of SPAC on a dust ray, and this is,
this is one of them, and they have statistics on who basically does module load SPAC. And that's,
like, we're talking about 1% of the 1200 users of a dust ray. Again, this is just one example,
maybe it's different if you go to Lumie or Jewels, I have no idea. But I do have a feeling that
package managers are not so widespread, still in HPC, at least, not in the hands of users.
Well, yeah, people in Seoul are using a package manager, but we can have that discussion
afterwards. So yeah, all right, so so what's next? What can we do about it? Actually, I have no idea
to be honest, but I can answer at least for the geek side of things. So I'm, I'm part of
NAMPEX, which is the French HPC program from the French research program for HPC, and we get to work
with a bunch of people in academia, mostly people developing scientific applications, typically,
so users of HPC clusters. But we will see get to talk and to work with people who operate
clusters, and that's kind of interesting to me, because I would like to understand, you know,
what's happening? Why are they not promoting package managers more than this? And we learn from it,
so, and by the way, we're hiring, so if you're interested in all this, you can check out the
HPC.gigs.info page. So one thing, for example, that we learned is that from this perspective,
geeks is a showstopper, because it has a build demon that runs as root, right? So if you're not familiar,
while when you run a geeks command, it makes remote procedures to that build demon, that build
demon might well be software or download tribute binaries, or do nothing if it's already available locally.
The problem is that this build demon was running as root, the good news is that it's now fixed
because, well, the unprivileged user namespaces and Linux, which is what enable this work,
are now widely available, even on supercomputers, and so we developed a rootless variant of the geeks
demon, and that's much better, you know, from a security perspective, for HPC CIS, that means
it makes quite a big difference. And so we have some good news, thanks to this, to additional work
we've been doing over a few months now, geeks is coming to one of these supercomputers to a
dastore operated by CNS in France, which is, you know, one of these tier one clusters, so
soonish, like in the coming months, and we'll see how it goes, I hope it's going to be fruitful,
I think it will be, let's see. So I think it's time to wrap up. My impression is that we have
package managers and in particular, spec and PCB, they are the stars right in the room,
because they're very popular in HPC circles. They're very much used by system administrators,
on HPC clusters, but they were not supposed, their audience was not supposed in my opinion to be
system administrators, right? I think the goal, you know, back in 2015 when we were young,
was to put those package managers in the hands of users, and there is some success according to
a quick poll remaining this from, but I think we can do better, and I hope we will actually
make it a reality. And this is it. We have time for questions.
Questions, anyone? Yes.
What's the thing? What's the barrier?
The question is, what's stopping me and people from using gigs directly on the cluster without
asking anyone? There are several things about this. One thing is, when you install gigs,
you have to run this build-demon thing. It runs root-less now, but the problem is, if you install it
by yourself on the machine while you're going to end up rebuilding lots of stuff, right?
And it's going to take ages, and you're going to run out of high notes. So this is not really
practical. So instead, you know, when system administrators install gigs, system-wide,
then everybody can use gigs and the storage and the computational course is shared among everybody.
So, for example, if I install GCC and somebody else install the same GCC, it's going to be stored
once on the machine, and that makes quite a bit different. Yeah, it's much better.
Ah, that's a very good question. How do we assure that people are using gigs typically
correctly on the supercomputer? That's a very good question. So, I don't have a very good answer
to be honest. So, for example, when I mentioned spec user and disabled user, typically system
administrators provide pre-configured versions of spec or easy-built. And so, they can sort of avoid
the problems, I guess, some of them. But in the case of gigs, well, I mean, what we do is provide,
for example, that generic of an MPI package. So, we know, we can just ensure that it works on all these
networking tokenics, but then people can still shoot themselves in the food, right? So, I don't have
very good answer to that.
So, the question, if I understand correctly, is how would I run proprietary software with gigs? Did I
install and manage? Okay. Well, so, gigs allows you to install software, be it proprietary,
or not. Obviously, I'm not a fan of proprietary software, but we have people using coulder,
obviously, on HPC clusters. And so, we have coulder packages, for example. And at the command line,
never, it works in pretty much the same way as any other package. You don't see the difference,
except that it's good. So, it's a bit clunky, but it's not a problem.
Can gigs be combined with the modules already available on the system? No, it cannot.
And that's one of the fundamental differences between gigs and USB exhaust pack. USB
and spec are specifically designed to allow you to combine, you know, spec-rated is a bit
provided packages with modules. And not just modules, but everything that happens to be in
userly, for example, which can be nice, but the downside is that if you go from one machine to another,
well, you're not going to have the same modules. So, how do you, you know, deploy the same software
stack, you cannot really do that, right? And so, instead gigs, you know, controls the entire software stack,
and so, everything is handled by gigs, which means you can also test the software stack before you
actually start choosing it on those machines. Question?
Does a bill demon has to be on the same machine as gigs? No, it doesn't have to be. Yes. So, in
practice it's on a cluster, there's one dedicated node for the bill demon. That's how people do it.
Probably you're wondering about the security thing, right? Yes. Well, the security thing. I mean,
it's some typically people on clusters installed on a separate node, dedicated node, but still,
you know, there's also securities also in part about paperwork and, you know, people are more
comfortable with having a rootless demon, which I mean, is also understandable generally speaking,
I guess. Yeah. Yeah. Yeah. So, the time is it basically, is it a security risk
users installed stuff and others use that stuff and that stuff is not secure, right? I don't think so,
because to reuse the software, you have to sort of do it on purpose. Like for example, the, you know,
the Intel MPA benchmarks where I customize and swap to an MPA by MPH. Unless you explicitly ask for
that, you're not going to get, you know, the fine name of that binary, so you cannot, you won't
end up running it by mistake, right? You have to explicitly ask for it to actually run it.
I guess we have to wrap up, but we can, no, I didn't speak here. So, question.
In the rootless bit lyrids installation directory is still a shadow, a new store directory, yes it is, yes,
that doesn't, does the demon have right access to the new store directory? The demon
has right access to the new store directory, but only the demon does. So, as a user, you cannot
write into that directory, only the demon does. It's a trusted third party.
Okay. Thank you. Now of a spark.