All right, we'll get started.
The next speaker was supposed to be at the aim, but he couldn't make it.
So Mark is filling in for him.
And he's going to get a talk from the environment of this boy.
So, hi, everybody.
I'm Mark.
I'm working at CES.
So, as Kenneth said, let me change.
So, please be lenient with me.
Xavier and I are working at CES.
We are a French government institution, a research institution.
And we are operating the HPC systems and a language.
Alice Rokok, that would be the second,
a European-exescal system that would be installed in two years or so.
So, today I'll talk about environmental modules.
And we discuss why it is all ID, is still useful today,
and what's next for the project.
So, for those who are not familiar with module,
module or environmental module is a tool that allows you to manage your environments within
shell session or shell scripts.
The building blocks of module are script files that are called module files,
in which you will define your environments.
So, here is a very simple example of a module file.
And then, if you need a software or library or something like that,
if it's not available in your environments, you can load it with the module load command,
and then it will be available in your environments.
And you can undo this seamlessly, so you can unload the module,
and you go back to a clean slate without this software on your environments.
So, how does it work?
There are three main components behind the hood.
The first one is module cmd.cl, so it's a TCL script
that will evaluate the module file I showed you earlier, so the first one.
It will evaluate it, and it will scale out the environment
and change that are described in the module file.
So, here is an example.
We will evaluate the content of a module file for bash scripts.
We will use the subcommand load, and we will do it for the module file full.
And you see that we will update paths, and we will define some variables that will be useful internally for module,
to know what modules are actually loaded.
Then, there is a second component that is a script function named module.
It's where the magic happens, and basically here we are just doing an evaluation on what was spelled out by the module cmd.cl.
So, we are just calling this the first script I showed you,
and we will evaluate it, and it will update our environment.
So, where module is used for?
The first usage main usage probably is to access software catalogs on HPCC systems.
It's very handy when you have different versions of the software version 1, 2, 3, and so on.
Our different versions of the software in terms of compiling tool chain.
So, it's on most super computers today.
It's also interesting because it's a shell-agnistic environment.
So, whatever the shell you're using, being bash, TCSH, fish, it will work.
And you have a common interface to access your software's libraries.
And one thing that is interesting, it's mostly used to handle software, but actually it's an environment tool.
So, you can manage your environment, and since you can interact with your environment,
there are lots of libraries or software that are configured by environment variables.
So, you can actually manage your configuration thanks to module.
And that's what we do for example, it's here with MPI.
So, module is an old story.
It's starting in 1991.
It actually predates both Linux and MPI, so it's really old.
It went through lots of hands across the 30-plus years of its life cycle.
And in 2012, unfortunately, the project was an halt for a few years.
And then it was taken over by Xavier.
And in 2017, you released module 4, and since then, it's in another active development.
So, you can find it at GitHub, or of course.
Module 3, the TCL is about 15,000 line of codes.
It's in our GPL 2 license.
There are two new feature releases per year.
And within those releases, you will have five significant new features.
So, those features are impactful for the users.
They are really interesting at new things.
And about 50, minor changes every time.
Module development is based on a test-driven development approach.
So, there is a huge non-regression test base with more than 22K tests,
with a 99.5 coverage, which is pretty good.
And the test suite is six times the module CMV.tcL, so huge test suite once again.
And an important point is that the module project is really listening to the community,
to the users, to the C-fnmns, and to the nearby communities like SPAC, EasyBuild,
Almod, we are really integrated within the communities.
Here is a non-complete list of new features in 2017.
If you want to have all the details, you can flash code this.
And I will just go through two features that might be interesting for you.
The first one I want to mention are Viens.
So, the basic idea behind Viens is that the module files are scripts,
and since they are scripts, you can pass arguments to your scripts.
And with the arguments, you can interact with the way you will set up your environments,
or the tool chain you will use, and so you can do everything in a single module file.
So, it's much more handy that what we are doing previously, which lots of module files.
And we are using the SPAC terminology.
So, those who are familiar with SPAC variants, it's very similar.
We have Boolean variants and valued variants.
And here is an example.
So, imagine that food is actually HGF5.
If we use the M.C. variant that is Boolean variants,
it would mean that we would use the parallel version of HGF5 instead of the second one,
and we can specify the tool chain so the way HGF5 is compiled.
In terms of a module file, the implementation is pretty easy.
You define a module, then you can evaluate and you can change your environment accurately
to the variants you are using.
You can use shortcuts, so it can ease again the use of variants,
and you can have a syntax that is really close to SPAC.
We also introduced activity logging.
So, previously, it was possible to have some logging with hooks,
but no, it's really out of the box.
The interesting thing is that since we can push logs within the log system of the system,
then we can rely on a traditional stack like ELK to have statistics on the module usage.
And here is a very simple example, so you can define the logger,
which command you will use to log, and you can log different kind of events.
So, the command you launch, the evaluation, if you have auto evaluation,
and here are examples, we launch a few commands.
So, you see that we trace the command available, the command load,
and then there is an evaluation for loading the app that we see also here.
I have to mention that an environment module is not the only module implementation in the wide.
There is also ALMOD.
ALMOD is written in LUA, it was the development started in 2008.
It is developed by Rob McCliff from Attack.
And the main difference between an environment module and ALMOD is that ALMOD can evaluate TCL and LUA module files.
ALMOD is actually very popular across HPC sites.
And there were many features, very interesting features that were introduced when module was not.
So, at some point, it was really in advance to compare to module.
Among those features, you have module hierarchy, module cache, and so on.
And as of today, it is probably the most used and the most deployed module tool in the HPC world.
I have also to mention that the both project are recolaborating.
The idea is that the user experience will be the same, where you are using ALMOD module.
One interesting feature I mentioned from ALMOD is the module hierarchy.
So, the idea of module hierarchy is that you can hierarchically organize your module files.
So, you have a kind of first layer of module files.
So, in your module pass, you will have just one pass with core modules, and you will see only that.
And once you are loading one of the modules, then you will update your module pass with a new module pass that will expose other module files.
And if you have a change in your environment, if you change one of the core modules, it will automatically reload the module.
So, here is an example, imagine that you loaded boost, compile with GCC.
If you change the compiler, it will change automatically GCC.
So, we are implementing module hierarchy in module.
It should be available this year.
And actually, we implemented it with the twist.
So, we have an automated handling mechanism.
So, it is a bit for a step forward compared to the module hierarchy.
We are trying to reuse IDs that comes from CISN mean and system package management.
And to be able to deal with requirements and conflicts efficiently.
So, the idea is for the user to have something that is really consistent.
So, imagine you load a first version of a module.
It will load the requirements.
Then, if you load another version of this module, it will handle automatically the unloading of the conflicting module.
And it will load the right module.
So, it is much more easy to use, much more seamless for the users.
So, module is a very well known interface in the HPC world.
It was used for more than 30 years.
So, it has some useful for sure.
But since the 90s, lots of tools to install a scientific software appeared,
along with easy-built SPAC geeks.
You probably know all those tools.
And if we take SPAC or geeks, for example, they actually do not need a module tool to provide their software.
So, they have built-in functionalities to access the software.
So, it is module still useful.
We think that, yes, module is still useful.
Because it is a common interface, whatever the tool you are using under the hood.
It is also interesting for sites that are working in a federation.
So, that is typically the case for your HPC.
We have users moving all across European sites.
And to have a single interface, it is easier for our users.
And it is also useful when you have different tools used simultaneously.
And that is what we do at C actually.
We are using both easy-built and SPAC.
We built our software stack with the tools and we access them with module with a common interface.
And I already mentioned it that module is not only a tool to access software,
but it is an environment management tool.
So, it means that you can manage the configuration of your software.
That is what we do with MPI at CE.
And you can also manage data endpoint.
So, for example, on our clusters, we have users that have different projects.
They have different data on the file systems linked to those projects.
And we can manage that with module directly.
So, glimpse in the future, there are two functionalities I want to mention.
The first one is the idea is to have an adaptable view of the module that you will see.
So, right now, if you launch the module that I recommend that we list the module that are available to you,
you'll see everything.
And if you try to load the module that is not compatible with the module you already loaded, you will have a conflict.
And note the idea is that we want to implement something that if you launch this module that I recommend,
you will see only modules that are compatible with your current states of the current state of your environment.
Another feature that is in the development is to hook, to use, sorry, hooks that are available in modern shell, bash, the usage fish.
If you launch a common that is not fine, it can actually trigger a prompting offering to install the software.
And we could use this hook, so instead of offering to install the software, it could offer to use a module that is installed.
The idea comes from the me project, but the me project is not an adult.
And also, if it's integrated directly to module, we could benefit from the module cache that is already implemented.
And sky is the limit, so we are really open to any suggestion for the community.
For example, if you want a lower support, we could do it.
At some point, module CMD, the TCL, might be a written in another language, like Python or Rust, because TCL is not very popular.
And it can be interesting to use another language, more mainstream language, to help you contribute to the project.
Our long-term vision is that module should be the glue between the different package managers.
As I already mentioned, you can use different package manager and have module on top of that.
And actually, we could even use the building functionality, the query tools you have on those tools to access the software catalog.
And if you want any specific development or expertise, you can also ask that, that is the new Xavier self-company.
So it's offering development and expertise on module and ecosystem.
So it's for very specific development, don't hesitate to ask him, if you need to move from one version to another or something like that.
So Greg mentioned HPSF.
So the module is actually in the process of joining the hyperform software foundation.
We are at the stage of drafting the governance and the technical charter of the module projects.
The project has been moved from the CE, the CE GitHub to an independent GitHub.
Independent GitHub projects.
And the idea is to be able to transfer seamlessly the IP to the Linux Foundation and to facilitate future and over.
And so in this process, we're looking for additional maintenance to join the project.
So don't hesitate to join us.
We're going to start to develop within module but also to be part of the project to be part of the to define the roadmap to make the community lively community with the mailing list, the chat and so on.
So key takeaways module is an active project, it's an ongoing development.
We are trying to constantly add new features.
If you have any ideas, please come to us.
We're very happy to collaborate with the ecosystem.
There are still many new things that could be integrated to ease the life of users and of CCN means.
Once again, we are happy to collaborate with the community and with other projects.
And actually the idea is actively contributing to to easy to expect to help model.
And we're an open community.
And so here are where you can find us on social media on GitHub and we're very happy to have no chat and metrics.
So come talk to us.
Thank you.
All right, we have time for a couple of questions.
Questions that this review will recommend to use the cloud model for activating item values.
There are different things.
Do you think we have the question?
Do we recommend to use a different way?
Can we actually invite us?
So the question is.
Do we recommend to use module for Python environments?
That's a very good question and it's actually a kind of a mess.
And there are very different ways of doing it.
What I can say is what we are doing at C8 to deal with this question.
So as I mentioned, we're using both EVB and Spark.
But for Python, we are building our Python distribution with Spark.
Because we're building everything from scratch.
We want to to compile everything ourselves.
We're not relying on condo our people.
And so we build everything with Spark.
And then we can do a view that is accessible with modules.
So we can module load this Python distribution.
It's not as flexible as virtual environments.
But we have something that is a bit monolithic that we can guarantee that is working.
So I think that's a good way of doing it to guarantee a good stack.
It's guaranteed by the HPC team.
And then you can build your virtual environments on top of that.
More questions?
I have one.
Correct.
If I'm wrong, but I think Elmo the security is more popular.
If you have a patient of the environment, what was concept?
Part of that is because of the history.
There was this was not activated.
Made in for the second.
That's a way to pick it up.
And then made it a very lighty project together.
Do you see that flipping back to this becoming more popular than Elmo?
And if so, why would that happen?
So the question is, Elmo is more popular at the moment.
And do we see an environment module being the most popular modular environments in the future?
So for sure, Elmo is the most popular module implementation at the moment.
And as you said, the history is, can explain that.
And there were lots of new features that were really interesting in implemented in module.
And we are still catching up.
So actually, the very last break for us, the very last feature that is in Elmo that was not in module,
will be implemented this year with the hierarchy.
So finally we catching up, but it took us almost a decade to catch up with Elmo.
So then, which implementation would be the most popular.
I don't really know, but we are trying to make module sexy again.
With interesting features, new features.
We are trying to go a bit further than some things that were in Elmo.
And we, so Xavier is a real, is a system in that core.
So he tried to push a system in ideas within module,
especially in the way of managing requirements, dependencies and so on.
So there might be features that are well adapted to our systems, things to that.
And it might be interesting, at least to try again module if you are using Elmo to see if it can answer.
If you can answer your questions, your problems.
Any more questions, yes?
Yeah, maybe a question regarding support for our screen development.
So if you look at the spec for pipeline, it's very easy to make your own application available.
There's something that you envision for modules.
Or would it then be via the spec route that provides a spec upping package or something like that?
Then I can tell someone if you want to install my application on an HPC system.
It's easy because I did the work regarding modules for you already as an software development.
So the question is, can module kind of replace spec or other tools like that to for installation?
Actually, we are not exactly in the same field, because we are not providing a way of installing software.
So you still need easy build or spark or easy or geeks or whatever.
We are just providing a way to access the software.
And so we are not providing kind of recipe.
If you need your module phase, you have to write them.
But the thing is we are collaborating with the other projects.
So the module can be automatically generated.
So right now, spec you can really use only spark internals and spec commands.
But you can also generate modules, module files with spec and then directly use module.
So that's the idea.
We want to, the idea is that you can have the freedom you want.
You want to stay with spec, good for you, continue using spec.
You want to use module, you should be able to use it seamlessly because we should be able to generate automatically the module you need.
Okay, thank you.
Maybe one more?
Yeah.
Can we, can we deal with different environments depending on the nodes?
And yes, that's actually what we do.
Because we have systems with different architectures and we have to deal with that.
And that's something that can be done easily.
More or less easily, and it will be much easier with variance.
So at the moment, we are doing it at CE using some specific modules and some ways of loading.
Requirements.
And with variance, it could be very easy to do because you could just add a variance and then it corresponds to your target architecture.
And you could have automatic detection of the architecture and load the right modules.
So that's something that is totally doable, yes.
Thank you.