All right, let's get started with Thomas and Drup.
Thank you.
Good morning, everyone.
I'm Thomas.
Thank you.
I'm working for the Ulysses Park Computing Center,
where we are hosting the first XSA system of Europe.
And if you're one of the people collecting stickers,
you might have something for you.
Yeah, it's my first time at Boston,
and today I'm going to talk about Drup,
which is an environment to run systematic benchmarks
and scientific workflows.
So benchmarks, I would say,
is a kind of subcategory of workflows,
so we are executing some workflows in a very systematic way,
with several dependent steps,
and this is usually done by different kinds of user groups.
For example, the system admins,
they use it for procurements to ensure the stability of the system,
or support stuff to detect changes in the behavior of the system,
or to reproduce problems that are reported by users,
or the users themselves to let them run their software
as the functionalities or run the scientific use cases in the end.
And one can do this in different ways,
good or better and worse ones,
manual, so you have nothing scripted.
You do everything by hand,
you haven't bellies anything documented,
you are running into problems,
like overwriting your own data in the worst case,
so that's probably not the best way to do it.
Then you have something,
they miscripted,
very benchmark specific,
this is also hard to port,
and to give to your colleague,
and he has to think what you thought,
how the script should work like,
so there's not no standard at all.
And then there are more generic tools
for configuring benchmarks,
and this is where jupe comes into the game.
Let's start with a little bit of history,
so I haven't been in HPC for so long,
but the colleagues who have started with jupe,
they told me it has started around 2004
with a procurement of a big system we had at this time,
and this was purely pearl based,
and this has been re-implemented in 2014,
with Pison,
and we are currently having the version 2.7.1.
And a little bit more about the history,
jupe has been the chosen framework
for European benchmarking already,
since the first years it was published,
in deja and in the praise,
preparatory phase and in several implementation phases.
And also it has been used in several projects,
nationally and internationally.
Here's a few selection of these logos,
you might recognize one or the other,
but this list is not complete.
So what's jupe actually?
It's generic, lightweight, configurable,
environment to run,
monitor and analyze application execution in a systematic way.
What it means will hopefully get more clear
in the following slides.
And I've mentioned a few use cases already for it,
and they also parameter studies,
production scenarios and more.
We publish that open source,
it's available on GitHub,
but also on other ways,
which I will come to later.
And a few key concepts,
which I will explain in the following slides,
is the workflow creation based on dependent steps,
and parameter expansion,
pattern substitution and files,
we create a generic directory structure
to avoid overwriting of data,
and keep doing things automatically.
So how would a generic,
basic workflow look like?
You have your data,
you have your application, your source codes,
but what you would have to provide is
jupe configuration file.
You either program it in XML or in YAML,
I will have this talk based on XML,
but don't be afraid of it,
there's also YAML,
and if you prefer that one.
And so it's also not free lunch,
so you have to provide this file,
but once you have it,
it's standardized,
and basically it's once you understood jupe,
you might understand all the other benchmarks
that are implemented in jupe.
So in this example,
we have configured our benchmarks
such that it should have a first-to-compiled step,
and this should create two work package,
one,
it should compile the sources with a one,
and second one with a three.
And we would like to have a follow-up step,
that depends on the compile,
and we run a scaling experiment for each of the executables.
We would like to see how it performs on two,
four, and eight thousand CPUs.
And then we want to extract some data from the logs,
and in this example,
we interested in the runtime,
and we provide this either in the table or in the database.
So this is the most basic work flow
that is implemented in jupe.
But in the following slides,
I will go a little bit more into detail how we do this.
So this can be either in jamele or in XML here,
it's in XML,
and you define parameters,
give them a name,
and to value a series,
like a one and a three,
and if you have a comma separate list,
a jupe makes sure that
where you use this parameter,
it creates two work packages.
So one directory for each of these pipelines
that jupes go into execute.
And if you have another parameter,
also with a comma separate list,
and jupes spans out the tree even further.
So just by adding another comma and another value,
you can easily scale it up to
more compile options,
more execution options,
or whatever your workflow is looking like.
And the file substitution is another key feature.
We define so-called subs,
and then we define a unique pattern,
or a non-unique pattern,
jupel search for it in the file that you give to it,
and replace it either by one of the values here,
and your parameters here defined,
or by a static string, for example.
And then it replaces the values in a file.
So you could think about
to have a template of a job script,
and you wanted to do the scaling experiment,
and jupes create three or five or how many,
and jobscripts you want,
just by replacing the values in this template.
The next one was the unique directory structure.
So every time you run juperun with your benchmark file,
jupel creates a sub directory to prevent
that you override your previous runs,
because you won't have them reproducible,
you won't have them documented,
and you might want to look into them at a later point.
So you run juperun,
and we get a new ID,
and every of these IDs you see these structures.
So you have some internal configuration files,
which we can ignore at this place,
a point, and on the top some directories,
each directory is one work package,
for example, the compile with the O1,
the compile with the O3,
and the different executions.
So they don't interfere with each other.
And within those directory, there's more structure,
and at the end there's where the magic happens,
log files, and probably your source code,
your executables,
further scripts, or whatever you want juped to do with your data.
I will guide you through these example scripts.
I hope it's not too small.
Let's start here at the bottom.
So we define a step,
and all the rest is just configuration,
or definition of things that juped to do,
but the actual magic happens in the step itself.
So we have created a step compile,
and on the right side you will see what jubes do with it.
So the first thing that juped does,
it uses the compile set.
The compile set is defined here,
and we define two parameters.
First is exec, name with string, my exit,
and we have another parameter with the comma separate list,
that we have seen already.
So what juped does it,
recognize that it's a comma separate list.
This comma can be also replaced by any other arbitrary sign.
If you have the comma in your strings or so that you want to use.
So juped creates two directories, two work packages,
one with your one and the other one with your three.
So we have gone through this parameter sets that we have used,
and then the next thing that jubes executing is using the sources.
The source is a file set,
and this can be used to link to copy some data,
and in this case copies all the data that is in the files,
and there is a c directory,
and these are your sources, your cpp files or something,
and the templates make file.in.
And it copies it in both of its work packages,
because it knows it has created these work packages.
So we are done with the file set.
The next thing we use the compile set,
and that's the substitutions that I've mentioned already before.
So we define a file where jubes should look for patterns,
and we can define our output file.
In this case, it's the input file that's make file.in,
which we have already in our directories,
to juped find it,
and it will create a new file where it replaces everything
that is defined under following lines.
And it can be the same, you want to override it,
but here we just want to create a new file,
and here we search for hash,
and replace it with the value that stored in this parameter here.
And we end up with a make file in both directories,
and the last thing that is to be done in the step is,
we don't use anything, but we execute something.
This is marked with the do,
and the do's are executing shell comments,
like if you would do it on the comment line.
And here we run or make comments with an argument
that gets its value from the parameter here.
So in this case, we end up with an executable compile with minus 1,
and in this case with all three.
And this was the last thing that we had to do in the two paths
to do in the compile step.
So we have our first executables to executables compile.
And then you will continue.
It finds another step,
and this step depends on the compile step.
So group knows that for compile step,
we have already two work packages.
So it will also create one directory,
at least one directory for each compile.
And then group also knows that this directory should be linked
to the O1 version, and this directory should be linked to the O3 version.
So that if we execute something here,
group can go via this symbolic link into this directory,
and execute the executable that we have compiled there.
And there's more things,
the script is not complete.
We want to analyze things, we can do much more,
but this is really the very basic structure
with some of the key components that the two paths.
Also, a few common line arguments here.
We can execute the script by jubran on the common line,
either the XML script or the YAML.
The functionalities are like 99% per 9% the same,
just a little bit different than the YAML version on a very specific keyword,
but that's not worth mentioning,
and we can also query previous runs.
So if we want to know which parameter has been set,
whatever kind of information group knows,
we can query it from the common line,
even after a benchmark has been run,
or while the benchmark is running,
we have the common line help to access the glossary
and many more common line options.
So I hope until now,
we have got a bit of know of you,
what you can do, and how it does it.
Now, I would like to go into a few more use case examples
that we have, especially at jac.
I've already mentioned that we are hosting the access cases in Jupiter,
and in the procurement process,
we had to come up with a big list of benchmarks,
like people who are involved in procurement,
they might have done similar things.
So what we did, we integrated several applications
and synthetic benchmarks in jup,
and defined them for different execution targets,
like the CPU, the GPU, or both together,
and yeah, we published this benchmark suite on GitHub,
so if you want to get started with jup,
or if you're using jup,
this might be a nice place to look at,
because there's some recipes that you can use.
So it's nice to have these benchmarks suite,
but we also want to make use of it.
So we use these benchmarks suite
to in a continuous benchmarking environment,
to get insights into system health,
what do I mean with that?
So we have written CI scripts,
components like XACB that we use,
but you can do it also without it.
Then we go via jaccoma on our HPC systems,
and then we execute all the jup scripts that we have.
We extract the information that we want from this particular example,
and we push them back.
In this case, it's a CSV,
and we use another software,
at JAC, which is called LLview,
which we use for,
usually for system monitoring and reporting,
lots of jobs,
but it has also the capability to,
flexibly and generically plot or visualized data.
You can do this also in other ways,
but looking at CSV files is,
it's very hard to get some insights,
so we want to have some more interactive visualization
of the data that we get from,
from running these benchmarks continuously.
So I will give a few examples here,
so we have then all list of benchmarks,
we see some status of it,
how they performed,
and then we can click on the individual benchmarks,
see we look at HPC, HPCG,
on a special system,
HAC,
and everything that you see here,
like the table and the plots is highly configured.
So it's not,
you will see this also in the following apps,
plots,
it's not depending on the metrics or so,
it's a completely configurable.
So what we see here is,
for example, the Gigaflops,
from the HPCG, on the right side,
the runtime, on the X axis,
the time from, like,
mid December beginning of December,
until, like, last week,
and if we hover over the graphs,
we even get more data,
we call them annotations,
and we can, for example,
see which job ID has been used to get this data from,
or any other kind of data that you would have available,
in your benchmarks,
we could visualize here,
same for another kind of benchmark,
this is the stream,
here we don't show geoflops,
but we have the bandwidth on the left and on the right,
and the last one is,
we cannot only show time series on the X axis,
but also more scaling plot,
like also from the stream benchmark,
and here we scale over the threats,
per task, for example.
So this is how we use our benchmarks lead,
to get also some continuous insights into our systems.
But that's about the benchmarking side,
we also use group for scientific workflows,
complex ones,
and here I brought you one from the energy system modeling,
fields,
usually the state of the artist there
to run the dozens of scenarios,
to see how to analyze,
or evaluate scenarios in the future.
So the further you go in the future,
the bigger the uncertainty space gets,
and usually the more scenarios you need to really find
the optimal scenario for a future case.
But the state of the artist that only runs a dozens,
and what we did,
we used HPC and run more than 11,000,
which is quite a lot more than it's state of the art in that fields,
and we did this for the German power system,
and was too coming into the place here,
I'm not going into the details here of this workflow,
but this should just indicate that
we have many different kind of data formats,
and different kind of applications,
and group orchards orchestrate them all off,
and runs them on HPC.
And this study has been reasonably published
right before Christmas in nature,
communications with several product partners.
Okay.
So now I hope you got a first impression
about how you can implement group,
and what you can be used for.
So I'd like to also get with more in the software side of things,
so what's the current status.
The main development is still done in an internal GitLab,
to choose some historical reasons we started long time ago,
in SVN,
and then we moved to GitLab,
and at some point we opened also repository at GitLab,
but the transition is still ongoing.
And we have some upmissures internally,
but they are really almost all future requests,
and these future requests usually do not reach us by GitHub,
or issues usually the interest of people reach out to us
as a developer directly.
That's why you don't see very much activity on GitLab there.
And the last point is more,
maybe someone can help me here with that.
We have tried to publish group on PIPI,
but they claim group as a prohibited name.
So if you know someone from PIPI,
we have opened an issue a few years ago,
and there's not much movement there.
And what are we working on currently?
We are actively working on the version 3,
so it's almost a bit more than 10 years after we have done the change
from version 1 to 2.
And here we would like to change some internal status files
into a database to increase or improve the data,
persistence, the consistency,
and these things,
and also to gain more speed up that we need to
do in the access data,
and also some features,
and features,
for example,
you can then not only push data into a database,
or in plain text,
but it can also use the Matplotlib to plot directly some figures,
and we have some more things to come up there.
Then I guess like all of the other projects,
we also want to extend and improve our token testing,
but we are not doing that bad with it,
but still there's always room for improvements,
and of course keep implementing new features
that are requested from interest to people.
And as I've mentioned,
the transition to GitHub is ongoing,
we still think about how we do this in a best way,
but that's also something we are actively looking into.
Then about availability of Jew,
we have it as a tab or on GitHub,
you can even use it installed via easy build or spark.
We have an online documentation,
advance,
a beginner tutorial,
FAQ,
glossary,
and more on this web page.
You can contact the JC developer's by this email address,
it's all on the web page,
or we also look into GitHub,
if you're something there.
And I think that's it already.
Thank you for your attention.
I hope you got inspired to use Jew.
If you're not yet doing it,
and if you want to get in touch with us,
just reach out to me here,
or in any time in the future.
Thank you.
The question was whether we test all the Python versions,
and we don't do this actively.
This might not be up to date on the Dooku.
Yeah, I would need to look into this word.
What's the latest version?
We need to end what we are testing right now.
I cannot say this for sure, sorry.
The question is,
how do you make your benchmarks portable across systems?
You need to get access to the systems,
and you implement it into the Dooku script.
So it doesn't know anything about the system,
unless you tell it about it.
So we can use something that's called taking for example.
You can define,
if you want to use the inter compiler,
you can create a parameter that runs,
ICC for example,
or you run a parameter with the name,
and then the calls,
GCC,
and you give it a take,
and depending on this text,
you can use them on the command line,
and then you have it flexible to decide,
which kind of compiler you use,
or what kind of architecture you use,
but you need to tell you how it should use the architecture.
So you just generate,
but you need to tell you what it should do.
So we ship something that we call a platform XML.
This is a generic template of parameters,
and settings that we use for example,
first learn,
but we also have it for other,
workload managers,
and with those,
you can prove that you have it for example,
and then you have it for example,
and those,
you can provide generic files that can be used,
you can include,
and you've descriptions into different benchmarks,
and I think where this mechanism,
you could provide different kind of configurations,
for individual systems,
and then tell you which one it should use
for the current execution that you're aiming for.
We are doing it in a way that,
at the question was,
how we run our benchmarks,
but that we create a reservation before,
or we run them in another way.
And what we do,
we create as batch slumpscripts,
and tell you okay,
as batch and then you script,
and we wait until there are free nodes,
so we don't block our general use system,
just for our benchmarking purposes,
unless it's really necessary.
So we use the regular scheduling of slurms,
and this is how we let our benchmark run.
The question was,
how we can use different kind of build system,
so,
does not know anything about easy-easy builds,
but if your application uses Python,
you can just run Python something.
If you have a compiler,
you can just run Python something.
If you have a compiler,
you can just run Python something.
If you have a compiler,
you can just run Python something.
If you have a compiler language,
you need to tell you how to compile it.
Either you configure,
see-make,
or any other kind of
a build system you have.
So you need to tell you what,
how you want to have your application compiled.
And you doesn't know it,
but you can tell it,
and you can do quite a lot,
or almost everything that you can do on the comment line,
so you basically put what you do on the comment line,
and then you have it kind of standardized,
and if you give this script to a friend,
or colleague,
if they know the structure of jub,
they can get very fast started with understanding
what you have been doing there.
The question is whether jub can access executables
that are installed on a different path.
Yes, if you tell jub were to find it,
yes,
you just can,
when you do it on the comment line,
it's the same that if you wanted to do in jub,
you just need to tell it where to find data.
And I think I get a sign that my time is up.
Well, no.
The question is whether jub can access environment variables,
and how it handles the parameters internally?
I'm not sure if we have really changed on this,
but I would say jub can do this,
because jub can read environment variables,
and depending on the value of the environment variable,
it can create its internal structure.
Jub can do it when you tell jub to do it.
So jub,
if you don't give the information to jub,
jub doesn't know it,
or if you tell jub where to find this information,
then jub can also grab it and put it into the output.
Yes.
And the question was whether one can use one compiles step
and reuse it for multiple executions.
There's the full flexibility I would claim there.
The time is no really up here.
Thank you.