All right, the second to last talk, and the things HPC, their room at Fatham, is by Edgar
on MPI-Bind.
We saved the best for last.
Edgar Leone, Lauren Thielmore, National Laboratory.
I work for Leone or Computing, which is the institution that or the organization that feels
the supercomputers at the laboratory, and we are currently hosting the Fatham
Supercomputer in the world.
I will talk about MPI-Bind, a library that maps applications onto supercomputers.
To begin, HPC users face complex architectures.
On one hand, you have applications with different requirements.
So they need portability, at least at the laboratory and all the institutions.
We need applications that run not only on one architecture, but on multiple architectures.
And we want our application developers to be productive, particularly when they move across architectures,
across vendors.
And we want them to be able to extract a significant percentage of the peak performance of the architecture.
And so here on the screen, I have several of recent architectures, including the AMD
MI3.8, which I will go into a little bit more detail.
But there is also the MBD architectures and others.
So to give you a closer look at how these architectures look like.
This is the node topology of the AMD MI3.8, which is the heart of the al Capitan Supercomputer
at Lawrence Livermore.
And this is the top super at the moment in the world.
On the right hand side, they have a picture of the MI3.10A chip.
And you can see that is composed of multiple chiplets.
So you have six accelerator chiplets.
Those are the ones that I highlighted.
And then you have the three CPU chiplets, which are not highlighted here.
And then you have HPM memory on the sides.
One of the key characteristics of this chip is that you have a single memory.
So you have this memory that is shared, physically shared between the CPU and the GPU.
So this is different than other architectures where you have CPU memory.
And then you have GPU memory.
And then you have to be moving data back and forth, right?
So a key distinguish characteristic there.
So on a compute node of the Capitan, we have four of these, right?
And so you have the four quadrants here.
And the three CPU chiplets show like this.
So you have eight cores per CPU chiplet.
And you have two hardware threads per core.
And each as I said, each APU has its own memory.
And here is the GPU.
And it also has a network interface controller as well.
And again, just to highlight that this memory right here is shared between the CPU and the GPU.
And so when you think about mapping applications onto this architecture, it's not easy.
I'm going to give you another example.
So here we have the Capitan architecture of the Frontier Supercomputer at Oak Ridge National Lab,
which should be, I guess, number two on the top half corner at the moment.
And so you know, again, it's not easy here.
So here, for example, the GPUs are local to an L3 cache,
as opposed to the, they might 300 a day, which the GPUs are attached to sort of an anomaly.
And what could this type of architecture?
Like what you're trying to port your applications onto these machines?
Well, one of them, if you have multiple hardware threads per core,
you could be mapping your processes onto each hardware thread rather than to a full core.
So you would be getting like half of the performance that you're expecting.
You could be launching multiple GPU kernels,
and they are being mapped to a single GPU, right?
That's supposed to multiple GPUs.
You could have, let's say, that I have tasks on this chiplet right here,
and instead of launching to my local GPU, I could be launching to a remote GPU.
If you notice, GPUs here is here, not here, this is GPU 4, right?
So if you think that you're going to be using GPU 0 here,
well, then you're using this one over here,
and so it will be much more expensive to use that GPU.
So again, it's not easy to map these applications,
but you could have you have MPI processes.
If you have open MP threads, if you have GPU kernels,
then you have to be very careful about how you map things onto the architecture.
So with that in mind, we created MPI bind,
which is mapping algorithm to take your application onto the hardware.
It does the mapping automatically for a user,
so that they don't have to think about all these things that I talked about.
And the three design pins, it principles of MPI bind,
is it focuses on the memory system.
I think that was key, particularly at the beginning,
that most of the mapping and affinity algorithms,
they were looking at the compute elements,
regardless of the memory system.
I think now they're coming up to speed, like slurren and so on,
but particularly at the beginning, MPI bind was the only one
that was really looking at the memory system.
It focuses on locality, right?
So it's going to give you local resources.
It won't give you a remote GPU,
and I'll tell you why that is.
And again, we look for portability across architecture,
across MPI libraries, across resource managers,
and vendors as well.
So for the remainder of the talk,
I will just focus on a few characteristics of MPI bind.
I don't have time to go over all of them,
but you'll have links to further resources if you are interested.
So let's begin with MPI bind and sort of how it works,
as I said, the key characteristic is to follow the memory system.
And so for doing that,
we first create an abstract tree of the memory system of your computer,
right?
So you have no more domains, you know, memory domains,
and then you have different memories,
the L3 cache, the L2 cache, the L1 cache is and so on.
So you first have that heater key.
Once you have the memory tree,
then you start attaching the compute elements to those memory elements,
right? So for example, L1 cache,
we'll have cores associated with them.
No more domains, they may have GPUs associated with them,
and then we're in different controllers and so on.
And so now when we get an application,
then we're going to try to map it to the memory tree first.
And then once we do that map it to the memory tree,
if that task, the device is associated with those memory resources.
So let's say that I have a 32 tasks,
and I want to map that onto this tree.
So first of all, I'm going to distribute my tasks over the nomad domains,
so that I will have 16 tasks on this nomad domain.
And then what I'm going to try to do is,
so I have 16 workers that I have to map.
So I'm going to start traversing them,
the top tree down, right?
And so I'm going to start looking at this level,
which is the L3 level,
but I only have 10 L3s here,
and I have 16 workers.
So I'm going to keep going down the heater key,
until I can match that request,
which happens to be at level B4,
where I have 20 L1 cashes,
so then I can map that request of 16 workers that I need.
And so I will map it at that level,
and of course once I do the assignment of the L1 cashes,
it will have courses associated with it,
it will have GPUs associated with it, and so on.
So everything will be local.
So that's basically how the algorithm work.
Now I'm going to go over certain properties of MPI bind
to tell you a little bit more about how it works.
So in this example,
I have a node with two nomad domains,
40 cores and 4 GPUs.
And let's say that I want to do something simple,
right?
I have an application with 8 MPI tasks.
So what I want to do is to use 2 GPUs,
but I'm sorry, 2 tasks per GPU,
right?
I have 4 GPUs and I have 8 tasks.
So if I want to do a reasonable mapping
on IBM's JSM,
which is like this learn of IBM,
if you will,
I have to do something like this
to be able to do a good mapping,
right?
And so for a user,
just trying to understand where all of this means
is not easy, right?
And so at least for our users,
they would be doing something like this instead.
Right?
So MPI bind,
the only thing that you need to pass
is really the number of tasks that you're going to be running.
Right?
And if MPI bind is on by default,
you don't have to say anything else.
If it's not on by default,
then of course you have to say,
you know, I want to be using MPI bind.
And it has a number of options that you can pass
to MPI bind through this minus minus MPI bind parameter.
The only required parameter for MPI bind
is the number of tasks.
The rest, you know, are options
and I will cover some of these options.
If you're interested in all of them,
of course, you know,
take a look at the links that I will share later.
So let's talk a little bit about portability
and how MPI bind achieves that.
But first, let me give you a counter example.
And I suggest please do not use the MPI constructs
for affinity and binding,
because if you do that,
you're not going to be able to,
you're not going to be portable.
So let's say that you're on a system
that is using Intel MPI,
and you're going to another system that uses open MPI.
You're even within that system,
if you just want to move to a different MPI
that perhaps performs better,
then you have to change your constructs that you're using, right?
So it's much better to do it
at the resource manager level in that case.
So how does MPI bind manages portability across system?
As I said, it only relies on this abstract memory compute tree.
As long as you can build that compute tree,
MPI bind will work the same in every architecture.
And the way that it constructs the memory tree
is using HW lock.
So at the end, if you can run HW lock,
you can run MPI bind.
So that's the only requirement that you have.
Now, it's important to highlight that there is a separation
between the mapping algorithm and actually binding your processes
and your tasks and your threats and so on.
And so MPI bind those in two different approaches.
So the first one, we have a center face to get the mapping, right?
And then for every resource manager that you have,
we created a plugin that actually does the binding, right?
So we have slorn plugins, some flux plugins,
fluxes, a resource manager that we use at Lawrence Leone
is getting more common now.
But so the plugins, the way that they work,
is they get the job information from either slorn or flux.
They will call MPI bind to get the mapping.
And then they will do the binding of the tasks
or setting the environment variables that need to be set to do affinity.
Okay, so MPI bind tries to minimize remote memory access
as much as possible.
And this is important because this is the stream benchmark.
If you look at your local memory bandwidth
versus remote memory bandwidth,
you can get like a 50% penalty, for example.
So what MPI bind those is that for every task,
it will keep it contained within a new model main
to make sure that all of the threats of that process
access just local memory.
But of course, if you need more memory,
then it can spill over two other new model main.
Another characteristic that is important for MPI bind
that I mentioned is that you want to be using local resources.
So if MPI bind gives a GPU,
it will be associated with local CPUs.
You get local CPUs, it will be associated with local GPUs.
And so here's an example or some performance moving data
between the CPU memory and a local GPU versus moving data
from a CPU memory to a remote GPU.
And you can get a significant performance degradation
when you're using a remote GPU.
So it does matter.
It depends on the architecture and the topology
of course, if you're using a MPI link versus PCI and so on,
but it can matter in many cases.
Another component that I want to talk about is MPI bind
has been an important component to address
system noise on high performance computing systems.
System noise refers to any activity
that runs along the application and affects
the performance of the application.
So think of the operating system.
The operating system has to run on the same compute note
and it may affect the performance of applications
if it's running on the same core or on the same hardware thread.
So one common approach to deal with system noise
is to specialize your CPUs.
So you will be using some CPUs for the operating system
and some CPUs for your application to provide
as much isolation from the operating system as possible.
So you have application CPU system CPUs.
And with MPI bind is very easy to control the placement
on your architecture, right?
So for example, you could say,
I want to use the bottom hardware threads of every core
because the top hardware threads of every core are going to be used
for system processing.
So you would just say MPI bind restrict and you pass
the system CPUs and then you're good.
So the system administrators they don't have to
change this long configuration or the flux configuration
they don't have to do any of that.
And just to mention, you can also use
MPI bind restrict not only on CPUs but also on memory domains.
You can say, I want to stay with memory domains
for and 7 and you will stay within that part of the compute note.
We've done multiple studies of system noise.
And I just want to highlight that here,
highlight here that MPI bind allows that type of study very easily
just by changing that environment variable.
Here I have two configurations where we use
the top hardware threads for the application or the bottom
hardware threads for the application.
Plus, some other kernel settings.
And here we use a micro benchmark that is just a compute loop
that runs over and over and over again.
And then you measure how long it takes every iteration
and then you plot those execution times.
And you can say there is a significant performance
improvement on this side.
And not necessarily to compare the two configurations,
but just to tell you that with MPI bind is very easy
to do this type of studies, as long as you have the c-sad
means doing the placement of the operating system process
on the other side, right?
But at least on the user side is very easy to change
if you're using MPI bind.
You can also use this MPI bind restrict variable
to place your jobs in whatever custom
where you have, let's say that you have a traditional
simulation application.
And then you also have a machine learning component.
It could be a surrogate model for example, right?
And you want to place them differently on the note
to say take advantage of the GPUs or the CPUs and so on.
So you can do that pretty easily.
So in this case, I have job one on the memory
domains 4 and 5.
And I have job two on memory domain 6 and 7.
And then the third job is using the bottom cargo
threads of the first half of the CPU.
So you can do that as it is shown on this slide.
So we have been using MPI bind that LNL in production
since 2015, so I guess it's going to be 10 years next year.
It has changed over time to account for the different
architectures that we see.
But the important thing for us is that our application developers
will want them on day one to get to a new machine
and don't worry about the affiliate and binding and give them
a reasonable binding to begin with.
And of course, if that is not enough, they can explore more.
But they don't have to think about it when they
start running on a new system.
You can build it very easily with the new auto tools.
You can use SPAC.
We have, as I said, this LORM and the flux plugins.
In order to enable it on SLORM is very easy.
You basically change one line of this configuration file
in SLORM and then you're good to go.
So MPI bind is an excellent initial policy.
But if you have an application that requires different
mappings over time, then this is not going to do it
because MPI bind is a static.
On the other hand, there is no modifications to your application
in order to use MPI bind.
So that's powerful.
We're developing a new library for applications that are more
complex that require different dynamic mappings throughout the
light of the application.
We have a lot of documentation and articles available.
So you'll feel free to use those ones.
These lights are on the web page so you can get those links.
And to conclude, I just want to say that MPI bind is the
goal of MPI bind is to make application developers more
productive.
And we focus on sort of three areas on performance,
productivity and portability.
And with that, I will take questions.
Thank you.
How are you?
Several questions, okay.
Let's talk about it.
Just a question of the relation between MPI bind and
the job scalar, because you will show you can run
as run and you just don't hate tasks.
And then MPI bind will do the magic.
How can MPI bind is a huge hardware role on the remote
notes.
Do actually figure out the resources there.
When I just told, as well, I want to take that.
Yeah.
So the question and I'm going to try to please correct me if I
didn't get it right.
But so he's talking about the integration with
SLURN and how would I deal with remote notes, for example.
And so MPI bind is running on every, on every note, right?
So the decisions are actually made local.
It doesn't have to worry about making decisions on a global scale
because from the resource manager, I will get say,
how many workers per node are going to be on that note.
And it will take that information and then do the work.
So again, all the decisions are local to a node
because I will get that information from SLURN for
flux.
Yeah.
But if I execute the run and the logging node.
Yeah.
So it kind of know that.
So it might seem that I have different confused notes with
different hardware.
How does MPI bind know to which one is best for my application?
So I guess the decision manually before that.
Yeah.
So yeah.
That's what I'm trying to gauge.
How could MPI bind work on the logging node?
How is that okay to interpret your question?
Or what is the logging node is different from the computer?
That is fine.
So what is the difference?
Yeah.
What if I have two notes that are different architecture?
So MPI bind will call HW lock locally on every node once.
And it will get the topology.
And the mapping will be done based on that local topology.
Okay.
So you can do topologies.
Absolutely.
If you have a cluster with GPUs of AMD and then MBD and
and knows without the GPU, that's fine.
Because there will be one call per node together topology.
And then it will make the decision based on that.
Yes.
Okay.
All right.
Thank you.
Yeah.
So when there's no perfect assignment of the resources,
let's say that there's either you have to choose a bad Nick or a bad GPU
or a situation.
How do you score them and decide which one is better or less better?
Yeah.
So what is?
How do we determine what's the best mapping?
Because in some cases, choosing a GPU over a CPU may give you a different mapping.
And so we certainly run into this.
And so what we have is at the beginning of this slide deck.
I show some of the options to MPI bind.
And you can say my application is really going to be focused.
It's a CPU heavy focus.
So optimize the binding for the CPUs.
You can say optimize the binding for the GPUs.
And it will do that.
Because yeah, there is no single ideal binding for all cases.
Right.
So then we have to pass hints to MPI bind.
So you will do minus minus MPI bind equal GPU optimized.
And it will optimize the binding for the GPUs in that case.
So yeah.
Absolutely.
Are there any plans to support Intel GPUs?
So if I haven't tried the HW lock on Intel GPUs,
if HW lock is working on Intel GPUs,
then you don't have to do anything.
That's the only constraint.
But I don't have any Intel GPUs that I can try things on.
And so I haven't tried.
I'm sorry.
Oh, okay.
So there you go.
I'm sorry.
Please tell me your question again.
If you are thinking to store the bind,
you get off the thread.
The event will be spawned by MPI task.
And if with the environment,
is affected by the environment?
Yes.
So the question is,
do we do any affinity thread affinity?
If if process is are launching,
you know, multiple threads.
So we take care of open MP binding.
So for open MP,
all of the threads will also be bound
to the to the right resources.
If you're launching post-extreds,
they will just be confined to the process.
The process mapping or the process,
the resources that have been assigned to that process.
So it won't be as fine grain as for open MP,
but you will be constrained within that process.
So you wouldn't be able to say,
do remote memory accesses
because the process is confined to a new model main.
For example.
So the question is,
do you support,
like,
see you aspect of device that's important,
where you actually,
like,
partition,
which threads would be also,
which see you something device.
So the question is,
do I support,
see you masking on GPUs that's for that?
And I guess the answer is not,
because I'm not familiar with that.
Yeah, yeah,
I love to talk to about that.
Okay.
Thank you very much.