Hello everyone, my name is Jade, I work for HPE and I'm going to be talking about productive
parallel programming.
So we're in a room about HPC, there's lots of parallel systems everywhere, but sometimes
they are harder to program than say a shared memory system, and you have HPC experts and
you have users who may not be HPC experts and to actually use these big systems is challenging.
You know, everybody knows OpenMP, for shared memory systems, this is like gold standard,
it's great, but it's all compiler, directive based, and it's just in C or C++.
Or for trend, yes.
If you want distributed memory, you get MPI, Schmamm, pick your favorite technology, you want
both, you use both, there's not one technology for both, and then we get to talk about
everybody's favorite, GPU programming, and pick your favorite solution, there are many, many,
many, many solutions, but the one big thing is all of those solutions are not distributed
memory, so if you want GPUs and distributed memory and shared memory, you kind of
hodgepodge everything together, and that's your solution, and so you have to be an expert
in each of these technologies.
So I'm not trying to say anything bad about these, they're all good, however, right, they're
all based in C++ for trend, so you need to know C++ for trend.
Some of these haven't changed in decades, I'm looking at UMPI, you know, it's been around
for a long time, but we're still doing things the same way, we were 30, 40 years ago,
long before I was born, and, you know, you have to mix everything together, so you have to be
an expert in everything, and if you want higher level abstractions out of C, C++ for trend,
you usually have to give something up, and so I say that HPC has a high barrier to entry
because of all of this, can we fix that, I hope so, so capital is an alternative for
productive parallel programming. It's a modern parallel programming language, portable and scalable,
so I can run it on my laptop, I can run it on a supercomputer, open source, like all good projects,
it's hosted under the HPSF, and the goal is to support general parallel programming and making it
productive at scale. So if I have a GPU or a CPU, I should be able to target it from chapel,
and I should be able to do that at any scale. So you should imagine a language that kind of has
all of your favorite characteristics, I can read it and write it, it's performing, it's fast,
I can scale up on my systems, I can target it, my GPUs, vendor agnosticly, and I can be portable,
and hopefully it's fun, programming should be fun if it's not fun, you're doing something wrong.
And all of this is what chapel is for, so let's look at some lovely code,
you don't really need to read this, all you need to know is this is a benchmark, it's the screen
try-on, it's just trying to drive the memory system, and there's a lot of code on this screen to
basically do one little thing and one little for loop that's doing a multiplication.
And so there's a lot of boilerplate and a lot of fluff just to do that. This is not productive.
First is we pull in the chapel code, this is the entire chapel program.
I can create my distributed array, that is, I created distributed domain, and then I declare my array
over that domain, and then I just write what I want. This is the entire program, you can compile it,
it works. And so this is our productivity, what about the scalability, it does the exact same thing
as MPI. So much less code for the same result. Another example, even more code for a slightly
more complicated benchmark, this is the HPCC remote atomic benchmark. If you're not familiar with
the benchmark, you basically have a bunch of remote variables, and you are updating them all
and committing them all the time. The nice thing about this code is right at the top of the program,
it tells us exactly what this code is going to do, and it has this lovely little for loop,
and then you write a whole bunch of C to do that. I don't know about you, I don't, I like C,
I don't want to write that. I'd rather write chapel. This is not the whole chapel program,
but this is the core piece of the computation right here, is one single loop, this does distributed
computing, this does shared memory, parallelism, all in one little loop. So that's pretty cool,
even better. So up on this graph is better, chapel completely outperforms MPI in this case. And
part of the reason for this is aggregation. So in this MPI, you know, something to get that aggregation,
it takes a little bit more work. In chapel, you sort of get it for free. You do have to add a
little bit extra, but you sort of get it for free, and so you get much better performance with chapel.
Yes, I know that seems unreal, it's real. How does this actually all get to happen?
There's kind of two concerns in chapel. One of them is the parallelism, you know, what you're
actually computing with. So I can compute with a single CPU on a single node. I can compute
with all of my CPUs on a single node, and we all love our GPUs, and I can drive the GPU.
Right, so there's my parallelism, but I'm still only on one node, which brings in the other concept
of locality, is where do I actually put my tasks, and where is that data? So I can have one task on one
node, I can have one task on all of my nodes, I can have all of my tasks on all of my nodes,
and of course I can have my GPUs. And then same thing with the data, I can just have the data
on one node, and every single node calls back to the first node to see the data.
I don't want all that communication, so I can replicate the data across all of my nodes,
or I can partition it, so I'm not wasting space on each node. Each node only has the piece that needs,
if it needs like boundary conditions, it can just talk to the other nodes and get that data back.
And again, getting the theme here, I can do the same thing with GPUs.
So I'm not expecting everyone to be able to be chapel experts at the end of this talk,
but I am going to look at a little bit of the chapel code here and kind of walk through how this
execution is going to work. So all chapel programs are going to start on locality 0. This is
different than MPI where everybody starts up at once. And variables are stored wherever you are.
So in this case, we're on locality 0 on our current task, so that's where array is going to be allocated.
I can then run a serial loop over all the locals, the nodes in my system,
and I'm going to go through those nodes, and I'll move computation to the target locale.
So in this case, we are moving to locality 0, which we're already there, so I think really happens.
And then, because we're now on locality 0, when we allocate a new array, so var b equals a,
it's going to be duplicated on locality 0. And I can access it just like I would write a variable.
I don't have to do an MPI call, I don't have to do anything special, it's just there.
Then I can kind of go through all my GPUs, and I can move execution to my GPU just like I would
move to a compute node. The GPUs are treated as what's called a sublocal. So you have your top level
locale, and that contains however many GPUs you might have. And again, the variables in scope,
I can access the variable and create a GPU array. So now b is my CPU array, c is my GPU array,
and have allocated the data. And I'll keep going through this program, just kind of walk through things.
All right, I go and I move to the second iteration of the GPU loop. I move computation to my second
GPU, and then I get my second GPU array. Notice that the GPU array disappears, the previous one,
right, it's going out of scope, it's no longer used, it goes away. And we continue over to locale one,
we move computation, we get our array, and we get our GPU array, and then our second GPU array.
So this is all of this is serial. This is distributed computation, but it is serial. So that's
this is one part of the feature set. The other part of the feature set is adding in the parallelism.
So this is mostly the same code, and I'll call out the differences here. So instead of using a
for loop, so serial iteration, I have this co-for-all loop, which is a single task, or it's parallel
task per iteration. So every time this loop will go, I'll get a new task. So if I have two
locals, the first co-for-all gets two tasks. The second part of this is I have this co-began statement.
So for each child's statement, I will get a parallel task. And so that's how I can drive both
the CPUs and the GPUs at the same time. All of this results in everything being driven at once.
I have my GPU arrays, my CPU arrays, everything's happening all at once distributed and parallel.
This is what I would call low-level chapel. In chapel, I want to be able to just write this
declarative program. I'm not going to talk about this too much. Just know that we have this for all
loop. It has this really fancy definition. All it means is the for all loop is magical, and it
gives you good performance. It will do the right thing. You can write this simpler. This one's
a little bit more easier to explain. But for all loop, it's magical. It does the right thing.
It will give me single node parallel computation when I give it a single node array.
It will give me distributed parallel computation when I give it a distributed array.
And it will give me a GPU kernel when I give it a GPU array. All from one loop, all from one generic
function, all of this happening at compile time, so without runtime overhead.
And we have this lovely loop, but maybe I don't want to rewrite all of my code into chapel.
I want to use interoperability. So instead of calling chapel code, I can call it an external procedure.
It can be C, Fortran, whatever. But it doesn't even have to be external. If you really,
really want to just write your C code in your chapel program, it's just kind of a nice little
feature. And then again, you know, this is now working with distributed arrays, local arrays.
The for all loop takes care of all of the communication, all of the chunking up of tasks,
and then you can just do your math in whatever language you want. So I've kind of showed some
benchmarks. Anyone can write a good benchmark. Let's talk about real applications.
I'm going to talk about two today. Well, two ones that different ends of this spectrum, and then
we'll get into Arcuda, starting with the bigger one, which is champs. This is a 3D unstructured
CFD framework, or computational fluid dynamics. It's very big. It was written by Professor Eric Lando
at Polytechnic, Montreal. And the reason they want to use chapel is because they can have
students, undergraduate students, master students, instead of spending most of their time learning
how to write good NPI C code, they just write chapel code, and then they get good performance.
They don't have that big learning curve, and so they can get right to doing science from the
get go. The other thing is that because it's performant, they're still competitive with other
centers and other solvers, so they're not giving something up by using this productive code.
On the other end of this spectrum, there's this biodiversity kernel, measuring core
reef diversity. Basically, this is just doing a whole bunch of convolutions, stencil code,
over some images. It's a much smaller chapel application. It was written by Scott Bachman
and Associated Authors. The reason why they were using chapel is they were using MATLAB,
and it was taking them months to do their code. It was very complicated. It was rewritten in chapel,
and they immediately saw performance improvements because they weren't using MATLAB. The other
thing is now that their computation was much more cleanly expressed in chapel, they could actually
do algorithmic improvements that they couldn't even fathom in MATLAB before, just because the
code was easier to read. They got algorithmic improvements on top of using chapel. Then,
because chapel has GPU support adding a little bit extra code, they could do GPU, enabled code,
and they ran this on front here, and going from four weeks of a desktop run to 20 minutes on
a handful of nodes, not only doing science much faster, but it far bigger scales than they ever thought
and this was with a very small amount of chapel. So, kind of two end of this spectrum of
productive applications. Again, why would you want to use this? Get out of the way of the science,
like just express what you want to say and stop worrying about managing the memory, stop worrying
about the communication. There's many languages that are good at this, but to get that performance,
you might have to give up those nice abstractions and Python if you want good performance,
you're ready to see. It's easier to read, easier to maintain. You can get people started on your
code earlier. You don't have to train them in a new technology. You have to train them in a new
technology. You don't have to train them in a difficult technology, I hope. All of this leads me to
Arcuda. Sometimes it's hard to get people to get out of their Python and their old ecosystems.
So, you want to reach people where they already are, and so you want Python at scale.
So, this is the motivation for Arcuda. You have people who already know how to use Python,
but you also have these big problems that you need to solve. So, Arcuda, there's kind of a lot of
definitions for what Arcuda might be. Basically, you have a Python client, a chapel server,
they talk to each other. The chapel server runs on all your nodes. The Python client is just
NumPy or pandas. So, you might say this is scalable NumPy and pandas. You might say it's a framework
for driving super computers. We would like to expand this beyond just NumPy and pandas to insert
your favorite framework here. That's all lovely and good. Performance is more interesting.
This is a comparison against kind of competing technologies for Arcuda, like DASC and Spark.
Doing exploratory data analysis. This particular breakdown is of a workflow for telemetry analysis
from Frontier. So, they have a lot of telemetry data that they want to do science on it.
And this is a breakdown of different steps in, say, a Jupiter notebook. And we've kind of
arbitrarily drawn this line of interactivity of about two minutes. So, if you're waiting
more than two minutes, your thoughts might have disappeared. So, you want interactivity,
you want results back quickly. As you can see, this graph lower is better. For the most part,
chapel does a much better job. Right up front, you can see that chapel is much slower because
Arcuda is going to Chapel Arcuda. It's going to eagerly read in all of that data from
Parquet files. But then, once it's read in all that data files, it's just significantly faster
to do computation. Part of this is because Arcuda joins across nodes are much more efficient
because there's much better computation because this is, Arcuda is running on top of a built
for HPC technology chapel, whereas desk and spark are not. The other thing that I will add to this
is I did not personally run these results. The person who did knows a whole lot more about
data science than I do. But they had a much easier time getting chapel and Arcuda set up,
then they did trying to wrangle a desk server and a spark server. It was much easier to just launch
the Arcuda server and do the job. All of this chapel is built from the ground up for productive
parallel computing. The language features let you express your computation and it meets or beats
the low level approaches while still being easy to read, understandable, maintainable,
and without all of that extra boilerplate. The last thing I'll leave you with is this quote from
one of the original authors of Arcuda. It's on the verge of resigning myself to learning
MPI when I first encountered chapel. After writing my first chapel program, I knew I had found
something much more appealing. Why would you want to write MPI when you could write chapel?
Chapel is open source. We would love to interact with you. If you saw something you liked in this
presentation, please come tell us about it. If you saw something you didn't like, please come tell us
about it. Come code with us, interact with us, help us make chapel better for your particular use
case. Thank you all very much. I'll take any questions now.
So the question was what's the out of core computing capability? Can you define what you mean by out
of core? So chapel does not, out of core by the amount of, can you put arrays that don't fit in
memory to disk? Chapel does not provide a built-in ability to do this, but you can just take your array
dump it to IO and then read it back in. So there's no built-in way in chapel to do this, but it's
trying complete language you can do it you want with a parallel IO to speed up that computation.
So the question was, is there already a implemented? The short answer? Yes. The chapel run time
interacts with the various networks and we'll do RDMA for faster memory accessing. Yes.
So the question is, are we using the topology of a given run to do any kind of optimizations?
Answer is sort of. For the most part, we just kind of assume a heterogeneous allocation of nodes.
So we're not doing any kind of dynamic changing of things. However, what you can do is say you have
multiple nicks, multiple GPUs, and you want the locales to kind of splice them up. You can use
something called colocals and have multiple locales in a given node so that you can kind of divide
up those resources a little bit more, but everything is still kind of assumed to be very homogeneous.
Any other questions? Yes.
Does it run up the cloud? The question was, does it run on the cloud? Yes. I have done testing on AWS
using parallel cluster, P cluster works great. I know of some people who have used it with
believe it was GCP. Worked great. I see no reason it wouldn't.
The question was, can you control where memory is loaded, particularly on the GPU?
I mean, you can control which GPU it is, but exactly where in the GPU.
Totally. The one thing you can do is interrupt with coulda, hip, whatever, and do whatever they can do.
Yes.
So the question was, when you do something in NumPy, it's the NumPy operations
of vectorized, but if you try and not use those vectorized operations, it starts to get ugly.
It's very similar in our CUDA. Don't try and iterate over in our CUDA array and pull all of your data
from your remote node to your client. So, yes.
So, the question is, when you're doing interoperability, does that included code get optimized by
L of the end at the back end? Is that correct?
A little bit, the question was, is it a DLL or is it optimized by a compiler?
So, the example that I showed specifically with that external block, that's going to get optimized
together. So, there's kind of no external calls. If you're calling out to a pre-compiled object
or a shared library, yes, there's like a foreign function interface passing between the two.
I'm not sure I can answer that well. We can talk after if you want.
Any other questions? All right. Thank you all very much.