the
Okay, hello everyone. Thanks for being here. It's my first positive, so I'm very happy to be here as well.
So my name is Antoni Hazapis. I'm from Greece, Greek. The institution is a research institute called fourth.
And today I'm going to talk about the work we are doing on running Kubernetes workloads on HPC.
Hopefully this could be on times we can have a discussion at the end.
So this is a picture of hardware. The question is what do you run on this hardware? Do you run your cloud workloads or your HPC workloads?
And what is happening now most of the time is that you have partitions and you can run some stuff on one part, some stuff on the other.
Okay, I won't go into the details most of you know that the workloads are pretty different, right? So the guys on the left run box jobs, they run tightly parallelized code, they run binaries, the guys on the right, the packets are coding containers, they use frameworks and they mostly run webs of microservices and that kind of stuff.
But what we want, okay, we don't want to make them all run the same stuff, we want kind of embrace the diversity and the run hybrid workloads.
So there it is. Here is that you have a big pipeline and some parts are like legacy code, important, some parts and they run via slum and some parts.
You may have found some, you know, Kubernetes tool but that's all the computation or you can have spark or, I don't know, pytor, whatever, and you want to run it on the same machine.
So the common approach is just to bridge the two environments, you run one code on one environment, the other, there are many tools that do this.
I have an extensive related work in the paper, I'll show you at the end, but the idea, there is the problems with this bridging are clear, you have to move your data around and you can have, there are many the restrictions in both environments.
So, for example, the HPC environments are mostly, you know, maybe they are aeridop, they have very restrictions for access, moving data and things like that.
In another project, we tried to run everything on a Kubernetes, so this is kind of interesting.
We run containers in Kubernetes that simulated an HPC environment, so you had slum inside the containers, but that required the slum scheduler to coordinate with a Kubernetes scheduler.
We tried that, it's pretty awful, but it kind of worked.
So then we tried what I'm going to show you today, which is a way to run cloud software in HPC, and actually we run many, many clouds as a slum job.
So it may be that some of you are not familiar with Cloud Kubernetes is, I won't go into much technical details, but I have to talk a bit about it, so we can go through the rest of the slides.
So in the cloud ecosystem, everyone has their code packets in containers, so we've solved the deployment problem, and now we have to solve the runtime problem when you have hundreds or thousands of machines, someone has to run your containers.
So you need kind of an operating system, because I like to call it an operating system, because it does what operating systems do.
It takes care of your resources, it runs your jobs, it handles networking failures, so you just describe your containers, and then you hand them over to some API, and it kind of runs it on whatever resources are available, even if those resources are distributed.
So Kubernetes provides an abstraction, it has a lot of primitives and conventions of how to write these descriptions, and it's compliant with a DevOps, let's say culture, because DevOps is not a job description, it's a culture.
So you have to, you can use the same environment for development, then you write these nice descriptions, and then you just fire it up to the cloud.
And there are many, many, many different packaging of Kubernetes, you can run it on the Raspberry, you can run it on Amazon, and many, many extensions and third-party tools, and this is very interesting, because with this work we can leverage those tools in the HPC context.
Now, in Kubernetes, you declare what you want to do, so you give like the recipe, and it is run at some point, so you don't type in commands like run this or run that, you say, I want these three things run.
I talked that there is an API, you have abstractions, so you run pods, which are collections of containers, these are organized in deployments, and these are organized, there are also services, jobs, etc., and the structure is a typical distributed system, so you have one head node or one let's say master that takes the commands, and then you have agents or workers that execute the containers.
So the image on the right, I will go into a, not a deep dive, but a shallow dive on what these components do, so basically the API server is the point where all the commands enter, and it's the point that all the different components coordinate, and that's very interesting, because in Kubernetes you can replace any component with another component, as long as it speaks the same language to the API server.
The API server needs some persistent storage, but it gets from the key value storage to CB, and then you have various, we call it controllers, because they just pick to the API, but they do specific jobs, so you have this guy that implements all the different abstractions, you have the scheduler, which selects the nodes, you have a DNS for discovery of services, and many different storage controllers.
It's the things that run on the action machines, you have the Q-blet, which is the most important in today's discussion, it's the agent that runs on the node that implements the lifecycle of pods, and then the other stuff, let's not talk much about it, it's mostly for networking, the components you have to different networking, have the actual pods, the networks, and then you have the service networks, which are like pointers to the service.
So to run a job, the user gives the description to the API server, this propagates to the different controllers, and at some point the pod is sent to a Q-blet to be run in the runtime.
Now in HPK, we want to change this flow, but we want to run this as a user, and this is kind of one-to-run, this is a user job, without root permissions, which is kind of tricky.
We would like all the abstractions to be available, as many as possible, at least those that don't depend on hardware features.
But what's most important, we want to delegate all the resource management to SLAR, so we want this to run as a job in SLAR, and this respects the organization policies, and complies with existing resource accounting.
So you know what your users run, you know that SLAR is most organizations that have SLAR, have heavily invested in all the utilities around SLAR, and scale across all the nodes.
And we want to use singularity as a container runtime, because it's kind of an established practice to have it installed in HPC, and of course make it easy for everyone, both administrators and users, meaning we want it to be as less intrusive as possible for the admins.
So what can we do? Actually, we can take all the things that are in green, the boxes, and just run them as a process.
They don't depend on anything, they're just services, user-level services, we can just run them as a process.
But these are the things we have to change, so the scheduler cannot make decisions anymore, it has to pass through the decisions to SLAR.
And most of the things that have to do with net working have to be changed, but the most important thing is a cublet, because that's the actual node in the worker, in the Kubernetes software.
So that's the worker has to be some custom worker that does another type of job.
We can discuss more technical, if you want, at some other point.
So what we do, what we have done is this is your cluster on the right hand, this is your cluster manager, and these are your cluster nodes, and this is the software that we package.
So we package everything that's all the standard, let's say, Kubernetes components inside the container, and this is run as a SLAR job.
Actually, everything is run as a SLAR job, but this is, let's say, one binary that runs, and this includes some changes, but we don't change anything in the Kubernetes code, so we don't patch anything or something, we just run the standard binaries.
And we have a very custom cublet, so that's like the secret source.
So this guy here, it's actually, it's a run, it doesn't differ from a normal, let's say, Kubernetes, but it has a very peculiar worker node, this guy, that kind of represents a whole cluster.
So when a command comes in to run a workload, it goes to the APA server, it propagates all the different components as it would, but then our scheduler just selects this node, and this node represents a whole cluster and creates a SLAR job to run it on the cluster.
So it has to convert the containers that should run into a SLAR job and give it to the cluster to run.
So practically, you can think of HPK as a filter, as a translator, between Kubernetes jobs and SLAR scripts.
So somebody that runs on a cluster, ideally can give either SLAR scripts or jams through HPK, and they would both be converted at the end to the scheduler, to the job scheduler.
So pods and the other jams enter and exit that scripts from the other side, and there is a small technical detail here that the pods, which are collections of containers, and there are some intrigues there that the pods has to have a unique IP address, et cetera.
So actually, these SLAR scripts do not just run containers, they run a hierarchy of containers, they run a master container that represents a pod, and this runs inside the other containers that represent the containers.
And we try also to keep the resource requirements, so if you say this container of this pod needs, let's say two CPUs and four gigabytes of memory, this resource allocation will go all the way down to SLAR, as your job description.
HPK runs as a user process, our users can run the workloads using either.
HPK itself is a job, but it doesn't need the special allocation, I mean it's very lightweight, and a little support is needed by the environment.
Basically, it's a sum-sigularity configuration, because we need singularity to give unique IPs to these pods.
So this is something we try to do it ourselves with user-eleven networking, but it was too difficult and they're too complicated.
So we delicate the job to singularity, and there are established projects that do that, and it's easier to convince an admin to install someone else's code that it has a big community.
And then something that's a resource project.
So an example of what you can run is, this is a very popular tool in a cloud-native world.
It's called Argo. It's a language in general where you can run workflows has a nice UI, that you can monitor your workflows.
This is actually an image from a workflow that doesn't be an analysis, it breaks down the different strands and it runs a branch for its different strands.
It has all the features you would expect from such a language.
So assume you have this kind of workload, right?
Somebody has written it in this format, you can now run it on your slum cluster.
And we also, this is kind of a hack.
We added some some extensions, some path, you can give your job in a yummy, you can pass through flags down to slum.
So this was kind of a neat extension so you could say to Argo, I want you to run this job, which is an MPI job,
but pass these flags to slum directly when you get to run.
So you could use Argo to run your MPI stuff integrated with your other stuff as well.
Now in the paper and we have reached at the point, this is describing the paper, we have reached at the point we can run more complicated stuff.
So we can, we fire up HPK, then we install several things using help.
If you know the, so we install Jupiter, we install different controllers, Minio, these are all standard and cloud native software.
And then we fire up the Jupiter notebook and we run some kind of AI tuning workflow, let's say script.
And this runs inside slum.
Also, there was an interesting collaboration with Yuli, has part of the DIPC project because ended.
These are, these are huge PCJU projects as described by the previous speaker, where we run some, some spark, jobs and some, this is, this was just as a proof of concept,
that we can run it somewhere else. The most difficult part was to prepare the test pad coordinate with administration team.
Disable, exclusive node policy from slum, because this means that even if you ask from slum one CPU, we give you a whole node with 256 CPUs.
So there are some things, but it, it has been, it has been running an environment outside for.
So the vision is that the cloud users can exploit the HPC hardware.
And as, again, the previous speaker showed you, there's a big movement in Europe to, to enlarge this HPC ecosystem, we're buying more machines.
There's a big machine in Greece coming as well. And so we expect that there will be users that have never touched HPC, and I've never seen it, and we will now have all this hardware and how can they use it, right?
So from an HPC user point of view, you have all this cloud software that does things very, very easily, right?
You can run this operator and do AI training or whatever. It seems easy, or you can use a lot of software, a lot of frameworks like Arbor or other frameworks already available.
And of course Jupiter, everybody loves Jupiter. So it's one way to run it.
And from the HPC center point of view, it's a music way to run, run your somehow, I don't know if this will avoid the second partition, but at some point maybe, maybe things get more standardized, at least from the administration point of view.
Okay, so that was my talk. I was five minutes short. The code is available. This is an actively being worked on. You can find it on GitHub. This is also a good opportunity to explore the other projects we have in our lab.
And we have also a paper online, which you can download, free, and read. And I want to thank our sponsors, which are here at PCJU and SipsJU project.
Thank you.
Thank you. Great job. It's super interesting. I was curious. And then you mentioned you installed other controls as to if you're good at install, like the GitHub, like CI controller on the Kubernetes and then run the actual pods through my slurm on the HPC system.
Whatever. What should that work?
Yeah. So the question was, if I have much other software than install, so actually we don't care what software you installed. So it should work. There will be bugs.
It's not a mature, let's say, software, but we have installed operators from the Q-flow project. We have installed, I don't know.
We have installed other things. And at the end, all of these stuff that get installed is translated to containers that need to run.
And the Q-blet just does that, right? It gets a description internal description of what container should be run and run. So why not? Probably it will run.
Thank you for that. That's the HPC allows for Kubernetes cluster to spend over a several physical mods.
That's the power. Once the speed of the game can control the speed, the speed will pause on different vehicles.
Okay. So the question was, does HPC take advantage of multiple nodes? What's the speed of communication between the nodes?
Right. So HPC does not, so the Kubernetes point of view, you have one node that spans all the cluster.
So your jobs in Kubernetes think that they run on one node that has like 14,000 GPUs and 50,000 CPUs and I don't know what.
So it spans, it just tells large jobs to run. So it takes advantage of all the nodes and the communication at the end uses what the nodes themselves use, right?
So it doesn't interfere with the runtime of the jobs, of the runtime environment. I mean, where they run and how they communicate, it does nothing interfere with that.
So ideally, it takes advantage of your whole cluster.
The speed of the network, the speed of the network, the speed of the network, we don't change that.
It's probably that when I said we need to have this plugin in singularity to take advantage of hand-out IPs, if these are based around the fast network, it will use the fast network.
Yes?
One of that, I don't even know how to do it. It's a right position. It's a step away, no, I'd rather marry them.
The more method of action, I can think of for one to pick on HPC, would they find that half?
Has that been thought of in trying to get into the container registry? Could it be done?
So the question was on how could we run binder hub? I'm not aware of binder hub, I would check it.
But we have run, I think we have run a lot different complicated things.
I'm not, I don't see any reason. If it runs in Kubernetes, it should run on HPC.
So that's possible.
Some issues at this point, in practical now, when you run something in a SLARM, you have normally time limits or things like that.
So you can't run a container registry forever. This usually has to, you have to package everything and run it as part of one job, one thing.
So it's not for running Apache. The problem won't allow you from the HPC center, right?
Or a container registry forever.
I didn't get the question.
Oh, okay. Well, practically, yeah. So the question, if I heard it correctly, was can I run multiple such HPC,
multiple Kubernetes clusters, right?
I don't know.
Which is the instance of the Comparapas.
What are the like?
You don't know what the Comparapas is on your pace.
Multiple instances of HPC.
Okay, control it.
Well, I, you could. I don't know. I don't see why you would need to do that.
This whole thing is run for a, for one user, right?
And one user can run it multiple times as well.
That's fine.
Thank you.
Yes.
Thank you.