So, hi everybody, welcome to this last session of this conference basically for us here
in this room. I'm Cassini Vinesky. I'm a research software specialist or expert at
Tudam Stutt and a researcher at HPC Specialist and today I'm presenting work that is a
collaborated effort of many. In particular, this talk was submitted by people from the Fritric
Alexander University and Tudam Stutt. So, that's me. And what I'm going to talk about today?
Well, first I'm going to introduce basically the basic question of why is
job specific monitoring something interesting? And I'm going to expand on the tools that
were developed by basically the project partners, cluster cockpit and puzzle jobs. And I'm going
to talk a little bit about the use experience of these systems. So, basically who am I? I'm from Tudam Stutt
Tudam Stutt. Tudam Stutt is one of the nine NHR centers in Germany, which are like not the top tier,
but also not the medium and lower tiers. It's like big HPC systems. In the order of magnitude,
each system has a thousand nodes. Each of the nodes has a hundred cores. So, we're talking about
hundred thousand cores in total. Each on average has four gigabytes of memory. And we have about
500 to a thousand active users give a take. And the question is, classical monitoring by
admins is on the system level. We see what each node is doing, whether or not the node is healthy.
We see whether or not the hardware's operational. We see whether we're getting the performance
that we paid for, but we also would like to see what are the users doing. Are the users actually
making any meaningful use of the resources they occupy? Are they occupying all of the resources
or are they just allocating the resources and leaving them empty? Would like to know,
what is the limiting factor of those jobs that are running in our system? Is the memory
bandwidth the limiting factor? Is the network hard the limiting factor? So, we can do better
for humans in the future. And then, in the end, we still also would like to know, how is the
performance of those particular jobs doing? Are they running at peak efficiency? Are they doing
the best they are doing? What is the software actually doing with the resources on the system?
Like, hard to perform scandalous and so forth. And why is the software behaving as it is as we can
observe it? And, classically, what we have is we have stuff like system monitoring here where the
admins have graphana, they have big visualization boards where they can see all nodes are green,
some nodes are yellow, some nodes are blue, what does that tell them? It tells them stuff about nodes.
Well, what we could do, we could go in and say, okay, who was on that note? Hopefully, it was an
exclusive job and nothing else was running. And then we can somewhat infer what that job was doing.
That's somewhat in the direction of application monitoring. And then there's something in the
bottom, what we HPC specialist do, we do performance analysis. We instrument, we profile, we modify
the codes of the software to get as much information out of them so we can be enlightened of what
is actually happening. But this middle region here is somewhat unsupported. Well, we can kind of
infer stuff, we can mock up certain tools to bring us that information, but there wasn't anything
on that particular level at the beginning when this, basically, when my colleagues and I started
working on this problem. So what we came up is something, mostly the people in Alan and I
have to admit came up is something called cluster cockpit. And cluster cockpit itself is the ideas
we are not tracking the whole cluster. We are tracking on the granularity of jobs. What does this
then look like? Okay, cluster cockpit provides everyone who wants it, users, happens stuff,
supporters with a graphical user interface where they can see what is running on the cluster.
So you can scroll down and for each job that is running on the cluster, we get one of those lines,
each of those lines then consists of configurable different metrics. Like for example, the first one here
is the UCPU utilization, then we have memory here, then we have any, it's a flop's metrics,
so how many flops are the nodes in total doing and how much memory bandwidth is consumed on those
particular systems. So when looking at that overview, we can get a brief glimpse of what is the
job doing, is it using the resources, is there anything a miss? We get additional information here
on in front, we get the username and not shown here, the username. So we can actually contact them
in case we want to support them in some way or another. We can get some survey information,
how many cores do they have, how many nodes do they have, when did it start, how long is the
job running, is it still running, or is it a job from the past that I can potentially
want to investigate, we can filter with different metrics like tags, how long did it run,
so very comfortable using the face. You can also click on any of those jobs and get even more
detailed information like, for example, again, all of those metrics, you can have those metrics,
you can scroll in on the metrics, you can select whether or not you want to have the metric shown
at the node level, at the core level, at the socket level, so very is going to learn it is,
you can interactively browse your CPU, your cluster workload, but not on a node level, but on a job
level. And in particular, something like these graphs here, maybe not as good visible here on the
projector, where you have this kind of spider diagram where you can see, okay, how much of the
memory is used on the system, how many flops are consumed from the computing, and how much
memory by this consumed. And ideally, we would like to have a full triangle here because the job
is burning, turning out the maximum floating point operations, using all of the bandwidth and using
all of the memory, and if you want it probably also all of the cores, a full system, great,
not that job isn't doing that. So we probably wouldn't like to investigate, and there's other
metrics that you can support like, for example, over here, this classical graph, the roof line graph
that helps you assess whether or not your job is within boundaries. There are many other views,
and I briefly want to just go with it because I'm not going to train you how to use faster
cockpit here, it's just gave you a teaser of what the whole system that we're capable of doing.
So you can also kind of open up for your user and then say, okay, how many jobs does that
particular user have? What is the total of all time? And then you can configure also how much of
these resources you want to see, do you want to see how many flops you used, whatever makes you happy.
We go back briefly to the overview, we can also investigate what is the total use of a
sub cluster. Okay, this is something a little bit more in the direction of administrative stuff.
How many nodes are actually allocated? What is the current flops rate in that cluster?
Oh, what is the current memory bandwidth rate in the cluster? And this again,
can give us a system maintainers and maybe even a procurement people, an idea of where we want to
invest our money in the future because if we have a system and all our users are hardly making
use of the floating-point numbers and hardly using the memory bandwidth that the system is capable
of delivering, why are we buying that expensive hardware? But again, this is entirely different
kind of worms and this is an example, this is not actual real data.
Might be, might not be, I don't want to tell. Again, you can also have different views like, for example,
who are the top users on certain clusters? What is the ratio? What is the average duration of the
jobs or how many nodes does that particular job have? So it gives you different views of what
the cluster is doing. Okay, how does it do that? Well, it all starts with an entity that's called
a metric store that is the shorter memory of the whole system. Usually you install that on a one
particular node, it needs a lot of memory because it keeps all of the data you obtain on the cluster
for the system to display you on a short term notice. Then, on each node, you install and start
a metric collector. That's a little, I would call it service demon, that starts on a configurable
time level, like for example, every minute, every five minute, every ten minutes and it collects
whatever metrics you want it to use. In particular, what we are using is liquid to gather
half of the performance counters for the system. We use CPUs that to get, okay, how much is
what is the load on the system? We use Mems that to get information from memory. We use Ibse that
to get information from the infinite bank card, but there's like, I think last time I counted
it was about 40 different interfaces that you could use to get information, even for GPUs,
or how full is the GPU, what kind of GPU, tons of data. All of that is sent to the metric store.
So you have a one, a menu to one, or a menu to two communication, and the metric store keeps
up that. That is a JSON communication. Then, you have the backend that is responsible for showing
you the nice user interface, this web-based user interface, and that queries this metric store
for whatever it needs for to display the view. But since you're intactively working with the
system that needs to be quick, so it's not on storage. What happens then, once the job has
passed in a certain point in time, usually at the end when the job terminates, the backend
system takes all of the data that pertains to a particular job and dumps all of that into a
job archive. The job archive is then the permanent record of what the job was like at this one
time. The metrics are resolved in time, so we have time series of all of the data and you can
get that. How do you get that information? Well, that is the one thing that isn't by default enabled
by the distribution. You have to write a little script that whatever batch system you're using
at the start and at the end of the batch job, you need to communicate, hey, my job ran on node 1,
2, and 3, and it had socket 1, 2, and 3, so all of the data needs to be communicated to the
backend system, so the backend system knows what kind of data to pull and put in the job archive.
And in this case, there is a slurm plugin, or not plugin, but slurm can drop, that checks
every five minutes, okay, which jobs are there, which jobs aren't there anymore, and communicates
all of that to the backend system. And then in the end, the user can log onto the backend system
and whatever it wants to see, it can see. You're optionally also can couple all of that with
data bases, time series data bases, SQL data bases, for whatever else you want to use it for.
And this brings me not to the second point. Since this is now a very complicated system with
lots of displays and lots of data, nobody actually wants to go through each job and assess whether
or not that particular job was a good job. And for that reason, the colleagues and I, we haven't
said, okay, can we automate this kind of checking by having a rule system that notifies us when
certain boundaries or certain thresholds are violated? Basically, some form of automated alert,
okay, I use a here or there is doing a very bad job of using the resources that you should be
using efficiently. So the idea there was, okay, how can we fit that in? Well, we use this cluster
copy technology that we have. We've write some form of rule mechanism that operates on each
job archive and assesses whether or not that particular job is where the or is or not well,
where there is maybe a wrong turn is good about. And what do these rules look like? Well,
it is a practical system. It's not very sophisticated. It's basically it's Python evaluated
in JSON files. So do what do we have here? Well, we have a tag or a description of what this is.
In this case, this is a low CPU load. So when the CPU is not fully used, then that should
alert us. The second part of the rule is, okay, we need some form of configurable threshold,
because we don't want to adjust all of these rules for each cluster to it that it might have.
So this is from a configuration file. Here we say, okay, we want to have an alert. If this
goes below 90%, for example, and then this one specifies, okay, which metrics from the
job I've job archive needs the rule engine to pull the rule engine actually to evaluate. And then we
have variables that are evaluated at top of the bottom. Basically defining certain thresholds,
being able to use all of the Python back math, whatever you want to use. And then in the end,
we have some form of output tricker in this case. Once the low load is true or above a certain
threshold, then we consider that rule to have failed and then we want to notify the user.
And how do we do that? Well, there is what we call it at M template that is filled with the data
that this rule kind of generated. And then what we do we do? Initially we thought about sending
emails to the user. And I'm going to talk about that in a minute. But what kind of rules do we have
at the moment? Well, we have low CPU load, we have low load imbalance, we have excessive CPU load,
we have memory use problems. So a lot of different rules that help us kind of get a gist of what
the users are doing wrong. A brief overview. So how did we fit that into the cluster cockpit
set up? Well, we simply wrote a little script that whenever there's a job archive on the disk,
a new job archive, we analyzed that one and we put that information about that to the output.
We deployed that in Darmstadt, in Allang, in Padaborn, and in Berlin. So there are four sides using
this kind of mechanism at some moment and we are still filling adjusting it so it suits us best.
But the key inside that we got is once we saw how many jobs actually had problematic behavior.
It's like 50%, 20%, 30%, we kind of thought, no, we are not going to send emails because
if you have a job load like what we have is like we have 250,000 jobs per month,
sending 125,000 emails per month would make us email spammer and we didn't want to do that.
So we really never really implemented sending the emails. What we do at some moment,
we have text files where this is dumped to that we then analyze a specialist and we tag the jobs in
the job archives so that the GUI once it gets to the point can actually display for each job
also those kind of failed metrics or failed rules that we have specified so that when
a user looks at his own jobs, he can get an assessment. Okay, I'm doing a great job because
there's nothing failed or I'm doing that a batch of using the resources when there are
lots of red flags at the top of his screen. Briefly commenting on the experience of how
the wasn't so perceived since I wasn't initially part of the cluster cockpit development team.
I had a great on us of deploying the system at our university and in principle it is very easy
to deploy they have on they have a good web page and there are binaries on the web page
that you can basically drop in if you trust the binaries, you can start the binaries and
everything is really easy. However, it gets a little bit tricky once you get to the point of that
the system needs to know what kind of cluster you're running on. It doesn't magically infer what is
the peak floating point capability of a particular CPU. It doesn't infer what kind of memory is there.
You need to, there are tools in there that benchmark all of this stuff and put that into configuration
files but you still need to bundle it together in terms of okay I have a cluster and in that
cluster consists of two or three phases each phase has a different configuration so you need to create
a hierarchical structure and for each of those you need to specify what is the peak performance
that this has in all of the metrics you want to have evaluated and then it gets a little bit more
complicated because all of the free instances need to have this matching configuration so for
example if the metric collector collects stuff that the metric store does know about the metric
store just rejects the information and dumps it so it doesn't pertain it so the metric store needs
to know which metrics to keep ideally the one it gets from the nodes and then the cluster
cockpit back and also needs to know which metrics it can pull from the metric store so at the moment
there is no automatically to couple all of these things together it's multiple configuration files
and you need to have a little bit of discipline not to mess up the names and whatnot but
once it gets started it is fairly well to use so in summary a little bit of the challenge we are currently
aiming to address hopefully is that all of these entities shown here have their own individual
configuration files and those configuration files need to match up it's not comfortable at some
moment but it's doable and as a last take away and we have a little bit time to spend the end
I would consider cluster cockpit and patho jobs as very powerful tools to have on your cluster
because on the one end it enables you as a user like it enables users to inspect and reflect on their
jobs on a per job basis without having to do anything they have the information they can log onto
the system and they see their jobs they can click on the jobs they can see what the consumption
of certain resources where they can see whether or not they're using the memory they can check
on which cores did they run did they leave some cores out was the core mapping according to plan
so this kind of information is available for the users for me as an HBC support person aiming to
improve the capabilities of our users I can have a look at every user's job once configured with
the right privileges and then for example SS okay this user needs some let's say counseling in
terms of how do you allocate your resources how can you make sure your resources are actually
used and for example if I get tickets like hey my job brand very slowly can you tell me what's going
on I can open cluster cockpit go to the past look at this job and see for example oh that one
particular note it had a cooling problem and the CPU clocked down because again I'm also tracking
the clock rate that the CPU do over time so I can match that with buck reports or tickets so
that helps me as a support users and for the admin staff they get a lot more information of what
actually is used on the system what the system is required and what the users actually make use of
and that's the end of this presentation I'm happy to answer any questions you may have
and give my own user experience of the system thank you
question for questions yes right how does it scale up what how much data are you taking for
so the question and each section would get its individual metrics store and again you could put
some form of a hierarchy in there but that is a manual process at some moment and in particular you
need to be able to tell at the at the back end level okay if I'm on that note I need to contact that
metric store and at the moment this is done by ranges so ideally no zero to 99 is metric store one
100 to 190 metric store number two you need to have something like that but since this is open source
you could go in and implement your own mapping strategy at your own risk
so that was also one of the questions I asked those guys and there's two answers and so
a number and so the question is why do you implement why didn't we implement that using
existing technology from ethios and graphana and so number one at least that's what I'm told
graphana has the problem how do you access user privileges how do you infer okay this user is allowed to see
only that fraction of jobs that's what I'm being told again that was mostly it's so the counter
answer is like you can put in infrastructure in profit and profit is in graphana to mitigate that
and since young icing at the initial design of cluster cockpit is not here I cannot answer that
however from his so let me rephrase the question hope I get it right did we validate our results
with information from profiling on tracing runs that's it's a moment we have a bachelor thesis running
which is precisely doing that particular part of the of the validation part so that the idea is that
he goes in submit jobs using classic performance analysis techniques like instrumentation
or even benchmarks that we know from the validation phase of the cluster what the benchmarks
presuits produce in terms of data and then we go back and see whether or not that information is
actually depicted in the cluster cockpit itself however since my intuition is since both of them are
using the same gathering techniques like hardware performance data it should be the same unless
there is somewhere a software buck in the system where we just do gross mistakes and just use
wrong data fields or whatever but again there's a thesis going on to validate that and to be frank
since that system is in production at some moment at 4 HPC centers and we haven't seen anything
overly strange I would say it's we can say with some sense of confidence that it is working
in a reasonable range I can say to what degree is this as precise because again we are not doing
anything particular to make it very precise but it should since we're sampling on a minute level
you don't see that much anyways you can see a lot but the fine detail that you see in let's say
a good perform an instrumentation for trade with let's say MPI calls in there what not that's not the
level of granularity that the system is aiming for
right thank you Christian there was so much so the question was is the system more popular
with the consultants over the users I have a problem of answering that because on the one
the consultants love it because they can see what the users are doing the users are very appreciating
when you get in contact with them and are actually able to show them what their jobs we're doing it's like
I don't it's not anymore let just believe me I've seen this stuff in I have a screenshot here it's
like here log in use your credentials this is your job you can browse it you can analyze it you can
submit another job and compare that it's that is a very powerful aspect you can actually
decide by side comparisons of your jobs as a user without having to know how do we instrument
messed up where do we place the data how do they use a data exploration system like vampia
it makes things easy and for that for that reason I would say it's a grivalent good for both sides
yet
you