All right, we're going to listen to tracking energy and emissions of user jails by Mahendra.
Thank you.
Good morning, everyone. As my name is Mahendra Pai Puri. Currently, as my day job, I'm working
like a XC and Spoparis. So, yeah, this focus not everything about a political sciences,
like I used to work like one of the national HPC's interests of France, like before this
my current job, and most of this work I've done, like there. So, just to go into the details,
yeah, I don't have to explain this context anymore. The AI servers are like,
consuming the world. Well, actually, this plot, I have to update this, like, because OECD has
updated the numbers and they look even back now. So, yeah, this is one of the, one of the hot problems
that we have right now, and this is something that we need to solve. Well, I mean, within the HPC
platforms, like, yeah, I mean, there's a lot of, there's a lot of solutions that come into to solve
these problems on the system level, like, like optimizing your energy usage of the anti-HPC
but of course, end users are still running like crappy course on the on the HPC platforms,
like, you know, they're just like burning electricity for no reason, right? I mean, even your
system level optimization software can do a lot of things to avoid that, like, the might take on
this problem is like the change as to always come from the people from the end users and, like,
you know, you got to chip the blame on them, like, you know, they had to make them feel guilty,
you know, for, for burning electricity. And that is the only way to, to the practical solutions
to engage them, to optimize their course and better use the, use the HPC resource and, like,
better use energy at the end of the day. And this can be only possible, but, like, if we
actually provide them with the, with the measures and the tools to, to encourage the optimization,
right? I mean, and, like, of course, like, small, so the HPC platforms, like,
are they, they provide very, very, very, very high level metrics, like, not really useful sometimes,
like, they sometimes give some useful insights, but, like, nothing,
two important to actually make them, like, optimize their applications. So, there's, like, a
sort of a problem we're trying to solve here, like, and then at my previous job, so we start to,
to build, like, a stack or, like, computer energy and emissions monitoring stack,
to sort of address these problems, right? I mean, so just to give you, like, a quick
introduction, what it does is, like, we started this tool to, to estimate the energy consumption
and the equivalent emissions of HPC, HPC, HPC bad jobs, because, like, our HPC cluster
back then was, like, a very AI focused and, like, there's a lot of people, like, burning electricity
by, like, finding, like, with the, with the fine jobs and stuff like this, and, like, people want
to know how much of energy they're using it, because they have to report that, like, in the
research papers, and thus, just one of the, thus, how the whole work started, like, and then we start,
we, we noticed, like, the potential of the approach we took to develop this stack and we,
you know, actually, extended the support to, but open stack and Kubernetes. So, the whole key,
one, one stack of teams can, can monitor a HPC cluster based on SLAM, and open stack and
Kubernetes, like, in a uniform way, so, giving, like, this resource manager, like, Nordstick flavor,
an SSS system level style, it's not something that users have to use it, like, on the basis,
like, something this is, it means an operators will instead, like, all on, on the entire HPC machine,
or cloud or Kubernetes, and you're, like, get, like, the metrics from graph on a dashbox,
right, I mean, and the hard of the teams, basically, the C groups, I'll just give you, like,
quick introduction of what is C groups. Perf sub system, I think most of the HPC folks know
what is Perf sub system, and EBPF is coming, well, I mean, I'll try to give you a very short
direction of what is EBPF, it's, like, pretty hard topic in the cloud network landscape.
And, well, I must have very pragmatic person, like, I don't like to reinvent the wheels,
which happens quite often in the HPC community, it might not be popular opinion here, but that's
how I see it. So, I try to reuse, like, as much as I can from the existing tools, and then
only implement the missing parts to make things usable, right. And so, if for that, like, we,
I use, like, a lot of CNC, open source components, the promises, like, database to store all the data,
and the graph on, for the visualization, and, like, for the people who don't want to use, like,
a graph on, and they can always use, like, a CLI tool, which is also available with this stack.
Just, like, a quick introduction of how games works, like, okay, just to understand,
for, like, everybody's on the same page, like, control groups is the way that Linux actually
managed the resources, like, CPU memory, I will GPUs whatever, right, I mean. So, effectively,
for Linux, it could slow me job, but open stack virtual machine, like, a Kubernetes part is a
C group. So, if you can manage to monitor the C groups on a given Linux server, we can manage,
like, anything, a slow job, and, like, open stack VM or Kubernetes ports. So,
so, just to show you, like, the whole idea of the games is, we have, like, a central unit here,
like, which is games, and we pull the matrix from different sources, like, for, for, for example,
from the kernel C groups, the CPU and data, I'm usage, and also at the node level, the same thing,
and we pull the matrix, when available on the on the bare metal, from the RAPL and KPMC,
as well as IPMR, I'd fish, HWM on, like, I think, we managed to add, like, support,
to most of the energy resources, we have on the, on the bare metal level. And also, we get, like,
the per-favorites from the per-psip system. We use the CPU, we, EPPF, to get, like, I go on network
matrix, I'll explain, I'll be clear, and for the GPUs, like, we take, like, whatever, like,
NVIDIA, DCGM, AMD matrix device export, it gives us, like, GPU matrix, and, we put all this
matrix into the Prometheus database, and we showed them to the, using graph on the dashboards.
And, well, this is how it pretty much looks, like, on the computer server, we have, like, typically,
like, a Teams exporter, a Prometheus exporter this running, and also, if you have GPUs,
yeah, in video, AMD, you get, like, the GP exporters that are running, they push all the matrix
to the Prometheus database. At the same time, like, we take the, the real time in the static
emission factors, like, from, from, you know, like, a pool, the dynamic factors, as well, into
the Prometheus database. But, from the C groups, we get, like, a very basic metadata of the job,
like, a job ID, for example, in, in the case of HPC, we need, like, a bit more metadata of the job,
to be able to actually show this, the matrix on the, on the graph on the dashboards well. So,
we pull, like, that has told the metadata from, like, a resource manager, for example, from
a, from a, from a, from a Kubernetes. And we put them into an API server, which is basically a
simple SQL database. And also, at the same time, we push this aggregated matrix into the API server,
because this way, we can get, like, the aggregated energy usage emissions pretty easily, because,
like, Prometheus doesn't, like, making a queries, like, okay, how much of energy consumption
the last six months. I mean, these are, like, pretty heavy matrix on the Prometheus, and, and, and,
that takes a lot of CPU, by, by, by storing aggregated matrix in the SQL database,
we can get this information much faster. And, uh, and finally, we have, like, a little component
called load balancer, it's more of a proxy than a load balancer, which can actually,
I say, an optional component, I mean, um, um, imposed the access control on, on, on, on your
metrics, because Prometheus doesn't offer any sort of access control. So, that's, what does it mean,
one user can actually, technically, able to look at the metrics of any other user's jobs,
by default, with Prometheus, and this, this load balancer, what it gives is, like, okay, the user
cannot, uh, look at the metrics of other people's job rather than his or her own job sector.
And, that's what, like, the load balancer does here, and you have, like, the user facing
side of the graphana that, that show, like, the metrics to the, to the end users. And, that's,
that's pretty much it, like, the architecture, like, a very simplified version of it.
And so, a lot of the features, like, as I already mentioned, like, yeah, I,
probably, like, do both energy and emissions, performance metrics, I will, it,
file system, agnostic level, so it doesn't matter if you use a list for, or BGFs, or Spectrum scale.
All the metrics are, are, are scrapped, like, a, I will matrix a scrapped in a, in a, in a, in a,
in a, in a uniform manner, and it works out of, out of the box. Uh, uh, supports, like, a different
energy resources, like an audio bill, his W1, Kray, BMC, via IPM and Redfish, supports both NVIDIA and
DGPUs, and real-term access to the, to the metrics, and, or no, access control to the
premises database. So, what does, uh, the end users can get, like, you know, they can get, like,
dashboards, like, this in the real time for, like, a job usage, for example, like CPU stage memory,
and, like, the host power usage here. And, you also get, like, the performance metrics, like,
CPU frequency, instructions per cycle, and software events, and there's more, actually,
like, wouldn't fit all the, all of them in the single-screen short, but, yeah, this sort of
the things that you can get, like, actually, as I said, in the real-time as the job is running.
Uh, the same with the GPU, like, uh, whatever, like, because we were using NVIDIA GPUs,
so, what, what, whatever, like, NVIDIA gives us, it's a black box, we just show, uh, take them and
show it to the end users. Um, uh, for the IO access, I like, you know,
you can get, like, the read-and-write bandwidths and requests and errors, like, from different
file systems, uh, different types of files system, like scratch work, stuff like this, and we get,
like, the networks, uh, metrics as well, um, but only for, like, the, uh, IPv4 and V6, uh, but,
both, like, the, uh, ingrisoned bandwidths and the packets, and this is how, like, uh, CLI
Clang2 looks like, you know, something like, uh, what ASAC does, but, like, some more enhanced
information, including the performance metrics and the energy and emissions. And also, that's, like,
what, what, what, what, I should kill now is, like, for the end users, like, uh, uh, and this is what,
like, a quest for administrators or operators can see, like, you know, uh, the, uh, the energy usage of the
entire, entire cluster, um, divided by the partition, and also, like, the entire, uh,
overall usage of the cluster, like, uh, based on the CPU, how much of the memory, the,
each partition is using and stuff like this, like, well, um, and for example, this is
can be, this can be, like, uh, uh, one of the metrics, one of the dashboards,
can be really useful for, like, the operators, because here, what you see is, like, uh,
I mean, I, I, I, I, I kind of mask them, but it's a project, and they use it in the project,
and, like, how many jobs the projects have, uh, uh, uh, uh, submitted in the last six months,
or something like this, and, like, the average CPU GPU usage, probably usage and stuff like this,
with, with, with a, with a dashboard like this, like, the operators can quickly identify the users,
which are using, like, uh, who are using the cluster in a bad way, like, say, for example,
here, you can see, like, the user has submitted, like, 14 and talk, 50,000 jobs,
with, like, uh, GPU usage only at 50% of the CPU usage at 6% and it's pretty easy for the,
for the operators to identify these users and contact them and try to help the users to optimize
the workforce to better use the cluster in the end of the day.
Uh, well, some of the metrics, I think, we've already talked about them, like, uh, well,
the only thing has been to talk is, like, uh, there's some RDM metrics available as well, but they're,
like, very, very low level metrics and, uh, if you don't know how RDM works, like, it's really
possible to make, uh, sense of them. We're talking about, like, a few pairs of memory regions,
protected domains, and stuff like this. Oh, just to, tell you, like, the last point, like,
all the metrics are part C groups, which means this part slum job. So, on your note, if you have,
like, one job running on your note, 100 jobs running on them on the, on the note, doesn't matter.
You always get, like, your metrics part, part job, like, even if the jobs are sharing the same note.
So, yeah, it's all nice, like, you know, you get, like, this metrics in the real time,
it's fair enough. Yeah. Okay. I can see, okay, my code is not doing well. What is supposed to do?
Well, then what is the, what is the solution for that? Right. I mean,
typically this is done by, like, uh, by the, uh, profiling. So, well, when the code is not running,
well, it's not doing the things that it's supposed to do well. Well, we have to profile the code to understand,
okay, what's the problem, right? The bottlenecks and hotspots. And typically, this is then with the profiling.
The get-emnistic profiling, what you do is, like, on your laptop or on a login note, uh, to record all the
call stacks, and it's very, very, very, uh, compute intensive, because, like, the, your process has to do a lot of,
which is between the user, user, Canon and the user space in the kernel space, and that's, that's quite,
the, this transition is quite, quite costly. And of course, you cannot always recreate the
problematic scenarios in the, in the, in the, in the production, because, like, a profiling, you
go down your laptop on a single process. It's not, there's a profiling, you go down, like, a MPI,
when you're running it, like, more than one note, like, same. And what is the solution for that?
Countless profiling, this is some, this is the topic that's, uh, cloud native folks,
being, I mean, this is something that Google has, has proposed, like, a more than 10, 10, 15 years ago,
I guess, almost, no. And, but this is something that cloud native folks use quite a lot, and it's
completely EBPF-based. No instrumentation needed at all, and, like, it's very, very, very low,
lower hold, and it's always on in the production, right? And it works out of the box for the
compile language, just for, like, C, C plus, first, fourth, and then go. Uh, well, some of the, uh,
tools that we have, like, do companies profile, like, graphana splung data, like, there's a lot of open
stuff. And, like, Kim's exporters supports graphana pyroscop, graphana pyroscop is, like, equivalent of
prometheus to store the profiles, like, basically, the, the, the, it stores, like, a times
it's database, and the graphana pyroscop, which stores the profiles. So, what we can get if the
continuous profiling. So, that, okay, this is the architecture now, it's very much, like, the same,
like, the one difference is, like, the exporter sends the profiles of your, of your user applications.
And, like, you, from the graphana, you can see these profiles, and what, what we're talking about
the profiles when, like, is the same as, like, the typical flame graphs that you see, when you
when you profile your jobs, and what you see is, like, a real slum job that is on your HPC cluster.
It's a C plus plus code, I guess. And, what you can get is this sort of flame graph, like, in the real
time, and what you see is, like, the names of the functions and the times spent inside each function,
right, I mean. And the only thing that you need to do with your code to get, this is actually,
include the debug symbols, without the debug symbols, I won't know what is the name of the function,
and the, including the debug symbols, it doesn't, doesn't hit any, give any performance drop,
generally. And this is for the C plus plus code, I mean, it's for the, for the compiler languages,
work pretty well. And it works for the Python as well, even though it has, like, two levels of
stack and winding process. But, yeah, for example, this is the example from the Python, and,
and it works pretty well, what you see is, like, the name of the file, and, like, the name of the function,
inside the file, and as I said, like, this is on the production machines, like, you know,
I mean, our HPC center now is using for more, like, more than one and a half years,
and this is a pretty happy, like, have to have this graphs in the real time, and things are working pretty well.
Just to show you the export over here, like, average of 16 nodes, like, you know,
this, like, almost the CPU usage, like, less than 5.5% and, like, the memory is, like, around 100 megabytes.
And we did some benchmarks, like, with the, with some computer intensity task,
like, without export, with export, with export and continuous profiling. But, from what we see,
it's like, there's no huge performance sheet.
Some of the technical details is 100% on the go, except the BPF programs, which is coming from
EVPS, which I've written in C. I'm not going to talk about the capability awareness,
EVPF and I want to mix. Okay, just to give what is EVPF, EVPF is, like, a way to
to customize the kernel at the runtime without having to use the kernel module. So, you can inject
any piece of code inside any point in the kernel at the runtime. I mean, it's not going to crash
your kernel, because EVPF programs have to pass a very fair stage. And that's how we, we can get,
like, a lot of information about I1 network. Just some more details about, like, testing and
see, like, the battery test, it's, like, a pretty good, you need to discover it, and, like, there's
a lot of interintests as well running in the C. I. And also, the packaging, like, all sorts of
options at a while, will both, like, binaries, pre-compiled binaries, like O.C. image, and hand chart
is also available to be able to run the thing. The final remort, so it's like, yeah, you know,
the teams provide, like, a very complex solution, like, very provides both the metrics,
and also, like, tools to profile your course in the runtime. In the production cases, at the real time,
like, that helps you to optimize your workforce, and, like, better use the HPC classes and the
energy associated with them. Well, it's running on over HPC classes of the joins, as I said,
like, for more than, like, 100, 100, 100, 100, 100 years, like, with the scrap frequency of 10 seconds,
so the users get, like, the metrics for each 10 seconds, for the CPUs, such and stuff like this.
Currently, I'm working to, to support the cloud VMs using spec power database,
because, like, on the cloud missions, we don't have any energy resources. The sources that gives
the energy usage, so, like, we need to do some sort of modeling, and that's the modeling,
we're trying to do the spec power database. The demo instances is there, like, to play around,
like, if you want to see what how it feels like. And finally, like, I use, like, a Greek 5,000,
like, one of the French testbed to, to actually develop most of this, talking with our data, like,
this wouldn't be possible, like, in big thanks to the team that's maintaining that.
And that's it, like, you have all the resources here, if you have any questions, I'm happy with it. Thank you.
Thank you.
Yes.
The question is about, like, there are any questions, I'm not going to be able to,
I'm not going to be able to, but I'm going to be able to, but I'm going to be able to,
these about, like, there are any solutions for the ARM architecture. Well, I mean, I'm
relying on whatever the vendors provide, like, if the vendors provide, for example, like,
doesn't matter, even if it's ARM, the BMC is always there. Every vendor is supposed to put, like,
a BMC chip on the BMC with always measure the total power intake. So, we can get, like, the same
information from the BMC because of both IPM and right-fisher supported, even for the ARM.
Yes, Samora, who else, if you will try to sample, and you say that that was a great,
you know, the sampling about the sample. So, for everybody, there's an absolute value.
But, for example, I mean, more interesting, I mean, of providing it. So, how much can I increase
the sampling that, or how much it's flexible to say, okay, for these part of the,
let's say, like, hundreds of milliseconds, I want to have an idea about the flexibility in that.
Okay, the question is about, like, if you can change, like, the sampling frequency for, like, a given job,
or a given note. No, with the primitius, well, I mean, it's possible if you contact your
cluster admin to change, like, the, yeah, I mean, if you can change for a given note and then
launch your job on this note, yes. But, like, if you really want to profile as, as I show,
like, the continuous profiling, it does with the hundreds of frequency, which gives you a pretty,
pretty, pretty good, like, ideas, already, but like, it gives you a flame graph, not, like, time series.
Yeah.
I mean, like, as I said, like, we're using primitius, oh,
I mean, the question is, like, if I understood correctly, if I, if we can integrate, like, the
matrix coming from the external power meter into the stack, I mean, like, you can push the
data to the primitius, right? I mean, if your external power meters have some sort of exposure,
you can position the primitius and then you can integrate that into the matrix, yeah.
The question is, like, if I have, like, any, I mean, I guess to, to, to integrate graphana
tempo to get the traces, that would be nice. I actually tried with Jagger to, to, to, to the
EPPF as well to give, like, an MPI, I would traces, it is possible, but it's just, like, the
question of, some time and effort, yes.
Yes, the question is, like, the power matrix work for the multiple jobs on the same note, yes,
as I said, like, all the matrix are coming from, um, C groups. So, if I have to split the
power between multiple jobs, I have to do some sort of modeling. Currently, the modeling is
based on CPU time and the memory usage of the job, compared to the, the node, and that's how I split the
power.
The question is, like, if it can, I saw the number of jobs by the total number of time,
totally yes. I mean, it's something that graphana does, I don't do anything, and I, graphana,
had this option of being that yes.
So, the question is, like, is there any support for the hardware based power
meters, and can you give the example of what is a hardware based power meter?
Mm-hmm.
Fair enough, I mean, as I said, just before, like, I mean, if you can, if you can measure this power,
and then push it to the, to the, to the, to the promutius, of course, you can use it inside,
and you have, like, as I said, like, the BMC is all, all, all already supported.
Yes, please.