So, as my day job, I work at San Pupari, which is like a political science school.
Nobody is being EBBFS political sciences.
I can tell you that.
Most of this work, I've done like in my previous job where I was working as a national
HPC center of France.
So, just like a first question, how many of you know what is HPC?
High performance computing?
Okay, that's not too bad.
Thanks to you, I know everybody is getting to know what is HPC.
So, just like to give you for other people.
Because my presentation is going to be a bit like focused on HPC.
I'll do like a short introduction of what is HPC is like high performance computing.
This is out of data center.
The one thing that is different from regular data center to the HPC clusters,
is like the interconnect network.
HPC clusters is like a niche network.
We just call it RDME, a remote direct memory access.
That gives you like a micro-second latency, which is not possible with Ethernet.
That's what makes HPC clusters different from regular data centers.
And thanks to the HPC, like RDME, all this crazy huge AI models can be trained on like thousands
of nodes passing the messages between like thousands of nodes.
I talk to each other with small messages.
And that's actually being done using the MPI, which is another very niche HPC thing.
It's a software that can be able to launch your application on thousands of nodes
and able to talk to each other in very, very small messages.
So yeah, just a little bit of the context as I was telling before,
like AI is transforming HPC.
Yeah, thanks to AI.
Like people are getting more and more about HPC because like five years back,
HPC such a niche subject, not a lot of people who knew it.
That's done.
And yeah, most of the HPC-traditional tools are like very focused on MPI.
This tool that I'm putting about like, you know, which is used to spawn like application.
On thousands of nodes at the same time.
Or very focused on HPC work course and they will not work for the AI work course out of the box
because most of the AI work course are like a package with Python work.
Of course, the GPUs are like the beast.
They can do complete very, very, very fast.
Most of the time they're limited by the IGO is, you had to feed the beast.
The feeding of the beast is by the storage.
And like, if you don't have the fast enough storage, it doesn't matter if you have H100's B100's here.
It's not, you're never going to use like the potential.
And again, one more thing about like a HPC system is like a parallel file system,
a parallel distributed file system, which I really really call them.
So for example, if you're aware of like a AWS FSX, it's called, it's based on LUS,
which is a parallel file system.
And it can offer unprecedented level of performance, again,
like thanks to RDMA, that the niche network I'm talking about, like everyone direct memory access.
And we have like a different types of parallel files system.
As a moment, like for example, LUS is one of the biggest ones, like very well known one.
Most of the HPC clusters use them.
And also we have like spectrum scale from IBM, like BGFS.
We have like CFFS as well.
I mean, which is no very high performance, but this is pretty much used.
So every files system, they bring their own telemetry.
Like it's almost impossible to find a common ground to get like a common matrix between all these different file systems.
And this is a huge problem.
One of the biggest problems on the HPC community is always the fragmentation.
Everybody does their own thing, there's no standardization.
And yeah, most of the HPC tools, like standard tools, like Dush and Dush and Dush,
and it's like I will monitor and build again, made for the HPC MPI.
It's strongly coupled only for the MPI works, and it doesn't work for all sorts of applications that use HPC, HPC platforms.
So yeah, there are these few of the problems that we need to deal with, like when we thought about like traffic profiling IW.
So ideally what we would be like to have, right?
I mean, we want to monitor the applications,
and the file system that we're using.
The portability is very important, and the standard standardization is quite important in the HPC as well.
Which doesn't exist at the moment.
And like let's be honest, like nobody wants to change that course, people are lazy.
That's the reality I can have except it, like asking people to recompile their course,
that instrument that course is never going to be worked.
This is an option, it's never going to work in the real world.
And of course, it has to be minimally increasing and negligible overhead.
I mean, because like at the end of the day, we want to debug like a performance issues in the production,
not on our laptop.
It doesn't make any sense like, well, it doesn't give you a lot of interesting information.
Being just out of, I will profiling on your laptop.
So we have to do it on the production, and so it has to be minimally increasing.
It should have a lot of work ahead.
And like, yeah, if you can capture that, you will even set any level of the stack.
That's even better.
And that's what like, EBPF can't offer.
I think I don't have to really do this like, because like,
you say EBPF development, I think everybody's ever wants EBPF use.
So yeah, I mean, so how we use like the EBPF to do this, I will monitoring,
just to show you like the tracing of the VFS file system.
So just to give you an idea how the, how the IO happens in the Linux is like,
whenever the process stops to the, once you do like IO,
either read or write CISCOS.
They're actually passed through VFS layer, which is virtual file system layer,
which is like the interface that like every file system has to, has to implement,
to be able to actually talk to the hardware.
So for example, like beta local file system, beta remote file system,
like last VGF spectrum scale, BTFS,
they all have to implement like the interfaces defined by the VFS.
And like, it's the VFS responsibility looking at the file,
the script of the file that wants to do the IO is going to delegate the call
to the correct driver that actually does the real IO.
So if you can actually monitor the VFS layer of the, of the, of the kernel
by a by injecting like a, a, a, bf programs,
we can actually monitor all the IO that's happening through your kernel, right?
So for example, here is a bunch of the functions that actually,
we can, we can use to monitor the IO for a VFS read,
it's like to read the file, write the file, open create,
make it, and we can remove it.
And based on the version of the kernel,
there are some variants of the read and write that we need to,
to monitor instrument to be, but like,
globally these are like the most important functions that by by by by by
instrumenting these functions, we can, we can capture all the IO activity
on the on the on the given one from the kernel from the kernel point of view.
And this is pretty much the basic idea of what we, what we're doing.
So just to show you like a sample code, like,
the headers and stuff like it's, you know, it's pretty standard.
So one thing I would like to talk here is like,
you know, the resource managers,
doesn't matter which are resource managers that you're using,
for example, be like a HPC basket, you just like slum,
or cloud VMMs, like open stack, or even Kubernetes.
To be able to manage the resources, they use kernel C groups.
So like each C group means a part,
the Kubernetes context or virtual machine in the open stack context,
or like a job, a batch of in the in the slum HPC batch system context.
So by by monitoring like the C groups and the mount point,
because like most of the time, like the the file systems are mounted
on the given given paths on the file and the links server.
So by by by by by keeping track of the combination of the C group
and on the mount point, we should be able to track,
like the iO activity of a given application of a given workload,
for example, for a given point in the Kubernetes context,
let's say, onto a given file system.
So that's what we're going to track as a VMFs event key,
what we've seen here.
And then like the, the even itself is like the number of bytes that,
like the, the, the, the, the part is actually writing to the given file system,
a number of calls and number of various engineering errors.
These are the structs we define and we put them into a,
a bit of map right on in with with this event key as a map key,
and then the, the even as like a value of the value of the, of the map.
And here is like the most of the heavy heavy work has been done basically.
Like we take the, we take the VFS right, for example,
it has like a file pointer, we take that as like input inside of function.
We get the number of bytes that they actually file is attempting to write to the file system.
This is pretty much easy, it's very easy to get the thing.
The most of the heavy work is done in this, in this part actually.
So EVP of helper functions gives like a C group ID for the V2 out of the box,
which is pretty much easy, but for the V1, there are no helper functions.
So I've written like another, another function that actually can work for,
but for the V2 and V1 to get like a C group ID.
And this, this line here, what we're doing is like get mount path.
Looking at the file struct, we have to get the mount point,
and this is actually most tricky part, let it look wild for me to get it right.
Because within the file struct, we have to recursively go back in the, in the, in the, in the paths,
to actually able to mount this mount the, get the mount path of the, of the file.
And because there's like very little bit of support for the loops.
In the EPPF and it's recursively loop, I have to do this.
It took a while for me to get everything right, but like, it's been working well.
Since since I got it right.
And then the last part is pretty much simple, like it's very standard, like you, you make an event,
you like, there is no event, you create a new event, and like this already event,
you just like increase the increase the calls and the bytes and errors as well.
I'm missing that here.
It's pretty much simple, but as I said, like this, this middle part is like most of the heavy lifting,
and this is where most of the, most of the computation time is happening.
So, well, to able to do the test, the approach, like, I did, like, a test using an NFS server,
and then like, I created, like, I took, like, an NFT server to do the test,
so that nobody is using it as only for me.
I created NFS and, like, mounted onto to the same server to remove any sort of network noise.
And I used Iowa benchmark again, Iowa benchmark is the one that's heavily used in the HPC HPC environment,
and also, like, any sort of platform, I guess, to be able to test the, Iowa performance of the underlying file system.
So, yeah, onto the, on this NFS server, like, so we, I tested, like, with the different transfer sizes of 1, 2, 4, and 16 megabytes.
I think, like, well, I don't remember how much, like, the entire data is, like, 10 gigabits, 20 gigabits of data in this sort of transfer sizes.
For both POSIX and the MPI, okay, I'm not going to go into the MPI,
Iowa MPI always, like, is a convenience wrapper for, like, MPI applications that enable,
to, for the multiple processes from different nodes to be able to write to the same file,
which is a given offset, it's like, the people from the MPI, they, they love to use MPI,
because it's a convenience wrapper to be able, for the different processes to write to the same file.
And then, like, most of the, this eBPS programs are implemented in,
in, in a primitive, export, or, like, King's, export, or, actually, this is the work we've been doing,
like, you know, in the HPC center, back then. And this export, actually, uses, like, the eBPS 2 to monitor the Iowa.
And we, I, for the test, we come, I configured the Prometheus 2 to scrap, I will,
2 seconds, intentionally, to create, like, heavy enough load, like, so that, like, every time the map has to be read and, like,
the, the Canon has to take a spin log that, which is going to create an over, overhead.
So, I've created, like, such a small scrap interval, just to create the overload artificially,
to see if it's going to have any, any sort of any sort of influence on the, on the, on the performance.
So, I'm looking at, like, two, two matrix, like, a relative overhead, which is basically,
basically, the overhead reported by Iowa, without exported to the wheat exporter.
So, higher, higher, higher, higher, it means, like, higher, higher,
higher overhead. And then, relative, like, higher, higher overhead.
And then, the relative error, like, basically close to one, is, like, the error is,
basically, the bandwidth reported by the Iowa benchmark to the bandwidth reported by the,
by the exporter from the Prometheus. And here are the results, like, the, the first column,
which is the positive side one, the second column, is, like, the MPIow.
And the top ones are, like, the relative overhead and the bottom ones are, like, relative
overhead. So, for the, both and read and write, our operations.
For the smallest transfers of one kilobite, I guess.
Yeah. You can see, it's, like, actually, the mean value is, like, less than one,
which doesn't make any sense, because, like, that means, like, with the exporter,
the performance is actually much higher than without exporter.
So, that tells me that, we are, like, in the floating point errors range,
so, so, the, the, the overhead is not very measurable, let's say.
But, like, if you see, like, all, all, all the values are hovering around one,
which shows, like, the, the overhead is pretty much negligible or almost non-existent.
And the, the same with the, relative error, all the values are pretty much hovering around one
as well, like, from most of the, most of the things, which, which tells me, like, okay.
Well, there's, there might be some more overhead, but it's not really measurable or not, really,
not feasible.
And yeah, folks from the HPC, like, to say, ah, everything works in the single node,
because, like, it's such an easy configuration, like, we want to see, like, if the things
work on the multi-node context, right? I mean, I don't really see why the,
why it should not work on the multi-node context, because he, because he, because he,
maybe, if it doesn't have to do with the single node or multi-node, but, like,
anyway, I, I get, like, the test, like, all the multi-nodes is in, like,
IOR, and the Jones-A, our, our HPC platform, which has around 2500 nodes.
So, I did, like, a test, I don't, from the four nodes, with 16 MPI processes, like four nodes,
four processes, four processes, on each node.
With the transfer, says, of one megabyte, this is something, you know,
to say, the default, like, the cluster has been using on the last Rafael system,
with the Pozics in MPI-I-O.
And Dersheng, which I've been talking about, like, Dersheng is, like, another standard,
I will monitor and include that, like, the MPI folks use, like,
so I use Dersheng to be able to validate my results from the exporter.
And Dersheng shows the matrix for the, we've been into, well, for three,
two seconds or something. This is something I couldn't change, like,
is, is, is something of the hard-coated, like, within the Dersheng.
And again, we, we use, like, a exporter, the kimchi's exporter that's been,
running on the, on the, on the, on the, on the, on the, on the,
HPC platforms, with this crap into, well, of 10 seconds here,
I could not change, I could not, I could not reduce this crap into, well,
because it's a production system, which has been running, like,
so changing that to the lower, lower can, can, is, is complicated,
because, like, as I said, we have, like, 2,500 nodes,
and it's not that easy to, to, take the, take the, take the,
take the, take the nodes out of the production to, to, to the test.
So, what we did is, like, to validate the exporter, we just,
compared the instantaneous bandwidth, reported by the promitius,
with the ones that are reported by Dersheng, which is, like,
more standard, too.
So, what we see here, again, for the projects in MPI,
will, the solid lines are from the EBPF approach and the,
markers are from the Dersheng, like, the other software, like, from what we can see,
is, like, the pretty close to each other for both,
read and write operations.
For me, you can, you can see, it's, like, one of the nodes,
actually, I have degraded, I will perform,
it's a correction, mission, things can happen wrong,
but, like, both, like, the Dersheng and both,
the, the exporter, based on the EBPF, the, the both, like,
followed the trend pretty, pretty well.
And, for both, projects, and MPI, go.
So, that, that actually shows, like, the,
and the exporter is working well, and the approach that we're using,
the EBPF is working, as it should.
Just to show you, for example, yeah, as it said,
like, the exporter is running on, on, on the HPC platform,
like, since, like, more than two years now,
like, as it said, like, it has 2,500 nodes,
and what we see here is, like, the, I will bang it from all,
all the, all the, all the cluster, onto the different file system,
we have, like, three different, two different file systems,
like, we actually monitoring, like, one is a scratch,
which is, like, hyper-form and file system,
one is a work, which is, like, a, low-performing file system,
from different partations, like, it has,
the HPC cluster has different groups of nodes.
So, from each group of nodes, what we see is, like,
the enter, I will bang it.
This is, like, the cluster side of sign, like,
the whole cluster view, and, like, for example,
like, we, we provide these matrix for the individual users,
as well, for the jobs.
So, for example, this part users can get it, like,
for the, I will, I will operations for, like,
for both read and write bandwidth and request,
for, like, given job, for example,
from the graph on a dashboards in real time.
Well, we have, back then, we had users,
which are, who are doing, like, a pretty high,
intensive, I will operation, like, running course,
that does, like, a lot of, I will operations.
So, after we deployed the exporter,
nobody actually came back to us saying,
okay, I will performance got degraded,
because of the, of the, of the,
exporters who I think, like,
we never had any, any, any, any, any complaints.
So, the exporter is working very well,
without having any noticeable performance difference.
So, what else, what, what more can we do with the EPPF, right?
So, yeah, again, like, I'm still talking about the MPI,
because, like, this is the one that's used,
heavily in the HPC, and now, EI, as well, like,
and it's very important to be able to profile the MPI functions.
We can actually trace the MPI libraries using, like,
the EPPF as well, like, you know, by sending the,
the events to the ring buffer.
And, like, the, the, the importance of the ring buffer
is, like, you know, it keeps track of,
I mean, it keeps the order, even the,
even the events coming from the multiple CPUs,
and, like, in the HPC, of course, like,
your job is using multiple CPUs as a given time,
and, like, you know, events can be generated from any of the CPUs
that your job is running at a given time,
and it's very important to keep the order of the events.
And, you can, if you can get these things sent to the OpenTea,
OpenTea limit restake is, like, if you generate profile traces,
like, I think, I did, like, I used the Jaguar, I guess,
but this, for this, profile concept, yes.
So, I really want to do, like, a very, very, very,
a quick profile concept to see if I can actually,
profile the MPI functions, and send them to a Jaguar
to see the profile traces, the spans, like, for the, for the visualisation,
for example, but, here's what we can get, like,
you see, again, like, the same IWAT test, like,
the IWAT test, like, how it works is, basically.
The, the series of events is, like,
IWAT test opens a file, write the data,
the given amount of data, closes the files,
and opens the same file, reads the data that he has written,
and then closes the files.
So, from what we're seeing here is,
basically, the events, the MPI file open,
and file set menu, like, and then MPI file write,
what we're seeing, like, the profile the spans,
and then, once the files has been written,
like, MPI close, I think this is where, like,
the sync is happening, that's where it's taken a long time.
And then, like, MPI file open for the read,
and then, at the end of the day, MPI file read,
and then, MPI, this, like, it closes as we're probably,
I couldn't get inside the screenshot, but, like,
yeah, with the EPPF and, like, if you,
I mean, if you can get the EPPF trace, like,
the EPPF functions, and, to this traces,
to something, like, Jagger,
we can get this out of spans and profiles,
like, in the real time, for the users,
and it's, it's really important for the HPC people,
because, like, HPC clusters consume a lot of energy,
and, like, you know, if you can get this sort of information
in the real time, if things are going bad,
you can stop your job instead of, like,
running the job till the end,
and only after that release, and working things are bad.
And, yeah, this, this, this, this, this,
something, what, what more we can do,
like, you know, it's only,
for any of the HPC community comes in, like,
that's, like, standardization of the monitoring,
like, but still, like, just to fragment it,
everybody does their own thing.
Just to closing remarks, yeah,
what we did is, like, zero instrumentation,
I'm monitoring framework.
It's true that, like, a spot only for the HPC here,
but, like, there is no reason that it wouldn't work for, like,
for the cloud, or, or, or, even,
Kubernetes has a set, like, because I've tested it on the board,
or open stack and Kubernetes, and it, it works,
already box.
Yeah.
Well, the, the approach to, like, a leverage,
as much as possible, the cloud needed to,
because, like, I'm a very pragmatic person myself,
like, I don't want to recreate the things just for,
because of, like, creating, you know,
so, most of the things we, we, we,
we're using for the graphon and parameters,
it's pretty good, like, as I said, like,
we, we, we, we're using this export on,
2500 nodes without any problems of, of the,
of the scalability since two, two and a half years,
and it, thanks to the permités community,
because they, they did create a job, like,
with the permités and, and it works very well.
And, yeah, the CVPF also gives us a ability to trace
any sort of function, any, any, any levels,
so, that, that can, that can be, very, very useful,
like, you know, in something in, in, in, in,
in, in, in, in this is, in, in, in, in, in,
in a, in a, in a domain like HPC,
and yeah, we can combine with,
we've, this continuous profiling,
to give, like, a complex monitoring and profiling solution
for the HPC platforms, not only, but also cloud and also
Kubernetes platforms.
And if you want to get the HPC, the EBPF course,
and stuff like this, everything is on the GitHub,
like you know, you can go get the have a look if you're in
faster.
That's it, we thank you.
So thank you for your presentation.
I have one question because you are profiling at the VFS
level.
The right or the read, but did you make some tests regarding the
metadata, like start, scrolls, and just the open, and did you
did you try to figure out what you could did?
So I see, you don't have the stats here.
So the stack, no, no.
Yeah, high code, not the stack, but yeah, I think that can be done
as well.
Because on some workload, you can have this typical stuff running
on with codes that can be really bad for the phase system,
as we're parallel for some sort.
Yes, yes, yes, yes, yes, yes, yes, yes.
It can be done, thank you.
Hello, thank you.
You just monitored the throughput, but could you also monitor the
time a single I-O operation was taking just to see if I have
some blocking I-O somewhere.
You mean like time stamps?
Yeah, just with time stamps, I want to dump my buffer onto the file
system and it takes normally 100 milliseconds, but once in a while
it takes a full second, and I want to know why.
Well, I mean as I said, for this sort of profile,
you should profile the Cisco, the Cisco.
Yeah, okay.
You can get by by profiling the function.
Yes, that's true.
It should be able to get that information.
Okay, thanks.
We're getting the expense.
That finished yet.
More questions.
Thank you for your talk.
Can you track the I-O on a by file level?
Yes, technically, okay.
It's just like your map's going to be super huge.
I mean, that's where like I hear like I used like the the mount point,
like to aggregate all the, all the higher having on the mount point,
but you can use like a file path, for example.
If you do it on the trace, I guess there you could include the exact fine.
Yeah, exactly.
It's possible.
You still have a couple of moments if anyone wants to ask some more questions.
Looks like we could.
All right, thank you.
Thank you.
Do you both want some coffee or something?
Okay.
Thank you.
So we're just waiting for a few minutes more.
As you remember, we still have that of stickers.
PPS stickers at the front if you want to.
To refill your stock of stickers.
Not that stickers are lacking it for some usually, but never.
That's why we have to clean it up again.
That's right.
So help me get some stickers.
Yeah.
So who has experience with manipulating strings in BPS programs?
Yeah, a couple of people.
The good way will be after the presentation.
All right.
So Victor's going to talk about simplifying the string and link in BPS programs.
So we start.
Take it away.
Thanks.
So can you carry this?
Cool.
So hi.
I'm Victor.
I work for the run hat.
And yeah, as kind of said, I'd like to talk about handling strings.
And you'll be able to program something that has been kind of clumsy over the years.
So I'll talk about these trink hay fangs we've introduced in the past year.