All right, thank you for the second presentation of God, something about, again, some
written of experience, but working with BPA to detect all the MA device driver bugs in real time
from Procure and Maxine.
So just from Maxine.
Thank you, you go ahead.
And if everyone can kind of just scoop to the middle, we are not allowed to have people standing
up because of fire safety, so you have to find a seat.
So please kind of scoop to the middle, so everyone can find something.
And now, okay, so this works.
So hello, everyone.
My name is Max.
I'm a production engineering TLM at Meta.
And today I would like to talk a little bit about how we use BPS to accelerate our troubleshooting
capabilities and basically how do we run GPU workloads at scale and how BPA helps us to make
it reliably and sustainably, I should say.
Also, it's the first time I present such a huge VNUS positive, so I would apologize if
they will be any hiccups or any issues.
So please take that with the grain of salt, but anyway, thank you so much.
And just before the begin, some small, a couple of words about the micro-authors.
So it's Procure and TL.
Procure is a software engineer at Meta.
He works in host networking space.
So for example, he actually worked on the implementation for this specific
program and there was a professor at CMU who actually inspired us to come up with this presentation.
And basically a couple of words about myself.
So I'm a production engineer and I work in GPU, comes distributed, comes in collective
comes space.
And basically, I actually came up with the initial problem, which tried to solve using BPS here.
So, and basically this talk is about how we use BPS for RJMA.
And it's necessary to talk a little bit about their domain subsystem, instead of an external.
Because it's a little bit different from any other system we actually do when we try to do BPS tracing, for example.
So let me start quickly with the quick overview of what are the GPU workloads at Meta and
basically everywhere else actually.
So we have a lot of GPU machines.
They have some number of GPUs typically eight.
And they are actually connected with the three types of networks.
So called scale up, scale out, and the so called front and network.
So scale up network is the sum of this proprietary GPU to GPU implementation.
For example, envelope or some alternatives from Google, from Indy.
So like there are millions of them actually.
There is also front and network, which is a regular access links.
We use to control the servers and load the data from the internet and so on and support.
And basically we are coming to the scale out network, which allows us to really run GPU workloads at scale,
and connect GPUs like thousands of GPUs inside the same collective operations at the same time.
And here is the place we actually use our DNA.
And obviously, like probably it's just a luxury for Meta that we have two separate front and
the back and network, so we have quite a lot of network interfaces and strategies in the server.
But anyway, as you can imagine, these back and interfaces, they are actually paired with GPUs.
And they have really, really high bandwidth, really high connection speed.
And in this situation, the regular network in stack can just keep up with the capabilities of such device.
So in this sense, this is why we need to use our DNA here.
The last, another important point is that RDMA unlocks the ability for so-called system parallelism.
When you can actually run data flow between GPUs and GPUs and nix completely independently from CPU.
So in this sense, it actually creates an unique opportunity to achieve lower latency and higher bandwidth.
So basically this is an explanation why we need to use our DNA.
And just let me quickly mention two main features of RDMA is that it obviously avoids network in stack.
And it also allows direct connections between GPUs and network interface.
Now, okay, we have an RDMA, so how we can actually use it in our workloads.
And let me also give some quick introduction how actually RDMA works together with GPU workloads.
So first of all, for the RDMA, and this is something which is more or less unique for the RDMA stack,
you need to do some sort of server coordination.
So obviously, when you have some GPU workload, you have multiple like hosts and also like multiple GPUs,
but participating in the whole workload.
And ideally, also by defining some GPU workload, you already understand which communication channels you need to set up between different nodes.
And in this sense, like you know the topology.
And but the problem is that you also need to do some sort of a discovery.
You need to basically change some sort of a tokens, which will allow us to initiate or make a connection.
And this is something which basically creates a huge difference between RDMA and the regular TCP network in stack.
When you can actually just connect to the specific IP address, like get automatic allocating outgoing port and port and everything will actually work.
In RDMA, you do need to know exact QP number, QP is actually RDMA socket in this sense.
You need to know this exact QP number of source and remote destination and actually both sites should know this information beforehand.
So this is why we have a front end network and this is why we need the server coordination.
To basically exchange all required information for every nodes to be able to establish a RDMA connection across the whole, across the GPU workload.
This was the first step and the second step is basically basically initiating a RDMA connection, which involves a lot of heavy lifting on the kernel site.
Because you need to create a lot of objects, you need to transition this QP between states between different states, basically to connect between different remote nodes.
And also you need to basically upload some of the, some of the comments, some of the information inside the NIC in order for RDMA connection to be established and to start functioning.
And the very last step of this set of preparation is actually mapping basically natural interface can fix space into the user space.
So basically creating this bypass mechanism, which basically brings the whole brilliance of RDMA stack there.
This happens on the last step and basically when everything is connected, when everything is initialized, when we also mapped memory regions towards specific, specific GPU NICs.
Sorry, to specific network interface, to specific GPUs, we basically say, okay, now everything is ready, let's map map the control control path into user space and allow application.
Her domain application, collective library, everything just worked directly with a natural interface.
So this is how this is the role of Linux kernel inside the specific process.
And the, basically, the most interesting part is that most of the job failures must, most of the reliability issues actually happen during step two during initialization.
So most of the time on practice, if you were able to establish all the connections, and if you were able to at least once run, send some data through them, they probably will stay reliable up until full job completion unless you will run into some real hardware.
Some real hardware issues, hardware or network issues, which are obviously out of scope of this talk.
But basically there are quite a lot of situations when initialization may fail and basically around 10 to 20% of a age of crashes, they actually related to this system called errors,
in which by GPU jobs like this is our metexample, but I think this should be a general thing at scale for any other company.
And specifically for us, for last six months, we actually observed around 32K plus GPU hours related to system call errors reported by our failed GPU jobs and 5K plus of them were actually attributed to our DNA subsystem.
So this specific issue is pretty significant, plus sometimes you can also run into very nasty situation, which hurts, I should see flagship model workloads.
Quite a lot is a crash loop. So basically if your system, if you're set up, if you're configuration actually run into some steady erratic state and your job cannot actually start.
Just keep wasting your resources in attempting to start the workload and this is the biggest issue and this attracts a lot of attention from everybody around.
And so this is why actually system call and nerdy may system call errors are really important in a real production if you really try to sustain a thousands of GPU training.
All right, so I hope that I was able to demonstrate to the importance of this specific specific topic. So let's talk a little bit about what we each type of errors we can actually see during the training and how they are related to DNA actually.
So this is a typical training stack we may observe for example at metexample like we have by torch and the model called as application layer is a full user space.
It actually uses collective libraries and this is a basic area I work in, collective libraries such as NCCL and media collective library, which does all the heavily, all the heavy lifting for ML or AI engineers in terms of organizing establishing and sustaining GPU to GPU connectivity.
In order to do that, of course, we need to interact with Rdma as well and here is where Rdma core set of libraries such as the by be verbs comes into play.
And basic it handles all the business but still on the user space layer, it handles all the business related to Rdma related to network interfaces and its connection with GPUs.
In reality, for example, when we map user space, when we map NCCL and fix space into the user space rather, it's exactly Rdma core, which contains this user space driver to interact with the network interface and it actually provides standardized verbs interface up to the, for example, NCCL library.
But again, in some situations, we still need to interact with the kernel and Rdma core obviously interacts with Rdma subsystem, which also contains with two vendor specific and vendor independent part.
And basically vendor specific part of Rdma Rdma subsystem and Rdma also interacts with the, with the NCC firmware or with the NCC hardware.
And as you can imagine, there could be issues at any layer of this specific stack and there is a plenty of work to troubleshoot it.
But let's focus on the system call errors, so something between Rdma core and kernel.
So as I have mentioned, they impact why it's a quite significant fraction of our jobs and we really had some that actually this work started with investigating of a single issue when we started observing some pretty important GPU workload entering crash looping pattern and emitting the specific system call error.
So what basically what what was the obvious idea which came to my mind is basically to use a trace in the function graph mode for us to be able to understand what's happening inside the kernel just to understand what specific part of the kernel actually throws throws an error and what's going on.
But it was like pretty actually a tedious process and also to go out to the manual analysis of the whole trace.
And also at some point we also realized that we have actually a lot of such system call errors happening in production.
So we understood that hey this, this manual approach is unscalable and we need to figure out something which will help us to automate this process of identifying and instigating Rdma system called errors.
And basically the obvious solution is EPPF so probably I just I won't mention all this benefits of using EPPFs for this specific kind of stuff because like it's pretty obvious to this audience.
But just want to mention that again, as I mentioned before, Linux kernel still plays significant role in GPU workload in civilization and again most of the issues are actually happening like if there are some issues happening with the GPU workload they're actually happening during an imposition phase and they happen inside the kernel.
So this is by BFF is a perfect solution for us to to look and understand what's going on there.
And before I will jump to actually details of our solution I also would like to mention very also refer the previous presentation because in the Rdma of course there are some situations when we may actually see failures during the data flow parcel like this third stage when we actually actively run communication and use user space.
And this is why this is this is actually the story about in lining you probes and all this stuff just because like for example some of the network issues are actually reported only to user space and this is one of the wired features of Rdma stack that.
If you if you actually get your QP transition in the theoretical state this information may never reach kernel so basically only user space will well know about that and if you want to get observability for such specific issues you need to mess with you probes and in lining is one of this specific pitfalls you need to manage on the come space with the tracing you probes and tracing of the image stack as well.
All right so let's get back to the topic of using bpf for gma system called error tracing and yeah basically when you decide to use bpf that you actually facing three three class of big challenges and let me go one one after another.
So the first one we need to actually select specific trace points specific functions you want to watch inside your ms stack and there are some caveats there are some interesting gotches there as well because.
Rdma subsystem I be very subsystem has only single entry point which is the single IOCTL related to I be verbs and this is the problem because like if you start tracing it.
You won't get enough information so you won't understand what's going on you will just like get some obscure error message and as you probably all know.
Sometimes every system called may return single error message for multiple different situations for multiple different problems so this is this is clear and issue if you have a single entry point to the to this the whole tree of different parts of the whole tree of different functionalities handle handle bait.
So actually in order to to be able to to pick proper trace points we applied static analysis so we just try to look at this whole call graph produced by.
And we also applied couple of heristics first of all we avoided read on the calls because like if you query some information this probably won't be to any workload favor and we are actually not interested plus also there could be some read on the system calls when you agree for some information just returns you know and no entity error which is completely fine for us.
We also tried to ignore sub trees related to pure kernel common code functions because like after all like if you have some management function function failing.
Probably it will also be visible not inside not only inside their day may but everybody else so this is why we decided to ignore this specific subtree.
And we also tried to avoid clean up reference counter functions just because like during the session you don't usually do clean up and in this situation this is why you you're not looking into it.
So this is the example of least of functions we actually used for for our for our tracing solution.
And then basically like we have established the set of trace trace points let's also think how we will implement that on the technical side.
So basically we also had this problem to decide between key probes, key red probes and the factory effects it.
But in the end we actually pick key functions we picked essentially effects it just because we actually first of all we did not need such huge flexibility provided by k probes because after all we only needed.
Return values and we only need specific functions so like hey you just to get to get the signal if you have entered this specific function or if you have left this specific function so we did not need all this advanced functionality provided to us by key probes.
Also speaking about stability of the kind of interface I should say that key fonts are actually a little bit more stable than key probe in terms of like function like if you're if you're lucky actually function names are not changing quite frequently so in this sense in terms of.
In terms of long term support using using key functions actually is more more pleasant but more pleasant thing to manage it.
And the basic for us the main goal was the lowest possible overhead because we wanted to have a scalable solution to monitor every GPU host in the fleet and obviously we may see different workloads with different corner cases so we wanted to.
Protect ourselves of any interference with the GPU workload because after all like if you bring some overhead you actually wasting GPU hours and nowadays GPU hours are pretty expensive for every company for every customer.
Of course like when we did our implementation we run the comparison between key fonts and key probes.
Applied to our workloads and then in this sense we see that basically a fancy effect the thing is roughly takes roughly 7 to 2% of a key probe so it was like.
Step ahead for us and it provided us some headroom in terms of enter the forward head so we actually used kind of benchmarking tool just for pure comparison and also we run nickel benchmarking tests.
Basically we see around some synthetic workloads to understand how it will actually change the installation time of some synthetic GPU workload.
I'm so basically ended up with key fonts and here is the whole enter and solution design so basically we have multiple.
Affentially effects it drops placed on in the specific spots inside the kernel inside the RNA subsystem all of them actually collect unit events we put inside.
Bpf ring buffer Bpf map with around 256 kilobit buffer and like around for around 8k 8k records.
Then we basically have a user space application which kept pulling this buffer and the loading of this information and actually like.
I should say busy pulling with some sleep intervals was enough for us to be able to not use any information and to get all the events we actually expected.
And basically we export it all the data in our observability DB which is scuba and I think like this although this specific system is not on the source it's actually public.
And but you know for your own solution at home you can use actually any observability database you want.
And also in addition to just emitting events we also implemented some counters which we exported to our time series database just for us to use for a lot and for the whole accounting for specific events or specific error knows or.
Just understand what's going on inside the system.
And yeah below you can see the example of some of our events detected by our solution.
And I've also during deployment we also did another researching in order to again to to ensure that we actually do not introduce any.
Regretions or any specific issues and as you can see like this is a graph of CPU utilization on the host.
Election nothing changed in this sense plus also we actually implemented this.
Regretion matrix or solution insights net edit because also I think it's it's also a non solution.
It's also public information available on the internet about this net edit system so like we have a huge I should say well this why distributed binary which manages multitude of BPS running on the host.
For example attribute it to the to the network in stack and our BPS related to the matrix actually took.
Like it was the probably the most lightweight B pf B pf on our fleet.
And let me also quickly touch upon the third main challenge is that okay we have this monitor and system so we get a signal that something is happening at scale.
In the fleet but unfortunately the problem is that you still need to process this signal you still need to go around some manual tracing you still need to use.
If trace or you need to use red snoop to get the whole calling stack and just to understand okay which specific place in the kernitors returns this specific error code.
And basically there are again another some some types of characteristics which help us to manage this specific situation and also you also need to remember that kernel changes and everything changes so we also probably need to extend the set of your trace points in your workload as well.
So you need to run ideally you need to run this periodical audit and you need to understand what changes inside the code inside errors and their code traces inside the kernel.
But unfortunately we have not able to automate it yet so we just know that this is one of the directions we need to move in order to create a evolving system related to monitoring the errors.
So yeah just a couple of words about the Demetraiser in production today so it actually significantly reduces diagnosis time from like tens of minutes to actually second say it allows us to practically detect some emerging the mayor system called issues on our fleet.
And also it help us to engage less people and instigation and instigating specific threshold for specific situations so yeah this is this is a success for us.
And basically like utilizing last couple of minutes of my report I also can mention our case study so this specific word actually started when the system called.
One of the system calls called IBV registered me the me above so actually it was ABV registered memory region failed with you know man.
Erick old yes you see a personal term it's about this is a different example.
So it actually resulted in a man and initially we thought like okay we're actually lacking some memory so like the the host is under memory pressure.
But when we started investigating we actually discovered that this happens during utilization so basically this situation memory usage is pretty low and also.
All numbers are actually verifying so like we were okay on GPU bar GPU bar usage on the memory usage so nothing will actually trigger this you know menstruation and we when we started to investigate that we actually.
Discovered some because like we use some like proprietary code and so they're okay all it there was some.
A variable conversion issues between like 64 and 32 bits integers and then we actually implemented this solution and we were able to discover another issue a little bit later which actually.
Flore it up with the same symptoms but it was the complete the complete different situation related to some issues inside the wonder when the GPU driver and huge pages config we used in production.
So yeah just a quick summary.
Again the dummy system colors have a system call errors have a significant impact at scale and so this is why we actually implemented the matrixer which is scalable because of design choices because and because of that just using bpf I should say.
And it the matrixer actually is a nice three lines are diagnostics and detection process and allow the lowest us to automatically see five issues as well.
So yeah thank you so much.
Yeah I need questions.
Just a little bit clarify so from the bb program what exactly information do you collect is it just the whole stack trace is it just a fact that there's some not in this called graph was hit or what exactly.
And so basically yes just the fact that we specifically we have this specific function not the system called but function inside the kernel was hit and also there are a total of this specific function which actually allows us to recreate the picture and understand what happened.
For this system we collect only selected traces and then when we basically understand okay this is a significant issue we go we jump and go it's trying to recreate the whole call trace to understand that can basically be the issue.
Thank you.