All right, can I at least some people hear me?
Yeah, all right.
I hope you're all here for the next talk, right?
You're not here for this one.
I hope okay, anyway.
So I'm JPE, I work at AMD.
If that wasn't obvious, I put it there as well.
First off, I wanted to say that I'm also the
Organized of the LVME TARP in Domestead,
or the LVM Social.
Like those were mentioned.
So if you are around the Domestead area in Germany,
you know, we have a meet-up every last Wednesday of every odd month.
So the next one will be in March.
So if we're free to join us, we don't know yet, you know,
what will be the topic, but anyway.
All right, so I'm going to talk about LVM offload.
Anybody heard about LVM offload so far?
Raise your hand. Oh well, okay.
That's more people than I expected.
Cool, that's great.
First off, very important disclaimer.
Most of what I'm going to talk about is not something
that I actually work on.
So the credit goes to all the people who actually
like implement the stuff.
Mentioning mostly AMD codeplay and intel,
and there's also LNNL working on this,
at least that's what I saw from the PRs,
and those are the people who joined the community meetings that we have,
which are every two weeks,
and they alternate between the OpenMP,
which is mostly OpenMP offloading in LVM,
and this like general LVM offload meeting.
Okay, so let me start off with some sort of a vision of the LVM offload.
By the way, are you showing me minutes for the full slot?
Okay, awesome.
So the vision of this is that we,
in accordance to kind of the LVM idea that we provide
a composable library, or a set of composable libraries,
that allows you to build GPU offloading libraries on top of it.
Right, so let's say you have some language
that you want to support GPU offloading in your language,
and so you can build on top of these LVM offload libraries
to actually make that happen.
So you don't need to worry about a lot of these things.
And the other important thing is, of course,
it should be cross vendor, right?
It should not just be AMD or intel or in video whoever, right?
It should probably also be not necessarily GPU specific,
but right now it is GPU specific.
And the green should be blue, but that's just anyway.
So history.
I duck through a bit of the history of the LVM offload project
to see, you know, how's it going?
And a lot of the credit here actually goes to your Honest Erfor
who started this with an RFC on this course on the 22nd October
in 2023.
And then we had a lot of discussions around how it should be done.
So everybody agreed, hey, this is a great idea.
We should go ahead and do this, right?
We had Lip on Target, which is the OpenMP offloading library implementation.
That was entry for a long time,
and we wanted to move it into this top level offload idea.
How should we go about this?
Should we just create a new one from scratch?
You know, do everything from scratch, and then at some point
re-implement Lip on Target sitting on top of it,
or should we move Lip on Target into offload?
And then, you know, make it better, make it so that it actually is
LVM offload.
I'm a fan of starting from scratch with the offload library,
and let's say we did it the other way.
You know, sometimes you win, sometimes you lose, I guess.
But so we decided that this is the way to go,
and so on the third April, 2024,
the PR got landed that actually put this offload top level
of the directory in three.
And two days later, there was a heads-up post on this course
that this is going to happen, or just kind of an interesting
time on here, but anyway.
And then, again, a little bit later on the 22nd of April,
the PR landed that moved Lip on Target into Lip offload,
or into offload into the directory.
In between, we did a lot of work upstream and downstream,
so our rock is a fork of the rock and compiles
as a fork of LVM, and we needed to figure out some of the
mechanics about this move because we have downstream stuff
that's downstream for several reasons,
but that's a different discussion.
So we finally worked with the community to get this done
and without breaking everything, so that's great.
And then, that's still Lip on Target, right?
And then, basically, we did write the offload API
from scratch why we also moved Lip on Target.
So we're kind of doing the thing that we decided to not do,
but we did it nevertheless.
So that was kind of interesting.
And then on 25th November, 2024,
APR landed that introduced the first offloading API,
and I see we get some, we get lighted here.
Thank you so much.
I see the light.
Anybody here loves the movie Boost Brothers?
No, I'm the okay, I don't know where.
Anyway, so we now have the offload API.
And then, quite recently, on the 15th December last year,
Intel landed the PR that actually adds support for Intel GPUs
into offload, which is great, so you can now offload
onto Intel GPUs without stream trunk.
So what is the offload?
So let's look a little bit of inside offload.
So you have some sort of user application at the top,
or maybe an offload library that you're writing yourself,
and you have some accelerators down there.
And then there's the good ol' Lip on Target
that everybody loves for the flakiness in the testing, right?
And so, we have all of this infrastructure.
And then, as I said, right next to it,
there is the Lip offload, which has an API that we're still developing
and it has a bunch of unit tests.
And probably one of the more interesting things is that
beneath both of them, we have what is called the plugins.
And I think for some legacy reasons,
they're called plugins next gen, because we already had some plugins
and everybody replaced them with some newer ones.
And the plugins are what basically abstracts away
the vendor specific details, right?
So there's one for AMD GPU, there's one for Nvidia,
and now there's also one for Intel's level zero runtime.
And so the plugins expose a C++ API
that then lip offload sits on top of.
And that design is by choice because at some point,
we actually want to have lip offload to be
a stable vendor, agnostic, and generated API.
So we use table gen for the API, the APIs, you know,
everybody loves table gen, right?
Isn't table gen, so it's generated,
and then it at some point calls into these plugins.
One of the interesting parts with
we're generating the actual offload API,
is that we also generate a bunch of argument,
checking, and log capabilities with it.
So you can basically, like for free,
you get complete tracing of the API,
you get complete argument validation of the API.
And so that is why we wanted to settle for table gen,
and then kind of split this definition of the API
and table gen, and having the actual implementation done by hand.
So if you want to add something to the offload API,
the workflow you follow is basically,
you add an API entry point into the offload table gen files,
you rebuild the target, so you would generate a bunch of things.
And then you copy paste, generate the declaration
into the offload impulsy pp,
and you go ahead and implement it in terms of the plugins,
and then you actually rebuild the whole thing again,
you basically now have a functioning API.
So that's kind of the idea there.
And so right now, this is working,
and it's unetested, so pretty much the whole API is unetested.
That's already running in the build box that we have.
And once we have more settled onto,
like how this is actually going to work,
we're going to put together some smoke tests and integration tests,
and whatever you want to have, right?
So that we make sure that it's actually working.
And now to some, so that's offload,
and this is great, right?
And then, well, sometimes upstream versus downstream
is still a bit painful,
and we experience this,
because as I said,
we have downstream changes in the bump target
for OpenMP offloading,
which is the OMpt or the OpenMP Profiler,
the OpenMP Tools interface, sorry.
Now, offload and the bump target use the plugins, right?
So kind of they make use of this thing.
The problem is that downstream,
some of the OMpt functionality is actually
intrusive into the plugins.
So you change the plugins,
you refer to symbols that aren't there,
and then when you actually try to build either of those,
you either get undefined reference errors,
or multiply defined reference errors.
So that's a bad position to be in.
Now, as I said,
the basically comes from the situation that inside the plugins,
you're using OMpt symbols,
and for historic reasons those are defined,
or maybe for reasons that make sense,
those are defined inside the bump target,
because the bump target is again,
that's the OpenMP offloading library,
and so OMpt is OpenMP specifics
or goes into the bump target.
So, what did we do to fix this?
We didn't want to touch the bump load, of course, right?
What we did is,
you basically introduced a new
generic profiler into the plugins,
or into the whole ecosystem,
with the ideas that all this profiling is now implemented
in terms of this generic profiler interface.
So the plugins only call into the generic profiler,
and then the generic profiler,
or at the implementation 40MPT
inherits from the generic profiler,
refers to all the OMpt symbols,
and they are still defined inside the bump target.
So you kind of split this.
And the way that this works is that inside the plugins,
you have only a reference or a pointer
to a specific profiler that you want to use at runtime,
and so lip-on-plug it instantiates
that with an OMpt profiler,
whereas if you're going to use a lip-offload,
it simply uses the implementation of the generic profiler
that's there already, right?
With that approach,
you can also think of having some sort of basic profiler
that would allow you to just generate CSV files,
or your kernel timings,
or for some API functions or whatever.
So it gives you this abstraction away
from being specific to one program language.
Now, we still have OMpt symbols,
kind of used,
referred to and defined in lip-on-target,
and in this profiler stuff.
So eventually what we want to have,
we actually want to encapsulate that into a lip-on-t,
which gives you all this OMpt functionality
that you can then use for the lip-on-target,
and for the generic profiler,
so when you build the open and piece of,
you're making use of lip-on-t.
And so to really package these things
more into specific use case libraries,
then follow this general idea of having
the separate libraries,
the separate interfaces,
these composable things that you want to have,
and so you can think of even for the buffalo
providing other implementations
of the generic profiler that you can link into,
whatever runtime you want to build
when you're constructing this runtime.
And so that's it,
actually for my talk.
I have a bunch of references here.
I have to show you the slide.
And with that, I'm very happy to ask,
to answer questions, of course.
So we have five minutes left for questions.
Okay.
Yes.
Can you speak up a little bit more?
What's the forecast for getting apples and apples?
What's the forecast for getting apple support?
I guess if someone wants to open a PR,
you can do that.
No, I don't know.
I don't know if anybody is working on apple support.
I don't know of anybody.
But yeah, you're absolutely,
if someone has the ability to do that,
I think we would be super happy to support it.
Yeah.
Yes.
So, are you talking about India, India,
etc?
Where would India,
lucky stuff inside this open source code?
Okay. So, I talked about AMD and Nvidia.
And so, where would Nvidia plug in here?
So, what we actually do these plug-ins,
as I said, so we have one for Nvidia,
you can offload onto Nvidia GPUs with upstream right now.
Basically, the plug-in is implemented in terms of CUDA.
So, we just, you know,
your regular CUDA API, so you CUDA malloc,
CUDA launch, CUDA whatever, right?
So, that works.
Yes.
This is also used by Rock Prof,
AMD's Rock Prof V3.
You mean the OMPT stuff?
No.
I think Rock Prof V3 uses their own mechanism
of the way on how they thank you,
their kernel timing mechanisms.
But we can discuss this more offline.
I don't think that's relevant for this audience here.
More questions?
Yes.
Okay. So, what happens if we have multiple GPUs in your system?
Which one gets picked up?
So, for lib offload,
the API exposes basically an iterator
through the different devices that you have.
And so, it would be up to a runtime sitting on top of it
to define semantics on what happens.
For OpenMP, the semantics is that you always pick the first device
unless specified otherwise by a user.
Right? So, in OpenMP, you can say,
I want to use device 3,
and then it will pick up device 3.
If you don't say anything,
it will pick up the default device,
which would be default device 0.
Yes.
How likely is it to combine multiple GPUs from different vendors?
Different vendors as well?
That would be great.
Okay.
So, I don't know for sure,
but I believe you can actually do that today already.
I only know, so the OpenMP lib on target,
you have ways to specify which accelerator you want to use in OpenMP,
and there you can specify I want to use an AMD accelerator
or an Nvidia accelerator,
and then it will match whatever accelerator you have.
So, it should work, you know, TM.
But if it doesn't,
you should open an issue on upstream MLVM,
because it's supposed to work.
Love question.
Okay, that's question.
Yes.
How explicit is the handling of data transfers?
So, in lib off load,
you would be required to do everything manually.
In OpenMP, it's kind of implicit,
because runtime does it for you.
But inside the runtime,
it's again explicit, of course.
In OpenMP, you would say,
Pregnant target teams, blah, blah, blah, blah, blah,
so it's parallel on the GPU.
And then when you hit that kernel,
the runtime will automatically generate
the data transfers for you,
and we'll put that onto the device.
Run the kernel and then pull the data back.
If you were to use the Offload API,
you would have explicit calls to do that.
Right?
So, you would say, as well,
copy host device, launch kernel,
and device vector host.
Because again, so offload is meant as a foundational layer
on top of which you build your actual runtime.
So I don't know, like, cray or HP, HP, X or whatever.
Cool, thank you.
Thank you.