I'll try to hear a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little bit of a little
little bit of a little bit of much a little bit more of a little bit and putting some little bit of a little bit of a little bit of慢慢 like a little bit
of a little bit of a little bit of a little bit
Okay, so now we have Jonathan, who will explain us about all the things that he's doing with the Federation of Education for AI and HPC infrastructure.
Hi, everyone.
So I'm Jonathan Kalmos, and today I'm going to present to you a Federation of Education for AI and HPC infrastructure.
All right, so it would be a background.
I'm part of the Applied System team at Nvidia.
And together we work on bringing up the next generation super-computer for AI.
So essentially we focus on providing a set of best practices for things ranging from like performance, reliability and in this case security.
And over the past few years we stood up numerous super-computers.
Many of them ranked on the Todd 500 list.
The latest one being EOS, which we submitted in 2023, and it's number nine in the world.
So for those who are not super-family with our super-plot architecture,
super-plot is basically a scalable unit that you're able to deploy in your data center.
It's comprised of several racks.
We have one rack that's dedicated for the control plane, where you have logging nodes, management nodes, storage nodes.
And all the other racks are dedicated for compute.
So it's where you're going to have your GPU heavy application.
All of these racks are linked through various interconnects.
So on the compute racks we have a memory fabric, which is very high speed, which is called multi-nodeinvillink.
And that allows all the GPU to communicate very functionally within the rack.
And then across the racks we have a lot of other interconnects, so we have an infinite fabric for storage.
We have one for compute, for things like AI collectives.
We have an internet network for managing the nodes.
And we have one for compute and storage, which is not performance critical.
Things like NFS.
All right, so when it comes to educating users on all of these fabrics and various services,
we had a lot of requirements.
So first off we needed the authentication and authorization of our posics account that's tied to our corporate identities.
I then video all our employees are storing the Microsoft Active Directory.
So we need to be able to fetch those accounts, permissions, et cetera, and comply with policies.
We also needed a unified solution that worked across all our services.
In our case we have SSH, a storm for job scheduling.
We have GitLab for continuous integration.
NFS and Luster for storage.
UFM, which is for a sharp, collective overflow to switching fabric.
In case you're familiar with that.
And then we have IMAX, which is for multi-nodeinvillink, which I just mentioned.
We also needed something where you can authenticate on prem, as well as in the cloud.
Short-lived credentials.
We want to rotate these credentials often, so we want short-lived credentials.
And we didn't want any client-side magnification.
So we want our users to just be able to connect to our data center with plain SSH.
We don't want maintaining clients on their side and having to upgrade that and fix vulnerabilities.
And then we also wanted multi-factor authentication and single sign-on where we are applicable.
And other things like we wanted it to be easily auditable.
These are friendly, very minimal data center footprints.
And so when we check out all the solution that exists out there, we sell on a free APA,
which is basically the only option if you try to comply with all these requirements.
And that's when we went with.
And so Kerbos, which powers free APA obviously.
All right, so to give you a little idea of the overall architecture.
So we have free APA, which has a cross-force trust with our Active Directory,
the corporate Active Directory that we have in video.
And from there, our clients, the users, are able to authenticate to our login note through Active Directory.
From there, they'll get an issue to tick it.
And then with this ticker, they can actually submit a job through sperm.
At this point, we'll register their credential and they're ticked inside service that we call civil.
I'll talk about that later in the talk.
And then at some point in time, the job is going to get scheduled.
It's going to end up on some compute note.
We're going to retrieve the credentials.
And from there, the application can access very services like storage or all the things I mentioned.
On the continuation side of things, it's slightly different.
Here, our users are actually running pipeline from GitLab, which is outside of a cluster boundary.
And GitLab is authenticated with OpenID Connect through enter ID.
So here, what we do is slightly different.
We effectively do a protocol transition from OA to Kerberos.
So effectively, what we do is we impersonate the user as if he was actually submitting from the login note.
Even though it's pipeline that's been triggered through GitLab.
Again, this is a chief through civil, which I'll talk about later.
All right, so as far as the support of authentication flow that we have.
First off, we have password authentication.
This is the classical Kerberos pre-auth, where you issue a password, you get ticket out of it.
So here we authenticate your active directory.
What we do, though, is we add a second factor in the.
So we add a second factor as an SSH key.
So you need your basically corporate credentials as well as an SSH key.
And the public key is stored in ID view inside free IPA.
So we can rotate the keys of our users and force them to update their second factor, basically.
Note that you usually want Kerberos armoring when you do these kind of things,
and also validate TGTs to avoid some sort of class of attacks that exist out there.
The second thing we support is where we call active directory single sign-on.
It's essentially where you client is already domain join.
So you have a corporate laptop, which is already bound to AD.
And you already have a ticket by the virtue of authenticating on your Windows laptop.
In this case, we allow our users to come in with their existing ticket through what we call,
what is called as unconstrained delegation.
So we'll delegate their TGT to our login nodes and from there same flow applies.
The downside of that is you usually need KCM on the login nodes to refresh the credentials,
because once delegated the credential won't get refreshed.
And so to avoid our users having to refresh manually their tickets, we use KCM.
The second thing is that on Windows at least SSPI and constraint delegation doesn't work
if you have credentialed guard turned on, which is a bummer, but you need to turn it off for this flow to your apply.
Slight variation is entry-acloud turbo stress.
This one is when your client is actually authenticated through the cloud.
So he has a PRT from entry-ID, but he's still has a leg on your corporate network.
So as far as I know it's only available through Windows Hello for Business,
but what Windows will do is we'll automatically convert the PRT to turbo stick it.
And from there we, it's the same flow as I explained before.
You delegate to tick it and you can reuse that on your infrastructure.
And lastly we support cloud authentication.
This one is achieved through the external IDP.
Support that free AP has has been describing in earlier talks.
Here we, so free IPA does the open ID connect device code flow,
which is entry-ID, which effectively grants you a carburett ticket.
Now there are some challenges with this approach.
Essentially, it's not AD, but free AP, we use a ticket.
We have to switch back to the Active Directory identity.
And for this, we use a local association.
Once you're a session on the login node, we re-assess you locally,
so as to change your identity with the Active Directory one.
So some benefits.
Using SSH will put you in new login session.
I will go through PAM, so you'll get a new kernel of hearing, things they did.
You'll go through turbo certification.
And SSH will also copy or take it, which is good.
If you were to use something like case, you wouldn't work.
So we had to invent if here.
In the future, we would like probably to use system D, run zero for that,
but we have to upstream a carburett support for it.
And other challenges you have with this approach.
Microsoft on devices, which are platform-assist those,
so that's Mac, Mac with Safari, and Windows essentially.
The device could flow, well, not work with single sign-on,
so every time you do this flow, you will have to re-authenticate in your browser.
You won't keep the cookie in your browser.
And to counteract that, we suggest our users to use SSH multiplexing,
so that they only have to authenticate once,
and they're controlled versus basically to a limit that's below their ticket lifeline.
All right, so quickly on the user experience, it's fairly simple.
The first flow, you get a password, and you have your SSH key.
Second one is AD single sign-on, or the cloud carburett stress.
You use SSH minus K to do just a CPI, uncontrained delegation,
or SSPI delegation.
Last one is the Entra Multifactor authentication.
This one will prompt you to do the challenge in your browser,
and then here, you'll get veto to whatever your policy requires.
And as a result of that, you get your usual tickets,
that allows you to access various resources on the cluster.
And for us, we just, then something to explain the job
with a container image that will inherit the telekering.
We've already a ticket in it, and everything just works.
All right, so I wanted to talk a little bit about civil,
which is a project we developed for bridging the gap of batch
and continuous integration workflows.
So civil is kind of a small project.
It's the mix of C and Rust.
And you can see some mostly of three components.
There is a server that you host alongside the free APA.
There is a CLI that you use to delegate or impersonate user credentials.
And then we have a slim plugin that will automatically do that
for you as part of the job lifecycle.
If you're familiar with AUX, this improves AUX on number of ways.
And you also add support for impersonation through protocol transition.
I'll describe that.
So it can basically forge TGT as if it was a KDC.
And we're still working on the full documentation for it,
but if you're curious, please check it out.
Okay, so the way it works.
So it's a little bit complicated.
I'll try to summarize.
So your user authenticate on the login node as usual.
It goes through the AD authentication.
It gets a ticket.
From there you will ask for them to submit a job.
So I'm going to allocate some credentials.
And from there, our plugin, the spank plugin, in slurim for civil.
We'll get your ticket.
And we'll send it to our civil server.
Our civil server in turns stores it in KTM,
so that the ticket will continuously get refreshed
until the job gets scheduled.
And at some point in time, the job will get launched
on the computer nodes.
And same thing on the computer,
we have our civil plugin,
which will retrieve the ticket,
so that the job can have all the credentialing needs.
We'll also spawn site-card process.
There will keep this ticket alive.
And it does that by reaching out to civil D.
There's a reason for that.
So that we don't do excessive request to the KDCs.
Pearl, because it could involve some cross-round interaction here.
So we only hit the civil D when all the nodes
basically start to run the job.
And the other thing is that if you take it word to expire,
while your job is still in queue,
what we do is we notify the users.
And we ask them to be really authenticated on the cluster.
The way we do that is we have a small wrapper around K&IT
that summits your ticket through civil.
And we'll also have a hand plug-in in civil,
so that if you adjust access to the cluster,
we'll automatically forward your ticket to our server.
And for CI-CD, what we do instead.
So we do something called as protocol transition.
It involves the service for user extension of Kerberos.
So here we have a user that's authenticated against GitLab
through OpenIG Connect.
And this in terms of authentication against Entra.
And for the GitLab summits the pipeline to our custom
GitLab runner that we developed.
This runner is going to use civil to essentially do cross-ground
protocol transition.
So it's going to get a service for you to self-ticket
for itself impersonating the user.
And with that, it's going to get a service for user to
proxy ticket to the civil server.
At which point you will be able to request basically the
forgury of a brand new TGT that represent the user.
And basically impersonating it to without
involvement of the user.
The thing to note here is that the actual ticket that's
being generated by civil doesn't exist in the KDC.
It's is effectively just a shadow principle of what
exists in Active Directory.
But we make sure that basically those two principle maps
to the same user so that we are allowed to do that.
And yeah, that's pretty much it.
This is a little bit more involved.
So if you have questions about that, I'm happy to answer.
Otherwise, just check out the code.
It's fairly simple.
It's a conclusion.
So free IP bridges the gap between corporate identities,
cloud authentication, and kind of the traditional
physics account that you have on HPC clusters.
We were able to implement all our authentication and
to end across all our cluster resources.
The only thing we had to do is develop these small civil
server to address some of the scenarios that
couldn't be covered by standard Kerberos and free
IPA.
And as the next step, we would like to look into
Kerberizing our slurmo authentication itself.
Right now, Slurmo does its own authentication across
the nodes.
Maybe also look into MPI authentication.
MPI is a very well-eversed in HPC.
So we would like to try to authenticate that.
And I mentioned it, but we would like to replace this local
SSH in our multi-factor authentication thing with
system DRUN zero in the future.
And that's it.
And before I take any questions, if you're also curious
about all the other types of authentication that we have,
I have some backup slide.
You have the SSH one, the NFS one, the last one,
the SHARP and multi-nodding link.
There's a really standard Kerberos stuff.
The only thing I would note is that for LASTER,
we did a lot of work with the community to
bring it on par with an FS.
And so if you're curious about this one,
feel free to look at the LASTER user group conference,
where we describe all the work we did there.
And with that, if you have any questions.
So thank you very much for this talk.
So we did a review of SIPL code.
And Simon and I, we believe that what you are trying to achieve
could be achieved with the standard Kerberos methods
through the SPU.
What perhaps we are missing is...
The slide difference is that the S4 is the first step
to actually get a full TGT.
And I think that's what you don't like.
You don't need to do a full TGT.
So what we do is product authorization,
you get a ticket to yourself, S4 to yourself.
Yeah, the thing is we don't want that because
they are users are able to SSHU the remote or
basically you need a full TGT on the remote
notes to then have the user be able to access any ticket.
Yeah, this is not on the user side.
This is on the SIPL as a service side.
Oh yeah.
We can do this product authorization there.
We don't need full TGT or the user.
And at every place where you need full TGT,
you need our RBCD rule and product authorization.
That's all you need.
Yeah.
The only problem is that you don't have a head of time
with the user once, right?
So you don't know what service the user is going to
access.
But we have pack information where we can put
additional details that can be used
during the authorization of RBCD rules.
So we can solve this much more secure
without the ability to leak any of those things.
Yeah.
I would be interested to have that even in the KDC.
I looked at maybe doing like MIT,
Cabro's plugin for that type of things.
Yeah.
I think that would be very useful.
I think CI and Procultransition in general is kind of
not addressed by free idea.
It needs a bit more context than you get
in the raw KDC with metrics to get the authentication
indicator support.
Okay.
Which, for example, we'll allow you to limit
GSA API authentication in SSH
to the point that only users who used
enter ID to authenticate allow it to pass through.
Yeah.
I know there's several patches that are not upstream.
There's like, I think the GSSR key that's not there is
I think the SSH upstream, just as if you are only
is limited to storing the ticket in a file,
not in the kernel keyring.
There's a bunch of patches I know that debut
and red hat have.
That would be great to have all of that in open SSH upstream.
But yeah, we don't necessarily need that.
Okay.
We still have time.
So any other question?
Okay.
So thank you.
And I'm pretty sure we are going to fix those things
together.
And yeah.
Thank you.
Okay.
Yeah.
I'm not sure if I can get this one.
Oh yeah.
Two out of three.
Yeah almost let me just check.