Hello everyone. Welcome to our presentation on building cloud infrastructure for AI. I think
everyone's quite familiar with what cloud infrastructure is. Today we'll be looking at it
in a context of AI. By AI, I simply mean accelerated compute. For all intending purposes
today, it will be GPUs. Who am I? I'm Dave Hughes, software engineer, system admin, network
person, done it all at some stage. Passionate technologist. I am a open source champion.
I love open source. I've been a digital maintainer at one point in my life. Contributed
back to various user space, normally bug fixies when I've been packaging. The user space
software, whatever else. Primarily problem is over. I like to solve problems.
I'm a customer. My background is originally in typefones computing, parallel computing.
That's sort of thing. I got into GPU compute when it was really new and everybody was excited
by Cudon OpenCL. Got into rendering through that, been contributing to Blender for 11 years
now. Through a pure coincidence, I grew working on distributed rendering for that. I got into
the whole cloud infrastructure area and I ended up really enjoying it. I stuck around
and here I am. Founded it's really cool. I'm still very passionate about making things
perform fast, but the whole performance aspect of things. I found it a really enjoy thinking
about and building the big picture, not just hacking away on one component of the stack
for years at a time. We're going to see a lot of this big picture later on.
First of all, what is a cloud? Simple. We have looking at it. I want to skip through. Again,
abstraction of resources and infrastructure. Self-service for developer teams on demand. Elastic
nature. I think everyone's quite aware of that. API-driven, multi-tenant. Kind of obvious,
but not obvious. How does it factor into GPU? GPUs are typically, other accelerators will
be common in the future, GPUs that type of thing, but GPUs used for accelerated compute. Typically,
RDMA interconnects, if you're doing a big cluster, not always. It's quite common that
you'll see cases where people will throw consumer grade GPUs into servers and do things
like that with no RDMA interconnect. Fast storage. I do want to caveat though. It's not necessarily
fastest. Everyone seems to get hung up on this topic where they think they need to be fastest.
You just need an acceptable level. Not necessarily RDMA either. There's been a bit of a discussion
on this. A company-bending name of XTX released a new distributed file system recently named
TurnFs, and they chose not to use RDMA in that either, and generates quite a bit of discussion.
Dense, very big servers. Again, we think a big, beefy storage server historically, but these days,
it's very big dense compute servers. Pilling as much as, like, between 12 and 20 kilowatts per node.
Free rough skills of consumption in this model. A single GPU, again, just passing through the GPU
to a virtual machine, or an entire node that could be 4 GPUs in the box, it could be 10,
it could be 8, it could even use envealing if you're using Nvidia. Could be cluster,
and that's where I came back to earlier. You're dealing with cross-note interconnect,
and basically tying thousands of these things together. GPU clouds, the good to bad,
it's in the ugly. So there's a few different ways of what these GPU clouds actually come together.
First of all, you'll see many people sort of masquerading as a cloud,
but what they really do is purchase assets, image the assets, and simply hand over the keys to the
BMC, and now you break who buy type deal. Other people will throw open stack on there. It's quite an
obvious sort of path to a quick MVP. It's rough with problems, I would say. Quite tightly coupled.
I mean, many cases where we couldn't have high availability on control planes, you would restart
DHCP server in the news part of the network. It's nice in a lot of places, but it
detect coupled nature of many of the components makes it really difficult to use in production.
Arguably it's designed for private cloud operation on your own premises or whatever,
and people insist on using it as a public cloud for multi-tenancy, and it's not ideal.
This is coming from experience of running it in production for like half a decade or more
and trying to build a GPU cloud with it, not hacking around for six weeks and deciding it
wasn't fit for purpose. Kubernetes, another one that's sort of used today, and people seem to think
that they can throw a workload manager onto a cluster, and all of a sudden it's a cloud,
arguably not. Out of the box no strong isolation of tenants due to the nature of using containers
first as opposed to virtual machines on Linux, shared kernel and drivers, not exactly ideal when
you might have people that want different driver versions for their compute instance and so on.
Various other parts and networks, not many of the common networking options for Kubernetes,
like Kaliko and everything else, not quite what you want for a cloud infrastructure of the box.
Other, you get a few other options that normally come up in cloud infrastructure,
and no mad by Hashecorp was used quite a lot again for additional sort of cluster workload
manager. Some people have successfully built things out of that.
Triton was the cloud stack by giant, very popular back in its day, still being used today by M&X.
How to design a cloud 101? We're done today everywhere, again, kind of obvious, but not obvious.
Scheduling maintenance windows doesn't really scale when you've got a thousand customers.
A lot easier to just be able to take the control plane out of production and still have to available
in the infrastructure. Avoid as many manual steps as possible during bootstrapping, network
allocations, everything in there. Isolation, again quite obvious, most people think it's purely
for security and privacy. It's actually for QLS as well as security. Standardization quite key,
again obvious, but not obvious, but being able to move customers' compute instance
is machine to machine will save you in production. Minimum state, we found this, again, the sort of
MVP that was originally built for the GPU cloud that we had was state full everywhere and it quickly
became a nightmare. We sort of came to a conclusion that if you kept sort of core infrastructure
state full and had everything else stateless. Part of that was down to the not going to say data
centers, but facilities we were operating in former mining sites, power could flap constantly
and if you've got power flapping on a state full infrastructure, it's a nightmare to bring it back up.
A hardware selection for this, there's a few comments that are passed to this.
Standard path normally is OEM kit. Again, Dell, HPE, whatever, it does what it says in the box,
I think about three or four years ago, it was quite difficult to combine most of this kit.
I've super micro were one of the few OEMs actually selling HPE servers in any particular amount of
number. Dell, everyone else doing it these days. Again, the most identical, the all-follow sort of
reference designs from AMD and Nvidia, so on quite overkill and specs. Most people don't actually
need an HPE user server. I think four-way are quite common these days as well with a focus and
shifting to sort of inference in data centers with lower power density. Another important point on
the overkill specs is that often they will have a single spec where they say, if you want this box,
you also need to buy 60 terabytes of SSDs, you also need to buy 800 gigabit nix because that's just
how we sell it. Most people don't be that. Next one. Dell stands at approach. Again,
mainboards, PLX, switch and GPUs. This sort of approach originated from the mining and VFX rendering
or, again, standard server with some GPUs and not great from a density perspective, but if you're
operating in legacy data centers where you've got 8 kilowatt racks, if you've got an AMHP server or
something with a couple of three PCI slots, it's easy just to drop a card of two and then actually
try something out. OTP has obviously been proven to be king at scale, but you really need the
scale to justify non-OEM kit. You're going to have to either sit down with ODMs and Taiwan in
really based or designs of the open compute projects hardware. firmware. I do world, you want to
sell control of the firmware, both from a security perspective and scalability. By open firmware,
we typically mean core boot and open BMC. It's really, really rare to find this in the wild
unfortunately. There's a lot of room for improvement here. I have been really happy to say that
there's active work going on from some of the vendors. Super micro-contributed quite heavily to do
getting intel sapphire and emerald rapid CPUs support into or chipsets support into core boot.
I think the super micro nodes we had in production previously, they had a sort of work in progress,
core boot port for that, but it never finished. Seeing a lot more work being done, though, there.
From the open source community, we've seen obviously great work from free MDB and nine elements
over the past five, six, seven, eight years, plenty others in the community as well. We've got red
fish on through an IPMI solution. Also, fun facts. You find the case where in video GPU
trays have their own BMC, which is good fun when you're trying to control your infrastructure.
Right, with that now we have hardware, we have firmware, but we need something to run it on. We have
a lot of workloads, both from the customer workloads, of course, the people are actually paying,
you have internal workloads, all of your services that you need to run the customer workloads and
supporting workloads, things like monitoring, logging, the bread and butter stuff that everybody does.
We decided to go with Kubernetes here and not out of any deep love for the technology, but just
it does the job, it's widely known, it's widely supported, whatever, pick your battles, that's not
something to spend a lot of time on when there's something to work. Need to stress, we use this
purely internally, it's not like the case we mentioned earlier where people just let people
let the customers run parts and their Kubernetes know, this is purely internal, all
customer workloads are behind the virtualization barrier, just for security reasons.
The nice thing about Kubernetes is that you can also use it as a platform for managing your
own resources, so called CRD is, this is quite common for projects that build on top of
Kubernetes, things like QBurt or Rook or whatever, but you can also use it for your own application.
Quite convenient, like all of our internal state things like Overlay Networks, instances, IP
allocations, all of it is model as a customer resource and Kubernetes, and that gives you
right, and that gives you automatic API implementation, things like patching endpoints,
resource validation, watch streams, libraries, text UIs, all of that for free, which is
quite convenient, and then the rest of the stack, all of the internal services that I mentioned
are basically built as controllers, operating on these resources. In our case nowadays, mostly written
in Rust, similar reasons as Kubernetes not out of any deep particular left, but because a lot
of people on the team knew it, the others want to post to it, so it's what we use now.
Boxstation is nice, but you need a new devs host to run it on. For that, the traditional model is
probably put the server in a data center, installs an OS on it, maybe have a network boot
based solution that images the servers. Problem about that is we did this earlier in earlier
iterations, and over time you end up with a lot of state drifted, you're like, oh, we kind of
reboot this machine, this important customer on there, so at some point the images two years out of
date, and then you have something with Ansible and all of the machines are different, and it's not great.
So solution here is, again, as mentioned earlier, minimize state wherever possible,
in our case this means that the servers are completely ephemeral. They boot over the network,
they load OS into Rambus, can just run from that, and when you reboot them, they just pull
whatever is the latest image, and now you have to use data on them. Some status required, of course,
in practice the Kubernetes control plane needs to persist things across reboot, and set for storage,
of course, also needs to be persistent, so that just gets mounted in after the image boots,
but everything else is stateless. For the images, we built them with make OSI, or however you can
set it's part of the system D project, it's quite new tool, we use it to build a Debbie and
base images, nothing particularly fancy there. The only interesting part is that we don't have a
classic root of us for network booting, but we just pack the entire distribution into the
inner drd, because, well, it's running from rm anyways, you might as well, and you skip the
entire part where you now also need to configure your inner drd to bring up the network to
pull the root of us, just a nice simplification. Speaking of network boot, this is roughly
the flow we have there. So, traditionally what you would have, you would have a pixie rom, which
can chain loads into i pixie, which fetches a config, which fetches a kernel, an inner drd,
which boots, which brings up the network again, which fetches the root of us, which then boots.
A lot of steps here already mentioned we're cutting out the root of us, another thing we're
cutting out is the whole pixie thing. We use uefea, hdp booting, tends to be quite well supported,
by firmware nowadays. So, basically the firmware of the box just brings up hdp, does hdp request,
fetches a uefea binary boot center. You can pack linux and an inner drd enter uefea binary nowadays.
One more thing we do here is config for the host. So, when the system boots, it sends a request
to this net boot service that I mentioned here, this looks to netbox, fetches configuration
for the server, packs it into an inner drd, appens it to the image, packs it into a uefea
merge, strings that out, server boots it, goes into linux, and then it's pid1 is a quick utility
that rend us out templates in the image with the config values to come from netbox,
and then it hands over to system d interest brings up the system normally. Sounds quite hecky,
but works very well in practice, and we don't need to worry about being persistent across
reboots because everything is a few mirror line ways. So, with then then we come to networking,
which is arguably the core of a cloud operation, because there's a lot of features, and so only
you want to build the tip end of the network. So, there's a lot of thought you want to put into getting
that right, scalability flexibility are really important here. So, what we do in practice is we
have a strict separation between underlay network, whose only job is to connect these servers,
and overlays, which then connect the workloads on top. Underlay, as I said, only jobs that
this server can talk to each other, you can do it very basic, just connect it to a switch, and
PhDP, whatever, done. If you have a big operation, you want to want something
more powerful, something that we found there, which works really well, is routing to the host,
every host has a beach pdman on it, has a loopback IP, peers with the switches, and houns
a stat, that way you can build whatever network to apology you want, you can do things like having
two network cards, uplink to separate switches without having to screw around, m-leg or whatever,
everything just works. Under switches, we like Sonic, which is a switch from where open source,
based on Linux, has some problems in practice, but you can make it work, and when you make it work,
it's really nice, because then you can do whatever you want, which is fully programmable,
which is very powerful. For the overlay, we use Vxline EVPN, simply again, standard, everybody supports
it, hardware supports it, no reason not to. The underlay logic is isolated into its own network
namespace, so this is roughly what the host looks like, the actual network cards are here,
in the network namespace, there's a beach pspeaker, and then we have the Vxline
interfaces and bridges here, there we have a one Kubernetes overlay, which then is hooked up into the
main host, and it namespace sort of where Kubernetes and SolarLifts, so our cluster network
and everything just goes through this overlay, then we have a public overlay more on that next slide,
which is used to connect workloads to the internet, and then of course we have tenant overlays,
which is just L2 plus DHCP for our customer, to connect customer instances together,
kind of like a private network. For this public overlay, it's sort of a arguably at the end,
but very basic and just built in-house to do what we need to do, again, to have flexibility there
for offering various features, the logic here is this endpoints in this public overlay,
things like instances, load balances, services like object storage, and of course the edge routers,
which can act as to the internet, they also sit in this overlay, they announce switchability in
for about themselves and about the routes or prefix exit that they're advertised, again,
this goes into Kubernetes resources, one resource endpoint, so this looks like this for example,
so then the endpoints says, this is my IP and this overlay, this is my MAC address and
these routes, please send them to me. If you curious about this port sting, next slide.
This is a public IPv6 space, again, just hooked up to the internet, but you can also route IPv4
traffic via an IPv6 next top, of course, it's just simplified things to not need to do as
like addressing there, because IPv4 addresses are quite scarce nowadays, and then we just have a network
demon on each host, which watches all of these resources, and then sets up the networking state
within the endpoints accordingly, so we pre-populate the neighbor table based on the
back information that's there to avoid neighbor discovery flooding in the VX-lan,
we populate the route table, potentially with multi-hop based on what each node advertises,
and that ends up working quite well there. Of course, we want to hook it up to the internet
eventually, so then we also have one component that takes all of this and translates it into
BGP announcements, which then can go out to some hardware router that's connected to the internet.
The reason why we don't just use BGP for everything in here is, again, flexibility,
things like announcing specific ports, you can't really do there. We have status layer followed
balancer in their custom control plane, data plane is basic, Linux IPv6 right now, we're looking
at implementing something with the second chance routing that could up proposed in their
gilded load balancer and cloud flannel also uses their very good blockposts about that.
For traffic offloading to get more throughput, I mean, all of these servers have
looked for, and a little bit next now doing that with Linux kernel networking is kind of challenging.
One thing we did there in the past, and now looking at again is using VPP for this, so you bind
the fiscal network cards to VPP modifier network demand to push all this data into there,
and then you use a tap-and-faces to connect traffic back to containers applications,
like Kubernetes, and VMs that are directly hooked up with the host user. Of course,
nowadays, the very modern thing is to use a diffuse and offload everything there, but not quite
there yet. Networking for instances, again, each instance gets a public IPv4, subnet, and a
delegated network, instance traffic to the internet, just regular firewall, net fill the contract
works well. We run this in the instance to avoid, again, having a central point of
failure, because this needs to keep track of connection states. Public IPv4 address is optional,
because the scarce alternatively just use a net. There's some trade-offs there, because you need it to be
needed to be decentralized, again, you need to be scalable, but if you get several user ports,
you need to be able to make back to the customer. So what we do there is we slice up public IPs,
based on port number, we assign one port range to each instance, and then then we just
make the instance net only for that. Audium A network, again, is important for GPU clusters.
The classic number, there is around a 3.6-terror bit per node. Obviously, it needs to be hardware
accelerated. Luckily, you usually have one network or a card per GPU, so you don't need to slice these up.
For infinity band, you partition, the infinity band is one Audium A network, you partition with
something called peak keys, it's kind of like a VLAN, sorry to skip over some things here for
time reasons. Rocky goes over Ethernet, it's kind of similar, you can do basic VLAN isolation
on the network card, but it's better to do it on the switch. Again, Sonic is an ICM. For storage,
you want something that's fast and reliable, unfortunately, in all ways quite feasible. So in practice,
we split it. Fast storage is just local and we meet on the server. Again, if you're
in the middle, because we don't want to guarantee anything about state, so this is used for cashing,
and then we have network storage, which needs to be robust. The local storage, it's easy to
bottleneck on that. If you have four gen 4 NVMe drives, you want to slice it up for customers,
you want to encrypt it, you see the bottleneck. So we use SPDK and practice there. It's again,
like the PP uses space off load, but for storage, and we get 98% of the raw throughput,
even though we do crypto and LVM and rate and pass you to VM. So that's quite nice.
For the network storage, we use SAF. There's everything, file system exposed. We have
a virtual IOFS, block storage to respdk, object storage, raw escape bay, and the Kubernetes cluster
on it to re-rock. And then virtualization at the end, obvious option. There's the first glance
is queued, we looked at that. But the thing is, last time we checked at least, it's designed to integrate
legacy applications into your Kubernetes cluster, we want the exact opposite. We wanted to be isolated,
and with all of the special things, we do hacking that into Kubernetes. At some point, it's easier
to just say, you know what, we just run our own Qemu and a container. That's the job.
You want to, so this is what typical APA GPU server looks like. You need to
mind the enumerant apology there, so you want to select GPUs that are close together,
then you want to select CPU cores on the same node. Then you want to pin the VCUs, both
performance and for security, again, side-channel attacks nowadays, everybody's aware of that.
And you want to pass this topology info to guess, so it can schedule accordingly.
Then, for the last point, GPU specific things about virtualization, you need to be mindful
of the PCA topology here. There's a lot of GPU to GPU and GPU to network hard traffic,
which is PCA, IPA to peer. If the goal goes through your CPU root complex, you're going to
bottleneck hard because there's not enough lanes there. Workloads need to know the topology,
so that it doesn't accidentally send traffic across the entire node because it thinks it's
local. You can configure this into the instance manually, but it's easier to just model it through
Qemu by having like virtualized PCA switches in the topology. Classic pitfall there. If you just
pass through these devices, you have the IOMM U and the data path, and that forces all the traffic
to your CPU. So instead of your 2.6 terabytes, you're suddenly getting 180 gigabytes, not very nice.
The way to avoid that, there's a feature called address translation services that you're
able on the network hard, and then you're able directly to traffic and access controls service
on the PCA switch. You just Google that, find the commands for that, and then you get your
full traffic. In that case, you need to trust the network hard firmware to not do malicious
DMI attacks on your host. Well, this is just something you have to do in this case to be honest.
Finally, the last point you want your GPU into the connect, something like enabling,
or a link for more general solution. If you do GPUs one at a time, it's really easy.
You don't have GPU in the connect, just pass through the GPU you're done. If you pass all of
the GPUs through it's also easy, you just pass everything through and let the guest manage it.
If you're mixing it, it's difficult because now you need to configure which GPUs are allowed to
talk to which very, very vendor specific right now. Unfortunately, I hope that this is
the area that will get more support and standardization in the future.
With that to the quick tour, unfortunately, you couldn't go into a lot of details on
many things just because again, big pictures, small times that. But if you have any questions,
any follow-ups, if you've read talk to any of us at the event, send us an email. If you want to
learn more about any of this, let us know and we can do another talk next year.
Thank you very much everyone.