Next up, we've got Tino's going to be talking about imutable all the way down.
They're going to go away.
Hey, Fox.
Thanks for joining the container, there are room for them.
So I have a few live demos, and those demos are downloading a lot of stuff.
I tried, and the previous talk back there, and it worked like 50 years per cent.
Well, but you know, if one of the demos gets delayed, I can just go to the next one of the demos are automated.
So you'll be able to download them from a repo.
Also, if any one of you in the audience is like leaching stuff over Wi-Fi, please stop.
It's going to break my demos.
Hi, I'm Tino, I'm a maintainer of flat-cut container Linux.
I'll give up presents, I'm a messer-down, or you can drop me an email.
I usually talk about operating systems.
It kind of a regular in the container, therefore, actually container optimised operating systems that don't get in the way of your workloads.
But today, I'm going to climb one level up, and I'm going to talk about cluster plumbing.
And that is how we kind of envision shipping Kubernetes, when you're a Kubernetes operator, as an immutable image,
that includes basically all of the bits and pieces that you need, including configuration,
basically all of the static stuff.
This allows us to separate Kubernetes from the operating system, which has a number of benefits.
It's easier to use, you have separation of concerns and Fs, slides on that later.
But it also enables very easily straightforward in life and place updates.
Before we start, my last demo will be cluster API, and that holy shit, this is small.
That usually takes a little longer to initialize.
So I'm going to deploy my cluster right now, and that looks good.
So this is using the cluster API Azure provider, and that's because I worked there.
And I get to use it for free.
I'm not trying to get you to use Azure.
This is the request for me, so that's why I'm using an Azure deployment here.
All right, anyways, let's start.
So I said in your shipping, this is a mutable images.
And we're using a format there that's called extension images.
It really has like a full one specification.
You can go there, you can click there, basically read all of that.
There's at least one implementation that's using this specification.
And it basically allows you to wrap application binaries, application supplementary file, files,
and static configuration, and an immutable file system image that only consists of the bits that you want to ship.
This immutable image will be merged into your route operating system file system.
Usually at boot, but it can happen at any point in time that you choose.
There's a nice tie-in with an update mechanism called system pieces update.
That update mechanism also works on single files, and if that file happens to be a system extension,
then you'll basically get these in place updates.
So inside of an extension image, as mentioned, you just spread your binaries.
The way you want to find them in your route file system after merging.
If you want to start services or use configuration files,
that don't need much changing, you include it in a Sussex as well.
Maybe some repascripts to get stuff started or bootstrapped.
And you can include system units there.
So services are being started at the point in time where stuff is being merged.
Just to reiterate on how this looks on your node.
So imagine your ship, Kubernetes, and an image, then it would basically contain this directory tree,
a little shortened here, with the Kubernetes binaries, some startup files.
And so this is going to be merged into whatever your route operating system is.
And it will end up basically looking like, it's kind of an overlay.
So it looks like as if both files, both binaries, always existed in your routeFS.
All right. Now you got your extension image, and how do you even get it on the node?
For those demos, I'm going to use FlatCar, which can be instructed via
Yammer configuration to do provisioning time downloads.
And in this example, in the first example, I'm going to show.
I downloaded from interesting GitHub releases page, a Kubernetes Sussex,
and put it on my node.
And this is what's going to look like.
This is the first demo, and you'll see how well the Wi-Fi works.
All right. So, who? Can you folks read that?
It's much smaller here, sorry.
So this is slightly different, and that's why I'm bringing it up again.
I will drop the extension file here, which is an entirely custom path.
The implementation I'm using on the node system, the Sussex, is not a very of that path.
And the reason I do that is because I want to make it a very manually.
So I can show you around before the whole thing is being merged.
So this is the point where things get exciting, because the download starts now.
All right. So it took like more than a minute when I was sitting there during the previous talk.
I hope this will be faster, but I can bridge the time by checking on my cluster API stuff.
Oh, here, stuff is being provisioned. So cluster API generates this stuff for you automatically.
So you get a fully operational cluster. It's still downloading.
Yeah. Oh, wow. 20 seconds. Okay. That's good.
I'm going to give that. All right. So as I said, this is now actually let me.
Do this. This has been downloaded to a custom path.
It's just a file there. Since this is a file system image, I can just go ahead and mount it.
Okay. So if you look at mount now, you see it's basically kind of like a file system structure.
If you do this recursively, you'll see that this is in the folder on file system.
Literally only contains the stuff I need to run Kubernetes.
And so two things might be interesting in particular.
I can find my short cuts here. The first is the ship unit files.
So, you know, the whole thing starts up as soon as the Sussex becomes operational.
And we ship metadata, extension metadata that tells Sussex what to do.
In this case, it just says that it will run on any.
It will run on any operating system, basically any distribution version.
It doesn't have any dependencies on the actual operating system image.
It's got, it's built for the X86 65 architecture.
And it tells the service manager to please reload all of the service configurations as soon as this is merged.
And we need to do that because we ship system the units and we want these units to activate.
So, if I now, if I now make system the aware of this by sum linking it into a directory,
that system the Sussex is to be off, then it will pop up in the, in the Sussex list.
And you see a number of Sussex here. And the reason for that is,
if I use the Sussex for a lot of stuff, not just for Kubernetes.
And the extensions right here, it's not active yet, it's basically staged, but not activated.
So, if I check for Kubernetes by no reason you are already on on.
Let's check for the Kubernetes, and then the binary is not being found.
And if I ask John a color, but it knows about Kubernetes, it doesn't know anything.
I will now merge the thing.
And that's usually what happens at boot, right?
And as soon as this is merged, multiple things happen.
First of all, we actually have Cubelet in the local file system.
Oh, I forgot to pin, no, the demo effect.
Okay, we have Kubernetes by no reason in the file system.
And John will cut it and knows about it.
And Cubelet, it's just crying into the log right now because it didn't come with the configuration.
So, it can't run any clusters, but it's there.
It's being executed to service starts.
So, learning from the previous waiting period, I'll start my next demo right away.
So, it's ready even when I need it.
So, this was basically a single-note demo that's image-based way of provisioning stuff.
And the only relation to Kubernetes is, it's being used, it's used to Kubernetes.
So, there's no cluster yet.
Before we go to that, we'll look into in-place updates.
And that works on the file system image as well.
So, as I mentioned, there's a complementary service called.
So, update can be configured to update any given image.
It is semantic versioning aware.
And all it needs to consume images from is an HDPS endpoint.
The kind of an index file in it, the sharsams in it, so it knows what files are available.
It knows what files are in the local file system.
It checks regularly that HDPS endpoint for new versions,
matching the semantic versioning pattern.
And it also is summing over.
So, it basically can update the summing that I used to make my personal Sussex
or known to systemly Sussex and will update that.
So, basically both put together a load, some kind of get-ups.
And that's what we're doing with the Sussex Bakery,
which in which you can regularly or on demand build or rebuild Sussex,
publish them as GitHub releases and then instruct your note configuration
to just go there, check for updates and download and stage updates.
As soon as new updates become available.
So, this is a single note, this is update demo and we have booted.
So, I learned from a previous mistakes.
This is slightly further automated in that.
In this configuration, we do a little more stuff.
We create the sum link.
So, this is basically merged already when the note boot.
And we add a configuration for Sussex update.
So, that Sussex update is aware of that file,
aware of the endpoint that consumes new versions from.
And basically, when it is all of these updates for us.
The one thing we don't do is we don't start Sussex update,
because that's my demo.
So, oops, yeah.
So, we have...
We have Kubernetes installed.
We have Q-blade, again, crying in its log,
because it came without the config.
This is the access merged.
And there.
And we have basically provisioned the Sussex and activated the Sussex.
And we know what to update it.
Here's our custom Sussex store.
And that has a single file in it.
And that was the file, the whole system was provisioned it.
And the sum link in it is the extensions,
where Sussex looks for its extensions.
Points to the 131.3 version.
Now, I'm going to start...
Sussex manually.
Usually, that will be done that went fast.
That's usually been done by a timer unit,
so you would set up a timer unit appropriately to start this.
I, of course, need to start system usage update,
and not system usage.
That's running on the time.
Sorry for that.
Okay, so we start system usage update.
That'll take a bit longer because it doesn't go into the background
when downloading new updates.
I made sure that it has an update to find,
because I want to demo something, right?
So, let's happening right now in the background.
It just goes there.
This is the same kind of waiting period that we have
when we started the instance.
It's usually faster, it's a lot faster in the cloud,
because the Kubernetes Sussex is below 100 megabytes.
So, this usually doesn't need like 30 seconds,
it's just because of the Wi-Fi here.
That takes a little longer and it'll install that Sussex.
Let me see what my, ah, nice,
so at least that one has been deployed.
We have, okay.
System usage update has finished.
We can even check.
We can even check the log output and we'll see.
It went there, it found a new version,
which is 134.
And it put it there right next to
where the other stuff resides.
There's a new file there now.
And if you check the, the sum link here,
it updated the sum link because we told it to do so.
Now, when I run,
cube let's test that version.
What version did I see?
So, of course, the old one,
because we, we staged the update,
but we didn't activate it yet.
There's a number of ways you can do this
because Kubernetes is a complex workload
to think the way I will use
is a node reboot.
And we're getting to that later.
That's nothing to be scared of.
You would usually want to act a vacuum agent out
and make sure like it's properly
according to before you do that.
And that's my last demo actually.
So.
That's the new version.
We updated in place.
Now, how does that?
First of all, let's wrap up on the demo on extension images.
So, this is basically a self-contained Kubernetes
that you can just grab and throw on
basically any operating system.
It'll come up.
You'll need to only do minor customizations
and specifications to make it work for your use case.
It is, it's relatively easy to generate.
If you want to make squashFS on a sub directory
and you get the image, it is easy to compose
into running systems or into new provisionings.
And it's easy to manage because all of those tools
are integrated in system D.
And it comes with a complementary update mechanism
and they're just relatively likely already supported.
Because it has been part of system D for a few years now.
Kubernetes deployments.
I need to rush because I only have 10 minutes left.
We are, we'll be provisioning like a full blown Kubernetes.
That is configured.
And that actually works as a cluster locally
and curing more images using this method.
So, basically, take a distro image.
In my case, that's a curing image locally
that I use with this snapshot option.
So, changes will not be written back to the image.
Create an instance from it.
Grab the Sussex from a separate source.
Compose it at provisioning time
and have the Kubernetes cluster ready.
Okay.
So, that's that.
This is again downloading.
Now, the benefit here is that you have separation of concerns.
You can have operating system folks care about data
to keep it up to date.
You can have Kubernetes folks all yourself depending
on the scale of your operation.
Focus on the Kubernetes bits entirely.
And, nonetheless, some kind of loose coupling between two components.
It even allows you switching distro's.
Basically, upgrading distro versions or switching to a whole new distro
without much concern about your Kubernetes.
Because you manage both independently and Kubernetes itself.
I mean, it's just a bunch of go binaries.
It's pretty independent from the distro
that you're deploying it to.
You can, you can in place update both separately.
Like, leave the operating system updates with the
operating system folks.
Have your own Kubernetes updates and the indicators you need.
And, don't worry about the interdependence of both parts, right?
And, for larger deployments, if you do it the traditional way,
is building images from doctors to images with Kubernetes included.
This approach will actually allow you to cut down on the version matrix a lot.
So, what do I mean, my luck?
Well, imagine you maintaining a Kubernetes image for your classes to deploy
with based on Ubuntu 22 Kubernetes 129 on AWS.
Maybe another one on an Azure, maybe another one on GCP.
But then you also want to ship Kubernetes 130.
So, you have six images and what's about, what about 31?
That's nine images, but what about you 124?
That's 18 images, but maybe you want to ship our nine.
And so, this is what I mean by, by kind of exploding version matrix, right?
You don't have that with the Sussex approach, because you combine these at provisioning time.
And you basically can take control of the parts that you want to take control of,
which is Kubernetes and leave this to maintenance to the lower level.
There's this inspection that exists, and you can leverage it with that.
Okay.
Nice. So, this put it.
We can go and fetch the, no, no coffee, not right now.
I'm too excited already.
So, we can go and fetch the, the cube config here.
And I think I'm going to shorten this just a little bit because,
I'm running out of time.
So, the full demo, which is published in the flatcard demos repository in the flatcard.
I can get up actually includes a full blown local Kubernetes cluster.
And I want to like take a shortcut here and only work with the control plane.
No, you see the other windows, those would have been the, the worker notes,
but we're going to go with the control plane in this example.
And we'll quickly provision a, see an eye because right now our notes are already,
and we'll only get those notes, this note ready.
Follow four.
This is too much demo effect for me.
Okay. I'm interesting.
Five minutes left.
The idea here is that you,
I can do this on the note actually.
So,
you have a pretty, you have a pretty happy cube that here.
And cube ADM basically trying to do its thing and it's waiting for the,
the C and I to be deployed.
On this version and the idea was here that I basically show you that from the Kubernetes side of things,
but it's kind of stuck from the, from the slow Wi-Fi and me running out of time to
update that to the newer version,
131.4, which I only can show you on the command line now.
Okay.
So, this basically pulls in new Kubernetes, in the instance, the publishing,
can happen via GitHub, which is what we do.
We have a GitHub action that basically rebuilds periodically,
and then publishes new versions, and then you can activate in place,
and that would have been the demo, sorry for that.
Operating at scale.
So, basically you don't want to log in, you don't want to reboot manually,
because that distributes, that interferes with workloads,
and it's also manual work, so you don't want to do that.
So, for the last scenario, and that fortunately concluded,
deployed a cluster API cluster on Azure.
So, that's here now.
Here's the version that I deployed.
You see that it's a slightly lower Kubernetes version,
and that's because the cluster API Azure automation uses that by default.
The thing I'm going to do now is...
I'm going to pre-empt us a little bit.
Yeah, there's a dash F missing.
And let it do its job so we can take a look later,
and then show you the slides in between.
So, cluster API is a really big thing.
You would only use it for large operations at scale.
It's not really an API.
Kind of a misnomer, it's a cluster management system.
And it uses a management cluster, which holds all the state for you,
so you can go and provision workload clusters from that.
One good example, which is being used is, if you host a hosted Kubernetes offer,
and sell that or provide that to other users or customers,
then cluster API is one kind of mechanism or tool they could use that.
Now, these clusters have an immense scale.
They are automatically managed from this management cluster.
And at this point, the latest, you don't want to log in on the worker clusters,
and reboot them.
What can we do that safely?
In cluster API, we ship cluster port with in-place updates prepared,
but disabled by default.
And if you prepare a new deploy and new cluster API workload cluster,
which is what I did in the other screen, you would first install the provider,
in my case, Azure, in the management cluster, generate a workload cluster configuration,
which is a huge YAML file, and then apply that config to your management cluster,
which will then spawn your workload cluster.
Now, after step two, before step three, I patched that configuration
to not just prepare and place updates, but enabled them.
So those nodes are just sitting there now and just waiting for reboot.
And they have a way of signaling that they just create a flag file in the fire system,
which is a known path for cluster reboot operators to pick this up.
And what I just did in the screen here, I deployed QD.
QD is a node refresh operator, where you can basically instruct to safely reboot nodes.
It's a CNCF project, just as cluster API and Fletcar are.
And it'll handle safely draining your node, rebooting your node, and uncorning it again,
so it's populated.
Yeah, it's very flexible, configurable.
So in a good bit of a time when the next target is starting now,
so you're going to have to take the rest outside for questions,
if anyone has any fear.
Makes sense.
That node updated.