Thank you.
So welcome to my session,
State of Checkpoint Restore and Kubernetes.
My name is Art van Rébo.
I'm really happy to be at first them again this year.
So I work at Red Hat since about 10 years now.
I'm involved in process migration checkpoint restore,
which is the basis for what I'm talking about today for at least 15 years now.
The project we are using today to do checkpoint restore processes
and containers is called CREU.
We'll talk a bit about that also.
There's a problem.
There's also a tool I'm talking about, a container engine,
which is called CRIO with the O at the end.
So don't get too confused.
I'm involved in CREU maybe since 2012.
Checkpoint restore user space CREU.
That's the tool we're using.
So why this name, you might ask, may ask.
The reason is there were,
checkpoint restoring things in processes is the thing
which has been done in operating systems for a long time.
And also people are looking at this in Linux for a long time.
It kind of started in high performance computing.
For more or less the same reasons as today,
you need four tolerance or you want to load balancing something.
So you need to find a way to migrate a process from one system to another somehow.
And there were some implementations,
maybe 15, 20 years ago, they were using some external code module,
which was never upstream.
Then there was some library interception intercepting system calls,
so you can know what the process is doing.
And all those implementations were not perfect because
they were not what you would call a transparent checkpoint restore
because you had to prepare your process.
You had to know at the point of time when you start the process,
at some time you want to check it.
For high performance computing, you would say,
okay, I do this with all my processes,
but it still limits you a bit.
And so maybe, I don't know,
15, 18 years ago there was an approach
to bring checkpoint restore into the Linux kernel,
and it didn't have a name,
but it would probably be called checkpoint restore in kernel space.
It was not accepted by the Linux community,
because it was a big and complex at a time.
And so a new approach was started,
checkpoint restore uses space,
kind of the contrast to the previous one.
And the thing which,
Cree you would try to do, or try to do from the beginning,
is use existing interfaces of the Linux kernel as much as possible.
And with this, it was possible to have some basic checkpoint restore
working maybe, I don't know,
12 or even longer when it the project started a year ago.
And over the years, Cree you introduced additional interfaces
into the Linux kernel.
And the interesting thing about those interfaces is
they are never really only for checkpointing restoring.
They are also used by other tools.
They basically they give you additional information about the process,
which you can read out while the process runs.
And then there's also usually some interface
to restore this value to set something in the process,
which makes it restoreable things like this.
So let's come to the Kubernetes topic of this talk.
So in 2015 actually there was a ticket opened
to get something into Kubernetes,
which you might call port or container migration.
If you read a ticket, it's still open.
It's a very, very high level discussion,
so it's not about technology.
It's more about the ideas how to have container migration
possible in Kubernetes.
And one of those things which is often difficult for a container world
is that container are supposed to be stateless.
So why would you want to need to migrate them?
Because it's stateless.
I'm just stopping it and starving it somewhere else.
The reality has shown that containers in fact are not stateless.
They are stateful.
And people are today using 10 years later container migration.
In production it's maybe nothing Kubernetes,
but there are a lot of different container run times today
and engines which support checkpoint,
restore all the level down.
You have run C, C run.
You have Docker, podman, container D,
LexD, Inclus.
They all in some kind of way work together with
crew to enable you to checkpoint and restore containers.
So that's why we started to look closer at how to do
checkpoint restore and maybe at some point container migration
in Kubernetes around, I think 2020 was our first ideas
when we talked about this to actually implement it.
So let's talk a bit about use cases.
Why would you want to checkpoint restore containers?
Why would you want to migrate them?
And most of the use cases are actually what people use today
to do exactly that.
And one really popular use case is the quick startup.
So you have and in most cases it seems to be a Java container.
It takes some time to start maybe a really long time.
So what you can do with checkpoint restore is,
so this is your container and this is your host running the container.
So you can take a copy of the container right into this.
Once it has everything loaded in initialized and then we write
a stateful copy of the container to disk with all the information.
And once I don't know, maybe a user or customer once to use the service,
we can quickly deploy additional containers from this pre-initialized
pre-initialized state.
And the container starts up much, much quicker.
And this is because preview just reads all the memory pages from
storage back into memory.
And there's no complicated library loading or data loading or whatever
necessary.
It's just we just loaded in one go basically.
And I have looked at container migration times,
especially the downtime.
How long will my application be down if I try to do a live migration?
And the most time you spend for the migration is always the network
writing the checkpoint to disk.
And you can, I don't know, you do it to RAM disk.
And it's even faster or restoring it from disk is never the bottleneck.
It's always the network.
And so what I want to say is reading and writing the checkpoint is a fast operation
usually.
So another use case which is kind of, it's a similar use case.
It's like the quick startup a bit.
It's a reviewed and safe state.
So again, you have a container which takes a long time initialized.
You don't want to have a long downtime.
So you, but you need to install some updates on your system.
So what you can do.
You install the updates.
You take a checkpoint of your container.
Write it to disk.
Then the container is there alone.
The system is gone.
It reboots into your new system.
A new kernel.
A green one.
And the container can be restarted from the checkpoint.
It's a bit like the quick startup use case.
But it's a bit, also a bit different.
Then container life migration is kind of the combination of those use cases.
So you have a container running on one system.
And maybe you have additional containers running there.
And resources are getting low or your machine has some troubles.
And you want to migrate your workload from one system to another.
And then the interesting discussion here I often have is life migration on
life migration.
And I think the definition of life migration is a really difficult one.
Because even if you think about life migrating VMs.
There will be some time when the VM is not running.
So I guess you could also call this container life migration.
Because we have to shut it down and move it somewhere else.
We can employ and create the same technologies as we do into EMU.
We have pre-copy migration, we have post-copy migration.
So we can minimize the downtime in pre-using the same techniques you have in virtual machines.
So from my diagram is you have the container running on one system.
You take a checkpoint, you write it to some storage, share storage, whatever.
And then you can start it multiple times on the destination system.
And if you stop a container after the checkpoint or if you keep it running,
it doesn't make a difference for the tool.
Because this is what Cree you does, Cree you basically,
and in Cree you speak, it infects the process with a parasite code.
And then the process is running under the control of Cree you.
And once we have written all the information from within the address space of the process,
Cree you is then cheering the process and removes the parasite again.
So the process will never know that it was checkpointed.
It will just stop for short time and this is basically the time it takes to write the memory to disk.
It's the most information written and depending on your process, this can be a lot of data.
Another use case which we initially didn't really talk about,
but people reached out to us are spot instances.
So spot instances from what I've been told are instances with a really cheap,
but they can go away really fast in 30 seconds or 2 minutes depending on your cloud providers.
And so what people do today is they use Cree you to wait for a signal that the spot instances spot instance might go down.
Then the evacuated node quickly using Cree you writing the checkpoint maybe to some S3 or whatever.
And then they can continue to work on some other spot instance or wherever they want,
without losing any of the work they've done so far.
And last year there was a lot of interest from people how to use Cree you in combination with GPUs.
And if I have had this talk last year then I would have said,
it worked with ANG GPUs but not with NVIDIA GPUs but we are lucky that NVIDIA released a tool to actually actually do the checkpointing of the GPU state into the main memory of the host.
And then Cree you can check point this state and so we can today with the help of an external tool called CUDA checkpoint.
We can also migrate workloads using NVIDIA GPUs and AMD also contributed like two or four,
maybe four years ago code into Cree you to do actually that.
I want to know more about GPU checkpoint there's tomorrow talk 9 a.m. in the HTC Deaf room and they talk about exactly this how they use Cree you in combination with Kubernetes and especially GPU checkpointing to optimize the utilization of the GPUs node without having them sit with nothing to do for a long time.
From the youth cases back to Kubernetes so we tried we tried to think how can we introduce this into Kubernetes.
And I mentioned before the container community often things as containers is stateless and we come along and say well we want to migrate them and then they say,
we don't know it's necessary. So we try to find a way how can we convince the Kubernetes community to get checkpointing into Kubernetes with the minimal change as possible.
So we want to be as least invasive as possible when we introduce this into Kubernetes and the idea the story we put around this and it's important to know this is just a story around it.
The actual functionality today in Kubernetes is not about forensic container checkpointing it's container checkpointing but we used this story to bring this feature into Kubernetes and one of the nice things about forensic container checkpointing was it's only about checkpointing the idea of of this idea is the idea of this idea is intent of this idea is that.
So there's a container running on your system and there is maybe you have you think I might be in a tech going on or not and what what what's done today is if you have an intrusion detection system something like this then maybe it detect something it kills the container and yourself.
And the idea is with forensic container checkpointing we now take a checkpoint of the container and this means we have all the memory pages from that point in time we have all and it's important we have all the memory pages means we have all passwords or random numbers or secrets whatever there is.
But we also have the complete stage so whatever the attacker might be doing in the container we can now reproduce it in a sandbox environment we can restore the container maybe we can just look at the memory pages.
We also offer to convert the checkpoint into core files so you can run it through gdb and see what was going on so there's a lot of possibilities you can now do with the checkpoint of your container and this is also what especially.
As far as I've been told banks are interested in they actually want to have a copy of the thing they stopped for later analysis to make sure if there wasn't a tech going on or not.
So this work started in 2020 we the code was was really simple to do this but we discussed this for at least two years how to how to do this correctly.
And in 2022 with Kubernetes 125 this was released as an alpha feature alpha feature in Kubernetes means it's default off but anyone who wants to use it can turn it on and we.
I have implemented the feature so so you need to implement it in Kubernetes and in the container engine currently I think there are mainly two container engines for Kubernetes it's container D and cryo and we implemented the feature in cryo 125 cryo has the same release numbers as Kubernetes I think so with the release of Kubernetes 125 we also had a container engine supporting this so with Kubernetes 125 it was possible to create checkpoints of your container.
And write them to disk and look at the image so and Kubernetes 130 in 2024 we moved the feature in Kubernetes from alpha to beta and beta means defaults to on so now we have a feature which is always on and Kubernetes and anybody can use it without any additional configuration so you can still turn it off if you think this is unsafe for my system.
I don't want it but currently it defaults on and with the release of or shortly after the turning it to better we also got the changes into container D so today it's possible to create a checkpoint with container D and cryo using the latest version of container D so you need at least two dot oh.
So we have check pointing in beta so let's talk a bit about container restore I already mentioned that the whole story about forensic container check pointing was.
just a story and we we put on on this to ensure it's easier to accept kind of so but we also were thinking about.
and restoring containers and we we left it out of the enhancement proposals for Kubernetes but we find a way to kind of trick Kubernetes to restore container and we were able to include this code also in cryo 125 so what we do is if you if Kubernetes stops a container it usually first does a.
container create and then does a container stop and we now hook in cryo into container create and container stop.
We see if the image path is a checkpoint image if there is a checkpoint image then we actually restore the container and do not create it from scratch.
And then we say to Kubernetes there's a container so Kubernetes restore the container without never knowing it's a restore it thinks I just created a container I'm greater that and so.
And this is also available since cryo 125 the pull request for container D is open for half a year now and they have been some feedback from the container D maintainers so it seems like they are interested in it as far as I would say it just takes some time to go over.
These changes in in container D to to get emerged but I'm positive we will also have the possibility to restore containers in container D is soon and you can always try out my pull request so let's do a quick demo here so now so this is.
System on my so this is the m on my home system couple of hundred kilometers away and I will start a really simple container here.
So it's it's I guess it's the most simple Yamazaya for a Kubernetes container it's a state full container of course because if I want to migrate it to the m locally it doesn't make me make much sense if it's stateless so let's start this container.
So it's starting you see great so I mentioned great install so this is what Kubernetes does for each container and that's where we hook into a restore so let's talk to my container.
So it's really state full and now I create a checkpoint the checkpoint is a tube let only API at this point I will talk a bit about this later more.
We have created a tool called checkpoint control to inspect the checkpoint image and it gives you some information here it's the image it's based on when it was checkpoint the engine the IP address the sizes and the process tree and it also gives you much more information if you want that.
So this is everything the tool can extract and this was written by some Google summer of code students were really thankful for the work so we see.
Kubernetes metadata annotations more annotations annotations annotations then we see the mount points from the annotation point of view then we see our process tree again so you see the complete command lines and the PID from within the PID name space you see all the environment variables which we've said.
And files another process in the container more variables you see also there's a TCP socket I'm talking to you see that one.
And then again mountains now from the point of view of the tool how we see it in the in the meta data and not in the Kubernetes meta data we all need all this information to later restore the mount points so what we do during restore we restore all the mount points mount points as they were before so that.
Tools can find all the files and it had before so now I will convert convert this to an OCI image using builder so I say.
So I'm copying my tar file to the container and then I add a annotation to so that cryo knows it's a checkpoint archive and then I do a commit and then I removed intermediate image and then I.
And then I push it to a registry.
There it is.
So the checkpoint image is really small it's like eight nine megabytes because we have only the differences only the memory in my test cases really small so let's go to my local VM here so let's see describe.
And now I have again a really simple gamma file and it's now.
Where is it 73.
You see I've done this a couple of times already.
And now I tell Kubernetes please run this container and I find out how to describe.
You see it said it pulled the image and it created the container in started a container and now if I talk to my container I should get a.
A three thank you and not a zero and there it is so you've seen.
Thank you. Thank you.
So this is possible since 2022 so.
Okay, this was the demo I have five minutes of perfect.
So next step so I opened Kubernetes enhancement proposal pull request to declare the feature as stable the requirements in our so we defined the requirements early in 2020 what other requirements that it requirements to container engines support it.
No serious bugs I think and and test yeah so okay test so everything is available right now so today if you do a pull request against the Kubernetes main repository container the will run checkpoint with a checkpoint tests using.
Our our code from container D and our test so it's all is there so there shouldn't be much.
Again, moving it to stable or GA so let's see if this works.
The next thing I mentioned is it's a cube let only API at this point so I opened Kubernetes enhancement proposal to move it to cube ctl because cube let only API currently means mainly you have to go to the node where the cube let runs and talk to the cube let directly if you have it in cube ctl then you can do it from wherever cube ctl is running.
If you have the right roles defined to your user account so this is also something I started.
The the process to get this into into Kubernetes this will probably take take a bit longer so.
The summary is the idea of container pot migration was already first mentioned in Kubernetes in 2015 we went off of the feature 2022 beta and 2024 we hope to be stable in 2025 and we are.
Working on cube ctl integration and with this I'm the end thank you for your time.
Thank you for your contribution I've been following the container D you have to.
Sorry I've been following the container D for request for a while thank you for the contribution.
I have two questions actually like first question because I've been doing this in production using everything you built that basically first question is what is the replacement for page server.
So you're getting more and more traction and spot instances without arm is very difficult to handle.
The second question is very difficult nowadays to separate the process itself from the state that's persisting in the machine.
So you move the process but you have a file open for example the famous storage do we have any traction to actually move files with the process itself in the same way.
Thank you very much.
So for the first question about page server we don't we don't yet work on any of the optimizations on the Kubernetes level we only have work with the optimizations for the migration on the lower levels.
For Kubernetes we want to make sure the feature works and then we look into optimizations for the files is we include all changed files in the checkpoint image so if I have a lock file or an input file and output file they are all part of the checkpoint image and they are restored on the destination system.
If there are inside of the container if you have a bind mount from the outside then the bind mount is restored and you have to make sure that the content you bind mount into is the same.
If you use a volume thing in Kubernetes then we do not handle it because the volumes are on a pot level and we do only container single container migration.
Pot migration is also something we loosely discuss but we focus on one thing right now.
Thank you I have one question.
Oh yes one. Sorry.
When do you think we're going to get this?
I don't hear your sorry.
I don't know what to do.
So the question is when can we expect that to make it to the public cloud type Kubernetes offering EKS?
So I can I can talk about for example I'm a redhead so I can talk about what happens with open shift.
So what happened with open shift is because it it defaults to on in the upstream repository it also defaults to on in open shift.
So if Google or Amazon are not actively disabling it an upstream feature then as soon as you have one thirty I guess.
But you must be able to access the Cupid API.
I guess that's the hard part.
All right thanks a lot. Any more questions you can probably find Adrian somewhere. Thank you.