Speed up live updates of your kernel and still be able to keep a, for instance, a
virtual machine's life while you're doing the live update of the kernel on the host.
There may be a, in memory file system that you want to keep its contents, for instance,
if you still be a fast, with lots of data and you don't want to lose that data, when
you do the update of your kernel, you want to keep PCI devices configuration, there is no
reason to really eat PCI config space because you didn't really boot, you didn't change.
There are via file attached to virtual machines that can continue doing the amazed
during the live update of the kernel, but when kernel reboots, their state becomes cleared
and they restart from the very beginning, so VMs get disconnected from their IO's.
And there may be different use cases as well.
For example, the first driving use case for resisting memory across KXXX was a P-ROM suggested
by a virtuoso that they wanted to create a version of TempaFest that would be persisted
during the KXXXXX and it'll survive reboots so that when you use commute to checkpoint
and restore your applications, you could store the checkpoints that the TempaFest and the
Blaz just persisted over reboots and then restoring the application contents would be
much faster than doing a cold reboot and then going to the disk and so on.
So, over the years, starting from 2017, there were a couple of proposals
how to preserve memory contents, and there was a pyramid.
There was PSRAM, there were different proposals of memory allocators that would be
aware of state persistence, and would allow you to somehow pass metadata from one kernel
to another, and then the memory that would be allocated from the allocators will survive
KXIC, and the latest proposal for memory persistence of a KXIC was KXIC hand over mechanism,
which will expand in this presentation, and the guest mammothes.
So, I'll plot the slides, and these links are live to the discussions on the Linux kernel
mailing list.
So what is KXIC hand over OKHO?
It's framework for serialization and serialization of the state that KXIC users can register
their callbacks into the KXIC and when the KXIC calls these callbacks, the users decide
what data they want to serialize, and on the boot of the next kernel, they decide what data
they want to serialize, and KXIC provides the ability to save arbitrary properties, and also
provide the ability to save arbitrary ranges, that a guaranteed in some way to survive
KXIC, in a sense, this is similar to Zen Live update approach, the implemented with Zen
breadcrumbs, and KXIC consists essentially of three major parts, first is the KXIC
WFDT, it's a data structure based on its essentially flattened device tree that already exists
in the kernel, and KXIC uses the FDT library, the FDT contains the properties, users
wants to use this one to serialize and serialize, and one of the properties that more
predefined is the memory ranges, that KXIC user can say, okay, I have this memory range
please put it downwards, so I will be able to read it after KXIC finishes, another part is integration
of KXIC boot, and the boot is set up with data structures, and currently it's implemented
only for X86 and ARM, on X86 KXIC, KXIC is appended to set up data structure with the pointers
to its physical memory locations, where it stores the device tree, and where it stores the memory
required for the next boot, and for ARM64, KXIC, KXIC is added to the device tree created
by ARMKXIC as a chosen node, and the third component is what we call scratch memory, these
are memory regions that KXIC only case early on the boot of the first kernel, and then this
regions declared a CMA with CMA mobility, so that they can be used by the system to
use the space memory locations for movable locations, and then these regions are passed to the
next kernels, and the only memory that is the next kernel will be using for its bootstrapping,
and now KXIC device tree, as I said, it's written device tree, it is the same format
as a device tree that VMware uses to describe the hardware, but in case of KXIC, it uses
to describe properties and memory ranges that we want to preserve during KXIC, and the device tree
is created at some point of KXIC process, and every driver that wants to persist its data
has to provide its own nodes in the TFT, and to fill the node data in the TFT, I'm trying to understand
what Alex wrote here, so as FDT is widely used, it all provides tools for verification and
some standardization, it gives a ability for driver writers to understand how much previous version
of what was serialized is compatible with the next version was what is serialized, and for instance,
if we are doing KXIC of some more advanced kernel version, if we go back to one of the previous
kernel version, there is a way to understand that properties saved in the existing data are
compatible or not compatible with what the driver can handle, and so this is an example of
how FDT may look, this example saves FDT buffers and events, and then the FDT can be reconstructed
from the saved data, so there is events in BitMusk that says what events were enabled, there are
not memory ranges that FDT buffers used on the CPU 0 that has, in this example it has four
ranges and pretty small, in reality it becomes much larger but you get the gist, another important
thing what happens with the memory during transitions of K joint KXIC and so on, so when the first
kernel boots very early during booted reserves some contiguous memory areas, which are kind of blue here,
then as the system continues to work the orange yellow, I don't know, on color blind a bit,
the orange yellow areas that's what we want to resist during the KXIC and these areas should
be designated as such by the users, when KXIC actually happens, the only memory that the second kernel
will use is the green area screech memory that was seen in my, for the lifetime of the first kernel
now it becomes like the only usable memory for early memory allocations pretty much until
a page allocator is ready and this way we can be sure that we can reserve everything that we want to
persist and once the system knows that the persistent ranges are secure and safe the screech areas
again became CMA and they can be again reused by the allocations of movable data and did
make a continue for years hopefully, there is a bunch of this is a fast control, it's not certain
stone yet but we think these are important, so first of all there are scratch fees and scratch
land that define physical addresses and lands of the scratch areas, they can be used by KXIC users
based tooling to understand where the kernel image and the other parts of the KXIC work KXIC data
should be stored, it's still not implemented right, then there is an active control that tells
the user if KXIC is active or not and let's use it to enable disabled KXIC and there are controls
for a device tree configuration which DTMAX says how much memory can the device tree use
if you cannot grow FDT infinitely you have to specify its maximal size when you create it first
and then the CIS kernel KXIC DT is the representation of the device tree, it's binary form,
it appears after KXIC is activated and it contains the device tree created up till this point
so what we have seen here if you use DTC minus I whatever and look at KXIC DT you will see
something like this and there is a CIS framework KXIC DT, this is the device tree that was passed
from the first kernel to the next kernel and from here we can see what was in the persisted
at least what we tried to persist and how we tried to persist it so how KXIC works more or less
I think I'm repeating myself in the way but still during very early boot time KXIC reserves
in scratch areas then at some point later in time it declares this area CMA because
you have to reserve contiguous large chunk of contiguous memory before you have the free
list and page allocator and you can't say what page block types I at that point have to be separated
in different stages and then the system leaves and at some point users decide that users decide
that they want to do kernel life updates so they enable KXIC by doing H1 into CXIC
each of a KXIC active a KXIC all calls to its users callbacks to perform their serialization
of data they want to persist and this is the KXIC data structures are painted to KXIC image along
kernel image in it or D and whatever else KXIC stores so this pretty much happens and KXIC
lot time is the there are two additions to KXIC image KXIC creates one is the physical addresses
of scratch area scratch areas and another one is the FDT block the physical address of FDT
block so that when KXIC moves pages around it will know where to place K where to place FDT
and how to pass the scratch area addresses to the next level when the KXIC reboot happens
there is some logic in KXIC that knows how to move pages from place to place it involves a lot
of copying unfortunately but it's something we're going to improve I hope and then when the second
kernel starts a setup arch on X86 on 64 passes the KXIC data that was there for in their setup
data structures it creates a very early in booting, it recreates the scratch areas and
they make sure that early memory allocations work only within that scratch areas then KXIC
passes the device tree and from that point it's possible for KXIC user to deserleys their state
depending on when the these driver initializing it might be in it called time it might be
module any time but the state is there we make sure it's preserved and it stays in memory and
nobody will touch it now there was a recent discussion about this thing on the Linux and
mammalian list how do we want all this thing to happen there are a lot of opinions obviously
and we still have a lot of open questions and we need to address some of them related to how we
managed scratch areas for instance it is possible to predict in some way how much scratch
memory the next kernel will need we know how much memory the first kernel used for its boot
time allocation so we can kind of do some math and hope that the next kernel is not too different
from the first kernel and calculate some scratch sizing based on current kernel use and I don't
know give it 150% or maybe a bit more and the reason opinion that it should be set in the kernel
common line and that the administrator knows better yet to be seen if administrator really knows
better another interesting thing is what happens if administrator thinks they know better
and they want to resize key show after boot of one of the next kernel and say okay the fourth
kernel will boot with different scratch sizes we still don't know how to address this to be honest
because if it's done late in the system lifetime there is no guarantee we'll be able to
to allocate contiguous physically contiguous memory chunks it will be easy if the administrator
would want to reduce but increase will be difficult and another open question is what do we do
if there is no enough memory scratch area when boot time allocation is happen do we panic
or do we just disable key show and continue with the normal reboot and another thing that was
discussed on the mailing list is what exact data format do we want some people say that
everything is too strict and rigid and doesn't allow for parallelization of drivers serializing
their data and from the other hand it gives some standardization and so there was a big proposal
about having some intermediate data structures for resisting device properties and driver properties
and memory ranges then in some point converting into FDT and somebody suggested that we just
reinvent some another format and we'll use that format so we'll continue to talk about this I
presume until we enjoy the merge upstream and another interesting point is what states do we want
to have right now what we have is activate case a cloud and they can exactly boot essentially
so the reason is are to make it possible to activate case or after case a cloud
and it's a bit complicated because of how case it works and it's not impossible but
still it would be quite intrusive and another thing that was raised that
in the complex device drivers like for example more than need cards will require much more states
in the middle like prepare prepare again and then save state restore state and the basic model
of activate load it doesn't work with them and it's also something we are going to look into
in the near future so that's all I had I think I made it in time
hard
right so you're using CMA because it's continuous right you need the scratch space to be
essentially I reserve man block because it's continuous and then I tell page allocator
okay it's a may you can use it for movable but it's CMA guarantee to be available for this
what do you if there's some driver that needs to persist state or it's in your set of
so now we can't persist and move we can't persist movable ranges so if it's CMA it's
going to be movable and if some driver did the page alloc of something it would be unmovable
so we can't persist and movable but we can't guarantee persistence of movable
in another quick question what's your I mean this is all needed to be exact very fast right
to keep the PMs and drive devices alive so what's your roughly your time budgets for doing the
character time budget is always as low as you can right the few interesting tricks we've done here
is the two-phase approach where you activate it in only KXC which allows you to move a lot of
the complexity like the civilization which can be expensive into a part where you can still
continue running your system so it doesn't actually buy you into the KXC downtime you only
probably really inflicts a pain in end-down times is the visualization which is reasonably fast
at least since you have a well-stocked data formula
yes there
time, so both of them did a couple of them.
Who was it there?
Hi.
One of the slides mentioned that the data are
mutable after serialization.
So this immutability, it's
and sure generally, or by each user of the API,
must ensure that the data are mutable after it's serialized.
The user must know that they serialized
and the data they serialized cannot change.
It's, Kicho doesn't provide any guarantees about it.
It's up to the drivers.
All right.