Okay, welcome everybody to this talk, today we are going to talk about error recovery
with microS and systemy blast butter.
So first of all, today we are going to talk about errors, bugs, warnings and I know that
almost all of us will going to encounter bugs and this kind of things every day.
So we are going to see how we can recover from errors in our distribution.
So I am Daniel Spinala, a quickly working assuse as a research engineer and we are going
to talk first of all about health checker, then about bootloader specification and at
the end we are going to talk about automatic boot assessment.
So to talk about health checker, we need to first talk about officers and microS, how many of
you know offensive microS, okay, that's a lot of people, nice.
So offensive microS is the offensive ratio, the offensive version of a susanim center price
micro.
It's an imutable system, it uses butter effect napshots and cross-sectional update and since it's
imutable, all applications are installed in containers, could be docked or put in containers
or flat-pack containers and it also tries to be reliable as much reliable as possible
by leveraging snapshots.
So this is an important thing to talk about because you are going to use snapshots for
to recovery from errors, every time we change the system, maybe we update all the packages,
we install or remove something, we need to create a new snapshot and how do we boot a
different snapshot, how do we select a snapshot, we can use this sample loop, counter-flock,
we can say which snapshots we are going to use, we just change the version number, this
case is the first one but we can choose whenever this snapshot we want and the key software
the key component that handles all these snapshots update, the packages update, it's
a transaction update, it also handles butter effect napshots, today we also have the transaction
update container here, it's glass foster, so let's talk about our checker, the checker
is the key component, it comes installed by default in microS, first of all what it does
is check the status of the boot, which is in each check different components, for every component
there is a different plug-ins, this means also that all the administrators can extend
what they mean to check the boot, maybe they have some different components, they have
some different service, they want to check, with a checker they can do that, and most
importantly it has a phase safe mechanism, so when there is an error, as checker tries to
reboot and tries to bring the system to our work is snapshot, let's talk more about
the phase safe work, so in this case for the entries, if we have a new boot entry let's
say we just update the system, which I boot in that, if there is any issue with that, any
failure, any failure, then checker automatically reboots up to three times, and if the phase
then it goes to another snapshot, trying to find the snapshot the works, of course,
at the time we created a snapshot, the new snapshot is at the fault one, and in this case
if it is not working we want to update that, and bring the default entry to our work in
snapshot, the last thing is okay, we have as a snapshot is working, we have a version of the
system that is working, but somehow we reboot, and there is no working anymore, in the case
what happens is that as checker will reboot one time, the same snapshot, the same version
of the system, and if it is not working, we are going to open an emergency shell.
Now let's talk about bootloader specification, which is usually abbreviating in VLS, our
many of you know the bootloader specification, okay, that's good, let's see what it is,
it's something very, very new, but it's getting very quick adoption, so this is a Linux
system, this is a basic system, just if you have a system, if you have a partition,
the own partition, the system partition, very basic, and then we might have an located space,
this is the Birmingham, no encryption, nothing, let's say that we want to add another system,
we have to want, we want multi-boot, how do we do that? So we need to consider a lot of things,
first of all the boot FBI partition, the type partition needs to be unique on each disk,
so we need to be aware of that, so the other system maybe will install a different bootloader,
okay, but which bootloader will be the default one, the first system, the second system,
also the bootloader we will choose needs to be aware and to be able to read the boot partition
of the other system, otherwise it cannot create the entry for both systems, like we will
obviously we will only have the entries for the last system, the restore and that's not something we want,
we get complicated very fast if we add encryption and system on different partitions and
different disks, so for this reason the bootloader specification has been created and it allows
multi-boot system to share the entries, our message does that, we have configuration files
that are standardized, we have two types of different configuration files and we will put them inside
the slash loaders, slash entries, path relative to the API partition, so inside this folder
we will have all entries so all the system is told that we want the bootloader to find this also
is very good because we have standardized configuration, we can, we can have all the applications,
user specification, firmware, bootloaders, read the same files, we can have a common interface that we can
use. Let's see at the first type of configuration that is the most common, we are not going to
share the second one, this is a simple key value file, text file, you can see in the name we have
the distribution, the kernel version and also in this case this is of course a microS boot entry,
we also have this snapshot which is the last one you can see before the suffix, so what do we
have here, we have the pre-title, we have the version of the kernel and this snapshot,
then we have the kernel arguments and the paths to the kernel and the intramethas, in this case
these paths are relative to the API partition, so very straightforward, very simple,
and the automatic boot assessment relies on the bootless specification, it's also part of the
bootless specification and maintain the existing D, with the automatic boot assessment we can divert
the system to a previous version, we can revert the kernel baby to a previous version,
it allows it as components, it allows us to revert the system and as I said, it relies on
bootloader specification, so let's see the first part of the automatic boot assessment that we need,
first of all we need boot counting, what is boot counting, we are the number of boot attempts
that we want to try for every new entry, how do we add that, we add as more suffix to the final
name of every entry, so we add a new entry, a new version, okay we see we want to try three times,
since we have not booted yet, we can say that in this case we have three times left,
that entry is called in the terminate, we try booting, the booting is successful, in that case
the boot counter is removed and we can say that this entry is considered good, we know that it works,
what happens if it fails, we try booting again and again and every time we decrease the counter,
so we have three, then two, then we go to zero, in that case the entry is considered bad,
and here we can see an example, the same entry, we boot counting enabled, in this case we have two
tries left and one already done, as you can see it's very straightforward inside the file name,
we have the plus prefix and afterwards we have the tries left and optionally the amount of tries we have
done, this is very simple because this also needs to be handled by the bootloader, so we want to be
as simple as possible and renaming the file is very, very straightforward, so what does it mean
installing new entries? The counter about the boot tries that we want to do is actually
can be found inside the slash it is e, slash kerna slash trice and usually when we want to
start new kernel, a new version, sometimes we call kernel isle and kernel isle is aware of the boot
counter, so it goes to read the file and creates a new entry with the boot counting enabled,
same thing about the STB utility, STB utility is utility that can be found on tumble with the
microS and most times, it manipulates the entries, sometimes maybe STB utility recraced the
interface in the case, it also needs to reset the boot counter and STB utility is aware of the boot
counting, so these are the two first components that creates the entries and are aware of the boot
counting, right after that we need the support of the bootloader, so we the inside the bootloader
we will have all the entries, there will be read inside the slash loader, slash shantys folder
that we have seen before and upside of the user sorting depending on the name of the distribution,
the version, we will have one other type of sorting which is the sorting depending on the
counter or the boot counter, so any entries that are the zero tries left would be boot at the bottom,
we know that the entry is not working, so the bootloader is automatically sorted as last,
and another thing that is very important is every time the bootloader boots an entry,
so we choose an entry or maybe the default one will be booted, then the bootloader needs to
read the file name, check the boot counter and rename the file, it needs to adjust the counter,
so we have three types left, okay we just did one, now we have two, we did one, so the second
count is increased, regarding the support system the boot as a computer support for bootloader
specification and automatic boot, automatic boot assessment, you have two aware, as part of
just support, we are going to see later, we are going to talk more later about that, but the good
thing is that any bootloader can support automatic boot assessment regardless, it just needs to
be aware of the boot counter and is to rename the file names, so let's talk a little bit more about
bootloader specification, and there were two different sets of patches, one made by Fedora,
and one made by OpenSUSA, did that the support, I forgot to the 14, just came out recently,
I think he was last week, and now we have an upstream support for the LS, but it's still in complete,
boot counting of 14 is not supported upstream and is still downstream supported by OpenSUSA,
but the patches will be upstream very soon, so we saw the boot counting part, now let's talk
about the boot itself, so we are testing the system is booting, if we want to have a service
the check if the system has booted successfully, like check in this case, but there are also
other services provided by the fault of the system, there is for example a service that says
we don't want any service to fail, if any service fail, then in the case the boot was failure,
we just need to add to our service unit required by boot complete the target,
boot complete the target is a special target unit made by system D, and vice versa,
if you want to have some service that acts upon a successful boot, like system D,
we can just have the requires boot complete the target to our service, so system D,
less boot, if you actually go read the service, just add these two lines needs to be executed
after the boot is successful, and it also needs to require boot complete the target,
so let's talk a little bit about system D black boot, this is a more utility provided by system D,
first of all, the thing that it does is it is executed by the fault when the boot is successful,
and it just removes the boot counting in the case, we can also use it to get the status of the
current entry, we just call it, and we are going to see if the entry is in the terminate,
it is bad, this is good, this is used by F checker actually, and last thing we can also change
the status of a boot entry, we can say, okay this entry, okay we consider it as bad,
we can just call the system D black boot and it will do the renaming for us,
why are we adding L checker on top of automatic boot assessment, why not using it directly?
There are a few issues with automatic boot assessment, first of all, it is very simple by the
sign, because it is made by implemented by boot loaders, the boot loaders are already complex
as they are, and we want to have something as simple as possible, that means that we don't have any
automatic reboot on failure, the default engines are not updated, if we set an entry as the fault,
and the entry is bad, by system D black boot status, we will still boot it again,
the boot loaders consider any default entry, the Goddess of the boot counting of the boot status,
they consider it as default, and they will boot it anyway, this is not something that we want,
so we need something on top, to add logic, and to provide a better free safe mechanism.
So let's go and see again how the F checker face safe work, now that we are seeing boot loader
and these tools provided by system D. So first of all, to check the boot entry status,
we use system D black boot, we check, we update and remove the default entry if we need that,
and so when the boot counting is enabled for an entry, then we know that the entry is new,
we leave the sorting of the boot loader to choose the next entry, and check the reboots on failure.
Now, if the entry was working before, we know that it was working, it still opens some memories and she shall,
so, there was the presentation, thank you for your attention, so we can go to the questions.
So the question is if the patches, the optimization of BLS support, is actually the patches from
open suzer fedora, I think it's a clear implementation made by the grab developers,
but yeah, so please.
So, against tempering, so the question is how safe is BLS against tempering, so in the case,
we using TPM, we can act to work tampering, so yes, so this actually is implemented by
a process of microS, by using a studio tool, we try to work against tampering, yeah.
So the question is if this is related to system D, what system D does,
yes, we use system D tools to work with the TPM, thank you, please.
So the question is if check it was just with UFI or with it works without, so
the automatic boot assessment, cool work without FI, there is one other thing which we didn't
talk about in this presentation, with this boot load interface, boot load interface contains
some metadata and data to be shared with the system D and the boot loader, and that works
only with FI, automatic boot assessment relies a little bit on the boot load interface so
currently it doesn't work without FI, but this support to be added in the future. Thank you.
Any other questions, please?
Yes, this one, yeah.
If you follow anything, any other issue, and there's something we can have about changing this,
I'm not very likely to have changing this, because what does default mean? Sure, it override
the assessments, definitely, but I think we can add another concept of something that was a little
bit less powerful than just basically says, yeah, I think these kind of items by default,
unless they are changing the assessments of the account, so I will not come like, tell the end,
tell the rules, you just say, but take to the bed, let's fix that.
So the question is regarding the default entry, because the system D upstream, they are going to change that.
Yes, I'm really in the discussions, they still a little bit on the fence to change the
this mechanism, hopefully they do, which are we can remove this from the check it as well.
And then the other thing, no automatic boot is available, you can do this here, if you like,
we just use this open to fill with value, right? Like, if the boot success stuff kind of
be reached, then some action should be taken obviously, but the processing downstream, this is like
the choice for you, like, what's right, please, okay, but so there is a thing like, for example,
you can send a time out until this point is reached, and that's typically what you want,
or you can have an action that you assign to it, like, when this thing never is reached and
thing like this, so there's actually already this, it's maybe we can get better documentation
to tell you exactly how this is about each amendment, any kind of additional infrastructure to
just make sure to make use of the stuff that's already then configured to the policies that you want.
So, uh, Assistantie actually provides the way to, uh, reboot automatically, for example,
by setting up time out after a couple of times that the boot doesn't, uh, didn't finish.
So, yes, we are well on that, but for for us, we wanted to, uh, we already had a checker before
so we're in new, what was our successful boot for us, and we still use the same mechanism,
but the boot if there is an issue. So, this is a other logic on top, yeah, we could also do that.
I mean, I'm very curious about, like, what kind of logic we don't cover yet, um,
if it's general, general useful logic, then I would like to know about it.
Okay, we can, uh, we can talk later about this. Thank you.
Yes, please.
So, the discussion was about adding, uh, more documentation, uh,
I'll confirm, okay, we find these things. We don't, we think this is indeed because they are
already implemented. Thank you. Any more questions? Yeah, please.
You mentioned that there are two set of patches submitted to grow up from Fedora, and those who
they are open to this. Yeah. Um, do you have different approaches in that regard, and do you talk
to each other or maybe come up with a solution together? So, uh, the question is regarding the
two different set of patches from Fedora and open sucess. So, I, unfortunately, I'm not aware
of, um, regarding the original work. Um, I don't know if they don't do the gap developers,
and they found a common ground. Unfortunately, I don't know to, I don't know it went,
uh, but currently the implementation of Club is, uh, pretty close. Um, they still a little bit
missing, but we're going to assume that now that we have a baseline in, uh, in common. Thank you.
Okay, uh, let's see cases where you need to add some sort of hardware, what's
up, if you have issues with common, where the common doesn't give you the question, starting
in it. So, if the question is regarding having a watchdog, if the kernel is starting, so in the
case, the boot model is still going to decrease the boot counting. And, uh, we have, um, uh, we have a
reboot on panic on kernel panic. So, if the kernel is panicking, it reboots, we, the count is going
to be decreased. So, system reboot, at one point, we'll choose another snapshot. Thank you.
Please. This is inspired by the same, because the, uh, the, not the network must do.
Uh, sorry, can you please, uh, please? Uh, please, uh, please, uh, please, uh, please, uh, please, uh,
uh, uh, unfortunately I, I, uh, I don't know. We got to think just the same thing. Okay. Yeah,
yes. I don't know if you have the recovery should be pretty similar, so, yes, but I, I don't know
actually, I do and so on, so that. Thank you.
Okay, so no more questions.
Thank you everybody.
Thank you.