Okay, let's start.
My name is Marig Washer, and I'm going to talk about booting blocks between U-Boot and Linux.
The talk actually has two parts.
I figured at the beginning I'll do some sort of a definition of terms, a couple of
terms.
I'll go through what is what, so we have that idea, and then once I'm through to this,
I hope it will also become clear what the problem is, and in the second half I'll tell you
how to solve it.
So first thing is bootloader stack, what is this?
So this is all the software that runs when you power on your system, and between that
point and when the operating system can not take over.
Traditionally, it was a very small piece of
software, or no software at all, because you just turned on your system, and maybe
they cannot boot it right away, but this was the 80s.
These days, embedded systems are much more complex.
When those started, the bootloader stack was usually single project, U-Boot, Corb, whatever,
this was the 2000s give or take.
The system was powered up, started booting from a reset vector, there was a parallel
flash at the reset vector, and the CPU was just executing instructions from that parallel
flash.
The software that initialized the RAM, loaded the kernel into the RAM, and the kernel
was running at that point.
Now, it got a little bit more complicated over time, because parallel, the flash has
way too many pins.
It's difficult to route and so on.
So the hardware vendor figured, okay, well, let's put a piece of software into the
sok itself, that's nowadays called bootroom.
This is a responsible for implementing complex protocols to communicate with storage,
and ultimately for loading small piece of software into internal memory, usually sok
internal SRAM, that small piece of software is the first bootloader, which will run
on your machine, and it will initialize the RAM, and it will load the next stage, whatever
is running in the SRAM, this could be either you would as PL, it could be TFABL2 or something else.
What is loading next is usually loaded into the RAM, that could be potentially T implementation
trusted execution environment, it could be you would itself, it could also be PSCI provider
and then you would.
So as you can see, at this point in the present, where we are kind of now, the bootloader
next stage, much more complex, it's a composite of multiple bootloader projects, some
of them close source, like the bootroom, some of them open source, like the TFABL, some
of their free software, like your boot, and it didn't stop here, it's actually now slowly
moving to the future where the bootloader's stage is no longer running on a single
core, it's running on multiple cores, when the system powers up, what actually comes
up is a safety core, which is small CPU core, Cortex Air, Cortex, that runs some sort
of a bootloader owner, it does have very initialization fully, it starts the RAM, it does
initialization of the core 3 and so on, then only then it starts providing some sort
of services to the application cores, it brings up the application cores and runs some sort
of a bootloader on the application cores, which then starts the kernel on the application
core.
But I don't want to talk about this, today I'll go one step back to the present and I'll
return to the situation where we are running bootloader on one single CPU core entirely.
And the next thing I would like to explain is exception levels, this is R&V8 specific,
but other R&CPUs also, hmm, what's up, yeah, other R&CPUs, the over cores, R&V7 and
stuff like that, they also have exception levels, it just looks slightly different, risk
5 also has similar concept, I'll talk about R&V8 because this is what I was working with
recently and it's easy to understand.
So the situation is this, R&V8 has four exception levels, or you can call them preveled
chains basically, which allows some sort of a layering of hardware access effectively.
The most privileged ring or exception level is EL-3, this is basically, so for that, that's
full unrestricted access to everything, all memory, all peripherals, so for in that exception level
can set up memory protection, they will also, it can set up IP, it can set up R&V8 basically
to prevent software in less privileged exception levels from accessing peripherals and memory,
that's basically the whole idea behind this.
The most privileged E-O-3, this is where secure monitor is running, usually some sort of a basic
firmware, which has the highest access, E-O-2 that's where nowadays bootloader is run,
Linux kernel, but with all runs, EL-1 is virtualized kernel and EL-0 is applications on
privilege to use it applications.
The one thing which happens here is that it is possible to switch between these exception
levels, they can switch from more privileged exception level to less privileged one, simply
by setting up CPU context and doing exception return, that's easy because, hey, you are more
privileged so you can switch to less privileged, that's trivial, but you can also go the other
way around and that requires for the less privileged software to trigger an exception in
the more privileged exception level.
For that, there are two instructions, as I'm seeing HVC, when that instruction is issued in
the less privileged software, it actually triggers, it switches the CPU state to a more
privileged state and it triggers exception handler in the more privileged ring.
The exception handler is supposed to do some sort of a permission checking in so on and then
maybe does some sort of an action, like something for which the system needs higher privileges.
By something, I mean, maybe enable clock, maybe bring up CPU core.
The thing is, these exceptions were less privileged software can request stuff from more privileged
software.
They serve, or, yeah, these exceptions were, yeah, less privileged software can request
services from more privileged software, serve as a way for the more privileged software to provide
services to the less privileged software with some sort of an access control.
But the issue with that is that when there is this kind of a contract between less and
more privileged software, there has to be some sort of an ABI effectively between those two software
components.
And that exists.
There are actually some standardized ABI's, already, PSCI and SCMI, but, unfortunately, vendor
BSP's, they extend them in all kinds of weird ways, that they introduce new shiny ABI's
which then suddenly in the next version, change ABI, so we have a problem, and the problem
is this effectively.
If your bootloader is providing some sort of a service, say, if you're operating system
kernel, or some sort of a previous stage block provides services to the next stage block,
it becomes an ABI.
Once you change one of these blocks and it provides different ABI, you have an ABI,
but you can everything falls apart, that happens.
It can happen in two ways, traditionally.
If the bootloader is providing ABI and you update the kernel and the kernel expects some
sort of a new ABI which they provide and no one, which they bootloader doesn't provide,
then the kernel will fail to boot most likely.
If you have an ABI, this is still recoverable because the kernel will fail to boot, okay,
bootloader will pick the other kernel, boot the Bcopy, and you can recover.
This is the better case.
If you have the right idea, then to say, okay, well, my kernel is a new ABI,
what's up with the bootloader as well, and suddenly your kernel doesn't boot,
then also your bootloader provides new ABI, the Bcopy doesn't boot, and your system is basically
unusable.
That's a game over.
So the issue I will talk about next is how to solve this ABI problem.
But before I get into it, there is still one thing which was mentioned to me while I was
preparing the work and slides by a colleague and he mentioned, okay, well, you should also say
that this isolation stuff and the privilege rings, they are not necessarily evil, that's through
the thing is, the higher privilege ring software can set up memory isolation, it can set up
ABI isolation to prevent the less privilege software from accessing these parts.
If the higher privilege software is some sort of a proprietary cluster blob and you do not know what
it's doing, then that's probably a bad thing because your less privilege, say, bootloader, then
cannot be used as a debug tool and analyze what the system is doing and you cannot effectively
debug the system, but if you are more privileged software is something which you set up yourself,
then you can set up the isolation and IP, access restrictions in such a way that what you can do
in the more privileged software is handled, for example, access faults and actually have that act
as a better debug tool.
So, yeah, memory isolation, not necessarily evil, it can be used for a very good stuff,
but back to the ABI problem, the solution to the ABI problem is actually super simple and that's
let's reorder the bootloader and blobs. The stack looks give or take like the one on top, right now,
so we power up the system, bootroom rounds, TFA, T, and potentially rounds in EL3, but TFA
most likely switch is the exception level to EL2, that's you but in EL2 and the Linux run in EL2.
What we will do is we will move the blobs, TFA and T after you boot and then you would
run in EL3, it will have access to everything, it can potentially even set up the memory restrictions,
memory protection if it desires to do so, we will start the TFA BL31 which is the PSI provider from
you boot and then we will start the kernel. The benefit of this is that if we start the TFA BL31 from
you boot, we can start both TFA BL31 and the kernel in lockstep, basically keep these two
blobs which have some sort of an ABI contract between them, started together and if something
fails we can take the other copy. Implementation is effectively simple, we have to do two steps,
first step, we need to make sure that you boot doesn't depend on any services provided by the
blobs, that means TFA BL31 or maybe T, means you would not have, you would must not depend
on either PSI or SCMI interfaces and in the second step we need to teach you boot to boot the
TFA BL31. The first step is easy because if you look at how the services are actually implemented
when you call some sort of a PSI or SCMI function or exception, ultimately what the blob does
internally is that it programs some sort of registers. So, you boot can also access these registers
if it runs in EL3 and in order to remove the dependency on PSI or SCMI all you have to do is
make you boot pop these same registers from some sort of a you boot driver model driver. So,
you would driver which matches that functionality and you are effectively done.
But there is a catch. Once you remove the PSI provider from before you boot,
when Linux kernel boots on our V8 it mandatorally depends on PSI being available. So,
if the PSI provider is not there you would, the Linux kernel will fail to boot. It will just
not do anything. Luckily, you boot can act as a PSI provider. But the PSI provider implementation
is effectively architecture or board specific. So, what can be done on the you boot side is
enable mando of some sort of a very basic very initial rudimentary PSI provider which
will only provide Linux kernel the ability to say okay here is a PSI. It will not be able to
start any CPU cores it will not be able to do any system reset or anything but it will be there.
So, in case the user has any problems they can still at least boot the kernel if they for example fail
to start the PSI provider and do some correct reaction. Otherwise, at some later point,
the PSI provider will be started. So, about starting the PSI provider, there are again two options.
One of them kind of the easy is to just load the TFABL 31 into memory, disable caches for that
you would ask commands and then jump into it. The only thing which you have to be careful about
in that case is to make sure that the exit point where the TFABL 31 will jump after it's done
doing it's job will be again the you would entry point because then the TFABL 31 runs
drops EL from 322 and restarts you boot which then runs in EL2 and then you have
you boot running in EL2 with PSI finally implies and you can stop Linux. The other option is to use
fit image which is I believe the more practical option for deployment and I'll talk about that.
So, fit image is a multi-component image based on device 3 just briefly it's container which
basically allows you to bundle together multiple blocks, kernels device 3s, firmware, FPGA bits
3s into a single file. It can contain all these images it also has configurations section which allows
you to tie together different images within the fit image and instruct you boot which of these
images to use for booting. You boot is capable of booting the fit images open and built for example
is capable of generating the fit images for you. Now, when you boot is booting a fit image
of course you have to instruct it which images from the fit image is supposed to pick usually
done by selecting a configuration which selects the kernels, selects the device 3s, selects
potentially another loadable. You boot revocates these images which are in the fit image into their
target memory addresses when that's done it runs loadable handler for all these images and then
at the very end it boots the kernels. That's basically how it works.
So, the loadable handler for the TFABL 31 unfortunately has to be board specific because
that loadable handler may have to do some additional configuration. The TFABL 31 forks from vendors
stand to have some special requirements may require some relocation tables, hand-off tables,
whatever it set up. So, these TFABL 31 loadable handler say they unfortunately will have to be
board specific. Before you boot jumps to the Linux kernel there is now an extension which is called
the jump prep handler right this is called just before you boot jumps into the Linux kernel
this can be used as a hook to finally jump into the TFABL 31. I'll show you the implementation
of this whole thing now because it's actually rather simple and how the TFABL 31 was the support
was added into your boot. It had three parts basically and it fits on three slides.
First part was extend the footage loader so that it would actually recognize the BL 31 image type
that's literally three entries in the right places in your boot that's how the patch looks like.
The next part is the loadable handler. This is board specific and it basically is executed
after the TFABL 31 loadable place somewhere in memory. Specifically for this platform all it does
is it checks whether the system is actually running in the BL 3 at all because you would might
be running in the BL 2 and at that point you don't want to be able to start the TFABL 31.
If it is running in the BL 3 then it marks the load address of the block and it stores it in some sort
of a global variable. The most interesting part actually happens at the very end of this stuff
which is the jump prep handler that one checks whether the TFABL 31 load it again if we are in the
BL 3 for that I think Quentin and finally after that after we are through all this stuff it
setups board specific hand-of-table for the TFABL 31 which instructs that one specific TFABL 31 to
actually return back into your boot into this function which is called unviates which to
EL 2 now why do we do that? This is actually an assembler function. We do that because we want the TFABL 31
to run the drop EL 2 and then return to your boot just before the assembler code in your boot
which sets up the system for starting Linux. So essentially what happens just before the
TFABL 31 load is that the TFABL 31 load is the same as the TFABL 31 load. It makes this
detour into the TFABL 31 then returns into your boot sets up the system for booting Linux and
then jumps into Linux. The integration of TFABL 31 both into a phyramage looks like this this is
phyramage source all you have to do is add another image into the phyramage source this time
of TFABL 31 this is the image type which was defined like 3 slides before and it has to be marked
as a loadable in the configuration section so that you would load it for you run the loadable
handler and then do the jump prep handler. And finally I actually have kind of a demo here in
the slides what happens when you build that phyramage using mkmage and start this kind of a phyramage
is this you will see literally almost no change when you boot such a phyramage. You will see that
you boot that start one more log but there will be no additional output you will just seal
in external booting and that's it. The reason for that is because the TFABL 31 doesn't print anything
and as you enable debug output but you will notice that things actually do work when the Linux
kernel is booting because at the very beginning it will report oh look I found some more advanced
PSI version 1.1 I think and if you look further you will notice that the Linux kernel actually
managed to bring up all the CPU cores not just CPU 0 and all the CPU cores bring up this is done
by calling into the PSI so hey cool yeah we could boot all 4 CPUs and it's great and to wrap it up
in some way I have this one last slide so what did we achieve basically we achieved the ability
to tie together Linux kernel device trees and the TFABL 31 which is the PSI provider
into one single phyramage basically into one file which we can boot all together this fits well
with AB updates because then you boot is capable of picking this or that kernel image based on for
example boot counter and if we can now pick this or that not just kernel image but kernel image
device 3 and the TFABL 31 PSI provider then we can safely update all these things together
as one single file or as one single roof file system image now if the phyramage fails to boot then
you would just boot the other phyramage which even if it contains older kernel version it also
contains older TFABL 31 with the old ABI and this way we do not break the
yeah this way we do not end up with having any incompatible ABI between the kernel and the PSI provider
and that's very much the point of this talk so what I want to show you and this is all
half thank you for your attention and do you want to get it back and then I'll have that one.
Hey Fuzzy this is really cool super exciting to see and at two release more questions one is the
Ubout SPL code already has support for configuring like the BL 31 parameters and jumping
at the TFA and launching up to even and one question so I'll actually go into that one yes
Ubout SPL can start TFABL 31 that's correct and then it can start Ubout yes so my question is if
like I did some similar kind of hacking on this and I just moved that code into the generic
like Bruton framework and it seemed to like I don't get why you have it in the board specific code
because the TFABL 31 blobs they have all kinds of vendor weirdness is that's why this is
board specific unfortunately okay the end of structures which you have to set up to start the
TFABL 31 they are not necessarily standardized I know there there was a discussion actually if you
look at these batches that yes some of it seems almost standardized but then you run the
CCL and start TFABL 31