Hello everybody, and welcome to my talk about single-image secure boots with Bevels.
My name is Ahmed Fortun, I work for Pingotronics, we do embedded Linux developments and consulting.
I think a lot of people here in the room might guess, a lot of our new projects, or most
of them actually are employing some scheme of secure boots.
This is mostly, I think, motivated by upcoming EU regulations, which should make it more difficult
for digital products to be compromised.
And in the embedded space that we are working in, this usually is, this is a, it's much
of a call, a single-image interpreter.
Yeah, and in the embedded space we are in, this usually takes the form of a verified boot chain.
So I have two short slides about that.
So usually you have some hardware root of trust in the form of if you use this for example.
In these if you use this, which you can only burn once, so you only can change a 0 to a 1,
but never a 1 to a 0.
You can encode the hash of a public key.
And using that hash of a public key, you can verify the boot loader.
The boot loader itself will have, in its verified contents, a public key, which is
used to verify the kernel image.
If you don't want to use the same key, that you are also using to verify the boot loader itself.
Then the OS image will contain, for example, the hash of a D&V root file system, and
that will be used to verify all blocks within it, and now you have a chain of trust stretching
from applications running in user space to the hardware with the keys that was burned in the
factory.
Of course, if you can mount and attack on the boot loader, all this securing is for
not because you can compromise everything from the boot loader onwards.
And because of that, you need to invest time to secure the boot loader if you are going to
do a secure boot chain.
And securing the boot loader is mostly about restricting stuff.
So you restrict what can be done, what it can do, and you restrict what can be done with
it.
And this has far-reaching consequences.
So development is certainly more awkward because you need to sign your images, always
testing is more complex, because you wouldn't give access to a boot loader shelf, for example.
And the kind of stuff we test, for example, if you have a watchdog, you want to decrease
an attempt, and if you get the watchdog after an update, you want to roll back to another
update to the old system, for example.
This is very complicated to test when you don't have a boot loader shelf, because you can
directly ask the boot loader and inspect the boot state, and because the development and
testing is more complex, also maintenance will be more work, and overall complexity of
the system is higher, the risk that you don't finish it on time, or that it costs too
much as high as well, and of course the manufacturing, the factory setup, and the infield service
when a technician goes to repair something, they will also need to be authenticated and
authorized, so that's also more effort.
So if you are going to secure a boot product, you will have to take this issue somehow.
The thing is developer solve issues, so if the problem is, it is very awkward to develop
for the hardware, they will find a way around this awkwardness, and for example, disable
secure boot for the development, or use a different build, or okay, I had the build this
boot loader that's unlocked, and I share it with all my colleagues for example, and of course
that's not a workable solution, because in the end it means the state in the field is not
adequately tested, and you don't want to debug two weeks before you're doing a null series,
if you want to debug all my the integrities setup is taking so long, and now that I have
a read only with FS, something's doing or to work, because we usually have an overlay
FS for the development, so project risk is greatly increased if you do not account for
how people are going developers or testers are going to use the product.
And I found a nice way to visualize this when we think about device lifecycle, so the
same hardware will have different states that it will be in, it will be a convenient development
state, it can be in the factory for provisioning, it can be deployed in the field, it
can be returned, and each one of them has a different profile or different threat modeling,
what needs to be able to be done, so if you get a field return, of course you want to
interact with the hardware, but maybe you should wipe the keys first, so you can access
patient information for example, and you need to look at all of this, and the way I see
it done in many projects is that you encode the lifecycle state into an image, so you have
this image is used for the factory, this image is used in production, this image is used
when we have a field return, it might be the same image for factory and field return for
example, but this is a disaster waiting to happen if that image leaks, here is an example,
so Nintendo for the Wii and the Wii U setup, they had special SD cards with a special factory
bootloader, and then after they were done with them, they gave them to their contractor,
they were crushed, broken, thrown away, resourceful people were able to get a hold of them,
with some, I like the way they phrased it, with a little straightening and convincing I was able
to recover the data, of course that's not the bootloader, that's actually deployed everywhere,
and it had much less eyes on it, much less usage, and they found exploits in it that allowed
them to roots the devices, so just using the image to handle a lifecycle state is not adequate,
because any leak would be disastrous, and thus the image should verify that the status correct,
so for example you have an e-fuse, and you burn that e-fuse in the factory, and after that the
image will check the e-fuse, oh it's burnt, I will not do anything anymore, so the factory
image is only valid as long as this e-fuse is not burnt, or devices that are returning from
the fields, they have field return e-fuse, and when that field return e-fuse is burnt,
wipe all data, unlock the device, but you can't go back to the normal, so you have
fuses that mark how you transition through states, and once you have that you can go the extra step
and have the same boot image handle multiple of these lifecycle states, because it's already
has to verify that a fuses burnt, then you just need to implement fallback behavior basically,
and one can argue that this increases complexity, but I would rather argue that it just shifts
complexity around, if you have a lot of multiple images, the complexity is in the way you do testing
and the way you maintain the complexity is like in the whole workflow and infrastructure around
it's that you need to juggle these images around, but if you put this complexity in runtime by having
by knowing I am in this lifecycle state, I will do that, you make it, you move it to a place where
it's more easily auditable and you can run tests against it and so on, and yeah that's enough for
the theory, and I have some more practical example, but yeah in summary what I want to say is
if you are the image is the better I think, and a single image is achievable, and I think it brings
real benefits, so an example for a few space state transitions and opti, so opti, which is
a trusted execution environment maintained by Leonardo, it's like what you would normally use
on many embedded systems, and often it's provisions with device specific certificates in the factory,
so you want the device to call back into the mother cloud, and then it has a device specific
certificates, and these certificates are stored into replay protected the memory, and you can only
do the storage, after you have written a key into the EMMC, which the EMMC will save in some
secrets place, and then sock can, I think, authenticate as a correct word, authenticate,
so what the EMMC was an HMAX that stuff is authorized to be stored there, so yeah, and the thing
is because you can only, because you need this sock specific key, it can only happen after
secure boot is enabled, so in your factory step, you must, after having enabled secure boot,
verified boot, write this RPMV key, and now if you had an image that writes this RPMV key,
and it leaks, then people could modify the RPMV contents, if they reuse that image, opti has an option
config RPMV writes key, yeah, and if you enable it, it will ask the EMMC, do you have a key,
built in our key program, if it's not programmed, it will transfer a new key, and you can't have
opti every time asking the EMMC, are you programmed, because on attacker, could just disolder the EMMC,
add a fresh EMMC, the fresh EMMC will answer, yeah, of course, I am not programmed, and then you can just
snoop the clear text sending off the key on the bus, and yeah, and to cartelset, my colleague Marko
Phelch made some cool requests for opti, and the idea is that instead of having one image where we
always enable RPMV write key, and one image where we always disable it, we always enable RPMV
key, but before the key writes, we always check is a specific if you've burned or not, and after
the key writes, we blow the if use, and this way we have in our lifecycle graph a link that only
goes into one direction, so it was in unprovisioned EMMC state, now it's in provisioned EMMC state,
and you can just do a replay and get the keys this way, for that he added for one opti, one time
had support for I am exprocessed as to opti, there is an open request for that, and there was a
second pull request to do this fuse, checking, and writing an opti, there was a very fruitful
upstream discussion about, yeah, about how to do this better, so there will be a second pull request
coming, and some form of replay protection in addition, but it would still be gated behind
an if use, and I think we can use this model of gating stuff behind the if use at many more places,
and with that we eliminate the image, the different image that we usually would have.
Another thing I wanted to show is access control, and access control in the bootloader,
so that's not something that's usual it's on, and the status quo is that one time access control
in the bootloader is a mess, because everyone using the bootloader in a different way, and everyone
defines the old threat model, and what might be secure for one is not secure for someone else,
and if you are writing your own bot code, whether it be in u boot or a barbox, it will be
very complicated stuff where you push into the guts of the bootloader to selectively enable
or disable features, and that's the way you would handle different security states.
So for example, here's an example from bearbox, it doesn't look very nice, so it checks
a locked-in variables that you for example got from the one time pad, and if it's set,
you can call boot and force signed images, this is a function that in forces, that fit image
is must be signed, and you can't get back from that, and alternatively if that's not the case,
it will get the console, and enable it and switch it to the open pin-mook state, and yeah,
this is a lot of specialized code, no one can keep all of that off mind, and it's easy to get
something wrong with that, and no one is upstreaming this code, so this is why this is more
infrastructure, it's both specific and no one bothers to upstream it, which makes every project
reinvents the wheel. For bearbox, I introduced last year, bearbox security policies,
and the idea is that you have these allowed markers that can check specific security decisions,
and at the privileged operation itself, so insights that get sharp function, which you would use
to get a character, there is a check, is allowed console input, and if it's not allowed, it will return
an error, and this way, you can have very future proof access control, because you are disabling
the feature directly where it is, for example, another example is the environment, you don't want
to load a mutable environment, arbitrary mutable environment in your infield case, and instead
of, but you need, but you want to have the environment for the development case, so instead of
trying to disable all ways, you might load the environment, you go directly into the environment code,
and you add an is allowed, that prevents loading the environment, and then you know, for sure,
if there is some new way to load the environment added, it will be still covered, because you
are directly at the privileged operation where you do this check, I have some slides,
how this security policies look like, so these are key config files, in the end, but they are not
used for compiler, but for runtime decisions, if you do security old config, you are asked
interactically, if you do security manual config, you've got a manual config, and there it asks
your question, should console input be opened, should share script be executed, should fast put
your m commands, be supported, and so on, and you can have multiple of these security policies,
so you can have one for lockdown, one for development, one for factory, one for field return,
as you want, you just define these are the security policies I have, you call these function,
and it will ask you one for by one, this security relevant options, and it's only the security
relevant ones, so it's not like K config, where you must really have the insight into the bootloader
to decide, okay, is something I want to enable disabled, so if you read for example fast boot,
we m commands, you might not know that this allows you to run arbitrarily commands,
so this is usually something you don't want, but in this security config view, it only asks you
actually actionable questions, then in the end you get a number of K config style files,
with this S config for security config in front, and these are normalized by K config,
and these are all compiled into the bootloader, and then you can have in the your bit time K config,
you can set a default policy, for example lockdown, so the device as soon as it starts, it will be
in the lockdown policy, which is what will be active, and then board code can decide to move
into a different state, depending on a fuse, for example, yeah, here it can choose an active
policy, and then the bootloader can do stuff with that, and the nice thing is you can have this
clean separation, so you have the policy sides that you are defining on a per board basis,
with this K config, like mechanism, and then you have the mechanism side, which actually applies
the security policy, and that's in common code that different projects can collaborate on that,
and not everyone needs to figure out how do I reliably disable the environment, or how do I
reliably ensure that only white listed variables are passed through, for example, and yeah,
then you can, yeah, and these decisions are enforced, so here is an example for this policy selection,
so chooses the NVM API to read fuse, this factory NVM cell, and if it fails to read that fuse,
it does security policy select lockdown, I only added this for, yeah, for descriptive purposes,
it should be already in lockdown mode, and if for example, a factory done fuse is not burnt,
it would select the devil mode, and then in the devil mode, you could say I have console input
enabled, I have USB, I have fast-boot, I have enabled, and so on, and in the factory you would
always burn this factory done fuse at the end, and then developers would get the device without
that factory done fuse burnt, yeah, last thing I wanted to talk about is runtime unlocking,
so going back to what I said at first, you want to give developers an easy escape hatch,
but you don't want to make it too easy, otherwise they do not test normal boot flow,
and this is something we had used for in a few projects now, and I like it pretty much,
which is we have two fuses, one fuse is the factory done fuse, and one is okay, we haven't decided
on a good name yet, one is partially done, one is very very done, and you would burn both fuses
for all devices that go into the field, but for developers you would only burn one fuses,
one fuse of them, and when you have only this one fuse burnt, it would say factory fuse intact,
and then it will just pull for five seconds, that you press the key D, it will tell you,
press D to enter devil mode, and then when you press D, it will enter devil mode, and then you have
a bootloader shell, and the testing system can interact with the bootloader, and do stuff,
and if you don't press it, it will just continue a normal secure boot with no console, for example,
or only read only console, and it will only check EMMC, it will not allow not work boot,
and so on and so on, and the nice thing is about that is if you just let the device start up,
the default behavior is that it will start up like it doesn't feel just with any layer of five seconds,
instead of having devices that are permanently open, and permanently locked, and not testing
the permanently locked state as often. Yeah, for unlocking, that's for development device,
if you have production device that are fully, fully fused, you can have also unlock tokens,
so I have seen this implemented in board calls, in your boot and in verbox before,
but there is no generic way to do it. For the box a few years ago, I added JSON work token support,
you can pass JSON in the bootloader now, but it's a verified first, and then it's passed,
and the total possible was RSA signatures, my colleague Jonas will, in an hour or so,
talk about the TLV format, which is another way to store the tokens also on the device,
and ECDSA signatures are also now supported, and an important point about this unlock tokens
is that they must be bounded to a device. In verbox there is now an API for that, that gives you a
unique ID, that's implemented for multiple socks, and this is very important because you don't want
an unlock token leaking or reproduced to unlock other devices. So the way it works is you have
a public key, for example, you see there is a public key, in the bootloader, statically compiled in,
and then it's used to verify the token, and the token has a claim that says, I am only valid on
this one device, and if it's not then it's not unlocked, and this is another way that you can
allow developers to unlock hardware, why the system is computer computing, but once they remove
a USB stick or once they wipes the data, it's revert to the original state. The same data
can also be in an Android verified boot tier, for example. Yeah, and that brings me to my last
slides, so for the future, there is work on generic system data TA, so if you don't have a full
blow in TPM, but you have opti, there are still some things that would be really nice to have,
just with an opti TA for exactly this purpose, which is having a key value store and having
world-back protection and what right one is variables, and you could store signed blocks here,
and then you can verify, for example, that this device is allowed to be in the developer mode,
and then you can send, again, it should be in a normal, infield mode, and because it starts in
the replay protected memory area, you can revert to the whole state. So this work is underway for that,
and yeah, we are looking forward to publish this at Pingotronics. I'm not sure if we will try
to get this into opti upstream, but yeah, it's, we'll publish that and provide user space tools,
and also bootloader integration, and I hope it would be a useful building blocks for this kind of
things. Yeah, I had a talk in August, which went more into detail about
barbox for secure booting system. We have fixed a number of memory leaks now, especially the
hash shell was very difficult not to crack, but it doesn't leak anymore, so I hopefully we can
actually run the fast test in CI with OS, as fast in the future. I'm looking forward to that,
and the big problems that we have in the computer tackle that is, we want to pass along the
security policy we have to Linux, for example, in the kernel command line. So if you select the
devil or the lockdown security policy in the bootloader, that should also affect the way you
start, for example, an SSH team in the user space, and we would need some way to communicate
this information. In a secure manner, for example, the kernel command line, we are not sure yet
about how to do that, but set on the agenda. Yeah, and that's it. So it's a call for the
bubble of security documentation, if you want to read more, this is a few links, and yeah, I'm
ready for questions. Thank you.
We have done for one question. Hi, thanks for the talk. I just want one question. The thing is,
you seem to look really completely the device, once it's in production. The problem is, most
really corner cases will happen at customer, and you will have to probably debug that. In this
case, is there a way, did you find a way to address this issue, to be able to take a device that's
in production, and you need to somehow find a way to authenticate with the device, to be able to
to debug it. Yeah, so that's what I meant here with runtime unlock, it's this one, but this one,
you can have signed tokens per device, and yeah, you would keep the private key somewhere safe,
and when you have a field return, you would get it out, sign a token with it, and then you can
use it to unlock the device, but you need to have beforehand included the key to do it for doing that.
Because I remember you said that you would use two bits to be able to say, now it's in production,
but it would be worth to have a bit that says, now this device was in production,
now we put it back in testing, so we lock some pretty cool stuff. Yeah, yeah, yeah. So the
thing is, what code has complete control about, about when to select the different policies,
that just a function call away, so you can check a fuse, but you can also check a token,
so you can mix both, and that's what you would do. Some processors also have built in field
return capability, like I'm X processors, so you can generate a special case, field return
bookloader, only valid for a single talk, use that to unlock the system, it will wipe
the hardware unique here or make it inaccessible, so it wipes all data that's the right
all data that's encrypted with it, and then you can do that debugging. All right, so if you have
hardware specific measures, you said, if you don't, you can re-implement it with Jason
WebTalk, and for example, in their box. All right, thank you. Thank you. Thank you really much.