All right. Welcome everyone. My name is Torsten, and this is Isha. Both of us will be presenting
the work ALIXOS has been doing with HSM signing, but I want to reiterate this was a team effort
from various other people involved in ALIXOS, like Mike over there in the corner, and also a
big shoutout to Tommy, who did really the bulk of the work that we're going to present. You
can't be here today, but thanks to Tommy. And let's get started with the lessons that we learned
from using HSM's to sign in. So, to start basic, like, why using HSM's all at all, and what
is in HSM? So, code signing is used in a lot of places, like even Microsoft uses it, Apple uses
it. You may know it from Google Play, apps need to be signed, like, easy mention it in the stock
before. So, it verifies the origin of the software, verifies it comes really from you, and
that from anybody else, because you hold the private key. And this is where the problem starts,
like you need to keep it secure. And if it's a file, it has the bad property that files can be
copied quite easily. And you don't know if it has to copy it. So, that's a problem, because
you may have lost your private key, and you don't even know it. Anybody who has a copy of your
private key can make valid signatures forever, and you have no way to stop them. So, even if
you just cover it, you know, oh, this guy is making up what you're going to do. The only option
you have is start again. Make a new version of your software under a new name, maybe,
with an Android use and you package ID, like, is it just mentioned people lose their private
keys even, right? So, they have to start over. They can, yeah, revoking the key in Android,
I think it's not a thing, and not aware of that. You can do it. So, this is when HSM comes in.
So, it stores the key in hardware, and there you can just copy it. It's in a little device,
or in a bigger device, on a special security cryptographic ship, and you're not supposed to
get it out anywhere possible. There's a caveat though, that if there is, like, a big, powerful,
attacker, like, nation-state-like, and they have physical access to HSM, they may be able to use
electron microscopes, and aced, and scrape off the layers of the key, and somehow extract
key material, but this is maybe not our threat model. So, one use case that is maybe interesting
from many people here is you have automatically automatic build signing set up in your CI,
like you take a release, CI builds it, and it even signs it automatically. So, with the private key
files, it's kind of tricky, because if somebody hacks your CI server game over, like they have
the private key in the scum, like if they hack your CI with an HSM, they may be able to do some
malicious signing, because they have access to the HSM, but you can stop them. You can plug the HSM
out, you can re-secure system set it up again, throw out the attackers, and they lose the
ability to sign. So, that's the key difference. Yeah, that's what I just said, and now
I'm handing this whole thing here over to Shur. Testing testing, okay, awesome. So, coming into
this, we now know we want to use an HSM, raw contents in this room, that an HSM is better than
having a private key file just following your own in the ether, and in order to make the decision
about what HSM to use, we have a lot of different options, we want to set a number of criteria for
us to decide this. So, the first is we want to be guided by HeluxOS values and principles.
The primary of months, those, you know, you see privacy, security, and accessibility, these are the
same values that drive a development of HeluxOS, the different features that we have.
Second to that is free and open source, any work that we did, we want to make sure that we could
share it with the developer community, but not only the developer community, we also wanted to share it
with our community who's using the phone, and so being able to have this level of transparency where
we share both the process but also the code is incredibly important to us. The second set of criteria
was security, and so it was incredibly important, of course, it's incredibly important to make
sure that both the organization, the team, and then the members on our project were able to have
ownership over this. So, who has which part, who is in charge of admin, administration, who is
in charge of signing, who is in charge of auditing. So, you wanted to ensure that we have distinct
role-based access and follow the principle of least privilege. This will help protect against any
authorized access and exposure, but of course we can't just say that we have to see that. So,
for that, we need logs. So, who does well operation? What are they signing? What are they doing?
And when are they doing it? The second set of criteria was more operational related to resilience
and scale. So, in this case, we first want to make sure that we are redundant if the team changes
something happens to someone God forbid, or someone spells water on the HSM, we are able to continue
signing, operating, and releasing scale-less releases. The second set here is we also don't have
infinite time to decide, create this provisioning, signing ceremony. So, we want something that was
tiered, that was expandable, and that will keep it improved over time. So, that we can eventually
if needed as needs capacity resources time allow, we can lift and shift it over. So, all these
in mind, we end up with the UV HSM, too. The biggest and most important reason that we ended up here
is availability, just getting the physical hardware to different folks in different places on our
team, our team is many different regions all across the world, was not the easiest and so being
able to secure that and secure that was really important. The second reason is that has open source
tooling that we were able to leverage and build on top of and finally affordability. If you search
up the prices of available HSM's right now, you can go from thousands to tens of thousands,
so at a unit price of like 650 bucks, this was a lot more affordable. Of course, because of that
also, there's a lot of room for improvement, and hopefully future plans. Now I'll pass it on.
So, we're seeing this as an intermediate solution, really. Like it's a cheap hardware
dongle, relatively cheap compared to the professional ones, and we are hoping, eventually we
can improve this, but this is what we ran with also to collect experiences. So, let's talk
already about some of the limitations of this little device that we're using, so we don't get
money for advertising that and we didn't really want to advertise that the device actually has
some issues, and one is related to key management. So, the storage capacity is quite small.
If you have just a couple of apps you want to sign, that is fine. But in our case, we are having
more than 20 different Android devices, and each of them should have its own key, and then there's
also different apps inside these devices, and there is different other partitions and keys
that all need to be different. So, we need roughly at the moment, if we trim it down as much as
possible, around 50 keys, and it's too much, doesn't fit. So, there is a solution for that,
and it's called key wrapping, and key wrapping means that you store your keys outside of the
HSM, but not like this, you encrypt them before, right? So, they get imported when you need them,
and decrypted only inside the HSM. So, they never leave the HSM in plain text, and there is a
wrap key that does all that work, and that one lives inside the HSM. Does it make sense?
So, yes. So, now we have this wrap key, but how do we back things up now? Because the
HSM, like I just said, may break, may get stolen, there's all sorts of stuff that things can go
wrong. So, we need to have backups, in case something like that happens, we can still continue to
sign, and we don't have to ask users, or reinstall our devices, because we need new keys.
So, the signed keys are easy, because they are encrypted. You can back them up, like you back
up anything else, like encrypted cloud backups or whatever. Because nobody has this wrap key,
they can't use the encrypted signed keys for anything. But what about the wrap key itself?
In the end, that is, again, some sort of key you need to keep private. If you keep it in the
file, you back to square one, and you have all the problems all over again. So, what we found as the best
solution is what's called Charmiers secret sharing, which is actually quite an old technique.
There was a paper published in 1979, and this is still used today, and this is quite good.
So, the idea here is that you have this wrap key, and you split it up in different parts,
which are called Charmiers. And if you ever need to get back to this wrap key,
like you would store it from your backup, then you need these shards come together, and it can be a
certain number of shards. Like if you have 10 people in your team, everybody has a shard,
you can say, oh, and one, at least eight of those people to come together, or set it. Let's
up for you to decide. So, that's quite powerful. But unfortunately, the UBHSM doesn't
support that natively. So, we had to do a complicated key provisioning ceremony.
All right. So, the complicated key provisioning ceremony, exciting. So, this is kind of where it goes
from theory, a lot of research, a lot of code written to actually conducting and provisioning
the keys, generating the keys. I'll make one quick note here, and one of, we have three main
ceremony goals. The first one is to generate the keys on two HSM's. The reason we chose to generate
keys in this scenario is that it gives us a blank slate to work with existing old keys that
maybe exists in backups. The other people had access to are not kind of going to be factored in,
and it gives us a reduced attack surface. So, if we are redesigning and we're doing this, we want to do this,
so the goals of our provisioning ceremony, one is to generate the keys on two HSM's. Again,
redundancy. Two, it's to be able to export the rat key, sharded, and the authentication keys for
different roles. And then three, it's to establish the audit, logging mechanism for all HSM operations,
starting with the provisioning ceremony. Now, the ceremony operation logistics was complicated,
took a lot of work, tried to cover it in a slide really quickly, but we started by trying to
ensure that we have a secure protected environment. And this, of course, starts physically, so
having somewhere that we could not disclose to our community, give updates on, on when we are doing
it, and where the place was, of course, physically secured. The second part is digitally, so where
we're actually running the scripts to provision the HSM's, where they're plugging them into,
that we decided to go with sales OS's, a femoral operating system that's audited to use,
and as well a known in the community. In terms of having the accessibility to our own team members,
making sure everyone can attend, we try to bring the majority of our team in person for this ceremony,
although we did have an option for remote participation. And then finally, the last main piece here
is having the hardware. So making sure that the computer running this on, we maybe had a little
purchase paranoia, but we went to a random store that we hadn't decided ahead of time.
Some of the computers were available, picked up the computer, same thing for any media we used,
the live boot USB, for example, or anything that we used to save backups on, and then making sure
that we both physically reset the UB to send to, but also via software. And of course,
no network connectivity throughout the entire ceremony. All of this was a learning experience for
our team, but something that really helped was working hand-in-hand with trail of bits and performing
a software audit at the end of this that we're looking forward to publishing soon. To make sure
that everything we're running here is following the principles that we want to follow.
So maybe to mention in relation to the previous talk that everything we did there is reproducible.
So the software is all public and you can reproduce the ISO image that we burnt to a DVD
exactly and validate its hash as well. So let me talk to you about PKCS11.
Which is quite the tongue breaker and that's why I guess somebody invented the name cryptocurrency.
So you might have also heard that, but everybody just says PKCS11.
Once you can remember that. And it's a standard C interface to have software communicate to your
HSM because this is in the end of the day what you want. You want your signing solution, you're signing
software, not take a key from the heart drive, but you want to have a key from the HSM to be used.
And let the HSM do the HSM signing. So all tools that are involved in signing Android builds
already supported. So that's great. So we just plug it in and we're done. But unfortunately that
wasn't the case. It was quite complicated actually because we can't just do it out of the box
like it turned out there is lots of problems along the way. And we have quite the complex
signing process. Google has huge scripts that handle this and you have to imagine you have
the build has lots of different partitions and then each of these partitions are signed.
But before you sign these partitions, you have lots of different apps and apexes which was also
not talked earlier that also have to get signed before and then you have to unpack the
partitions. You have to sign the stuff inside. You have to repack. You have to sign the partition
itself and you have to sign everything. And so it's quite complex and there's different tools
involved. The main three ones are APK signer, sign APK and even open SSL. And although you have to
interface properly with your HSM. So how did we hook it all up? I just give you some examples
because we don't have enough time to go through everything. But they had every tool at different
ways and every tool at different bugs. So one easy example to understand is that we found
APK signer for some reason wasn't closing the session. It opened with the HSM. And that caused
the build to have like some sort of a denial of service because suddenly the HSM tooling said,
I'm out of sessions. I can't do anything anymore. But our signing scripts were still running
trying to sign stuff. So we tried several things and in the end we patched the APK signer to
just close the session properly. Then there is this sign APK tool which to be honest we haven't
understood why Google is using two different tools for the same job. And we found that the sign
APK tool is also incredibly buggy and we couldn't even get it to work properly with PKCSC11. So
what we did instead, we also patched APK signer and reverse engineer some stuff of APK to allow
us to just use APK signer instead because that we already got working. And we needed to fix some
padding stuff. So the result is actually identical to what sign APK would be doing so the
builds would actually be booting on a real phone. So some some points about performance,
we also needed to tweak some stuff here because APK signer was being invoked a lot during the
signing process. And it loads when it gets invoked, it loads the key store which is fine if the key
store is on your file but if you have an HSM, it's kind of slow. It takes a couple of seconds to get
from all the keys since we fill up the entire HSM also with lots of keys. It's like taking much too long.
So we introduced some sort of a batch mode where it would keep the key store memory and would not
try to re-enusionalize the key store all the time. It's kind of a hack but there was
the best we could do for now. Another thing is that there is a different versioning
schemes for APK signatures. And the V1 scheme requires the HSM to basically go through all files
and do individual signatures for all the files in an APK and can be 100 results of files
in this way to slow. So the newer signature schemes allow us to just make one signature on the
entire APK invoked APK signer just once and let's wave faster. So now our last topic is auditing
which I shall talk about.
All right, so far we've talked about how we are having backups, how we can ensure that there
is no way to export the keys, but security is not just about making sure that we know what's happening,
it's about making sure that we can see it as well. There's proof of it. So so far, again, we
kept the key safe, secure, and private, but moving forward, how do we ensure that it is safe,
secure, and private, and stays that way that there's no unauthorized access operations,
signing operations that have occurred. And coming into this again, we did end up hitting
some limitations with the UVHSM2 keys. The first who was related to the space available. So this
time is not about the signing key space or the primary key space is about the audit log space.
So there is a circular buffer that's available that will be rewritten if you run out of space.
And so our solution here was one to enable a property on the HSM such that if you have
are not able to have any logs, then you are not able to do any operation. So did I complete the
Nile of service. The second part here was to have effectively an interceptor running that would
follow every signing operation and immediately write the log immediately, write the log
to the computer afterwards. The second issue here was the authenticity of the logs himself.
So the builds and everything that is signed is signed by the HSM itself. However, the logs are not
cryptographically signed. And so here, even though that we have the provisioning ceremonial logs
from day zero, we're not able to chain them unless we want to head and create a this get repo,
append only gets repo. And so the idea here is that you have automated log verification
using this built in CI. And so combining these two for the most part, recognizing here again,
there is it's not perfect, but you have mostly self-sufficient and self-expository logs.
I think that is the conclusion of our presentation. It's kind of a narrative of how we went through
figuring out defining and building this HSM signing solution. And recognizing that this is not
a perfect solution, but it needs a lot in the most of our criteria. And moving onwards, I'll
open it up to questions.
It's a good question to follow. Great overview. HSM, I'm impressed with what you're doing as well.
Why did you guys go with Metro key or go with that in a portability for more of a bit source things
like the RM or you go? Yeah, at the time we were able. Why did we go with UB? I'm going to summarize
the question. So why did you go with the UB HSM two, which was more proprietary, as opposed to the
Nitro key at the time you just weren't able to get our hands on the Nitro key?
Yeah, the Nitro key HSM is more is cheaper, but unfortunately we couldn't get it. And when I looked
into it, I also didn't see any audit capabilities. Like maybe you can tell me, does it have one?
Yeah, but it doesn't have built in auditing like the UB HSM has, I think. So that is also another
reason why we didn't go for that.
So specifically, other wrongs, wrongs also sign the wrong with the ability HSM.
So why are you running into the PQC as 11 bucks with sign it because now and how
will have other projects? So the question is, are there other wrongs are doing this as well
of PKCH 11? Why do we run into these issues and the other people not? The thing is, we actually
not aware of any other end red wrong that is using HSM's. Like do you know? And at least
publicly, like we even heard like some stories from OEMs that also just have plain key files
for signing. So I don't know, do you know any?
So you thought that your phone was using HSM, but to my knowledge, even they still are not using
HSM to sign the bill. So they also have files still. So to our knowledge, this is actually
first in the Android wrong ecosystem. So we kind of pioneered it in the space and that's what we
had to solve this problem initially. But we hope that other projects will also pick this up
and that we can together come to better and more sustainable long-term solutions for that.
One more question there?
Is there only a single key involved when signing and then starting to say anyway how through
HSM's could be required to sign something? Is there only a single key involved or is there
where the two HSM can be involved in signing something? So there's several keys involved in signing
a bill. There is several core system component keys, which each system component uses a different key,
then there's the AVB device key, and if we wanted we could have hundreds of keys for one
bill, but we try to keep it as minimal as possible. And the question about the second HSM,
I'm not sure, I understood like for parallelism that is faster or for a single component that
two HSM is required, for example, that the CIS signature is required and a particular procedure,
for example. Yeah, that's interesting idea to use two HSM's for that, but we haven't explored
that yet. We try to get the basics ready and solve all the problems and then iterate from
there on and improve the solution. Times up. Thank you very much.