Hello, everyone. I'm Vitali. I'm from Red Hat and I normally work on things like
either KVEM, you can meet me in the Linux kernel and other mostly like lower level projects.
And for the last couple of years, I'm involved in making sure Linux runs well on confidential
VMs. So I'd like to continue discussing the topic which I need to start it. And basically,
I want to have like a more generic talk on why we still have some trust issues when we use
confidential VMs and other issues going away anytime soon. So going to the basics, like what is
a confidential VM? What I mean, what's the main promise of the confidential VM? Why would you be
interested? The main promise is that it gives you data confidentiality, right? So nobody
but you as the owner of this VM, no matter where it runs, can get to the data which this VM has
a process or does something, right? So that's the main difference. And where can you get this VMs?
Well, becoming pretty ubiquitous. And over the last year, we've seen that some like cloud
providers, for example, Google, they just moved to a general availability back in fall, I think
like October and November, both with Service and P and TDX, Azure was doing Service and P for
quite a while. They have TDX, Intel TDX and like public preview. I hope we're going to see it in
J soon and in WS, you can get Service and P feature enablement for certain instance types, also
like in GA. So at least it's like major hyper scalers support this already. Then the question is like,
can you run it on premise? Because for quite a while, it was kind of interesting because
you look at the patches and they're still being like discussed upstream Thompson, but then you go to
cloud and they're like, oh, we're a KVM-based and we're already run confidential VMs. How do you guys
do that without the upstream enablement? Anyway, so last year we finally got patches for Service and P,
landed in like KVM-Kimu, I guess like live or than the OVMF. So basically you don't need to
do anything, you can take upstream bits and run Service and PBMs now. Intel TDX is a long story
because it's how it's done. It's coming. I guess that we will finally have a KVM enablement in the
next upstream kernel and KVM will land shortly after. For now, there is an effort in Santa
Ias, there is a Santa of Sig, which builds this not yet fully upstream bits. So if you are interested
in running Intel TDX on premise, check out this Sig. One particularly hard part of TDX enablement is
that you need to have this like D-CAP enclave for making this like TDX codes, which is basically
like an SZX enclave. In the way how Intel Pax did, it makes it really hard to put it in a distribution,
but I've heard good news that it's finally coming into Fedora soon, at least, and hopefully
like other resources. Okay, so you see that we can already get a confidential VIM either on premise
or we can get it in the cloud, right? And as I said, the main promise is that it's confidential,
right? So are we good, right? Like, all done, nothing to talk about? Oh, not quite, I guess.
Very simple, like, question that like, we don't want to depend on your infrastructure,
but you still need it to run your VM. So you think go to some your cloud providers,
web portal, click, click, click. Here is your confidential VIM, connect to this IP address.
Why do you think it's confidential? Why do you think it keeps any of the promises?
Yes, so if you go into this like, confidential domain, something people will tell you one
magic word about this, at a station. You need to test. And then you start reading more about
what a station means, and these people are really smart. They use their own language, and you will
very quickly get lost like I do, right? They like, oh, we have this rest, protocol, and everything.
So they discuss this protocol ever seen, and it's easy to get lost because the partners,
oh, we just get there some evidence from the instance, and then we do magic, right?
And it shows us the green checkbox, you're all good. What, what does it actually prove this?
At a station, what do we need to check? When we got this confidential VM. So I think that
the station is where they're like, umbrella term, that's umbrella. So my take on this would be that
first you need to make sure that you're actually running on the appropriate hardware, right?
It's not like some sort of an emulation of the hardware because otherwise you cannot really
trust it unless you grow this emulation yourself or something. Then one of the VM is built, right?
You need to make sure that it was built in the right state, the initial set of like CPU registers,
where you can set them like on 7th and P, and the initial memory, right? And then you get something
which is expected. Then you're VM probably boots somehow, right? So you have a chain of artifacts
from firmware to some bootloader, maybe another bootloader, and then to your external
and you kind of care what's in this chain. Second, if your VM has any storage, and normally does,
right, it even boots from a storage, then you may want to ensure that nobody is
tempered with the storage. And then comes to the last point that when you have the VM running,
that nobody else but you can actually access it, nobody can inject anything in this VM. And in
particular, some like provision agents are known for providing unlimited access to your VMs.
Anyway, hardware. So, well, we are interested in this confidential VMs, which are like
rooted in hardware, so you trust the CPU. And this VM technologies they normally can provide you
some sort of a think about it, like a sign document saying, this hardware is good.
How do you know that this report is coming from the hardware where you run, right? You can add some
data there, like for example, your SSH host key, right? So it's signed together with your
other information. And then you can look at the report and all key. I'm connecting to like a real
VM. But not all VM implementations will give you a raw access to this mechanism, which can
give you like a report, namely like on Azure, they use like a parameterizer, so you can still get the
report, but from a VTPM, which means that now you cannot do this like challenge, right? You
can put your own data in there. You have to trust someone who got this report for you, right? So your
VTPM becomes your new route of trust. And I'm going to talk about it a bit later on, like if you
cannot trust it really. Okay, fast forward, initial VM state is basically, you will hear, okay,
we have some launch measurement, which is like a hash described in the initial state of
straight registers and memory, which is great, but do you know what the hash is? Can you
pre-calculate it? So yes, if you run your own like CVM icon premise, there are tools where you
can give your like firmware and then describe the configuration of VM, like how many VTPUs you will
have something, and you will get some hash, which presumably you will later see in the VM.
What if you are on a cloud provider? Well, some cloud providers are nice, and they give you something
like a reproducible build of your firmware. Basically, here is the repo of the MF, with our patches,
build it, AWS does that. Thank you guys. You have to use like mix build system to build it,
which is funny, but the instructions are there, you can try it, and hopefully you will get the
hash, which you see in AWS's instances. Then, an is idea, if you can bring your own firmware,
you can also plug presumably eventually, eventually we don't have it on any platform available yet,
but eventually we may get this predictable hashes, and what if not none of this is available for you?
Well, then the only thing you can basically tell is that this hash never changes. Every time you run
your CVM on a public cloud, you get some hash, so if you don't think that this infrastructure
is that can hit it in the evil, and it was even from the world beginning. If you only care about
attacks like, oh, some intruder will get access to some host, which runs my CVM, it's going to change
some bits there, right, and hack into my CVM, then you may consider yourself protected against such a
attack, right, just because you observe the same hash over and over again. Okay,
bootchain, right, you're likely going to plan firmware, start firmware was measured,
ever seen you start booting, and how do you ensure that what he boot is valid, because normally
you load it from a storage, right, so it's not covered by any launch measurements.
Normally, two mechanisms are used, secure boot and measured boot, and again, it makes it sound
like as soon as you enable secure boot you are secure, but that's not always the case,
and the thing is what secure boot really is. There is some database of say good certificates,
and you check that whatever you boot was signed by one of the certificates from this database.
If you take the defaults, for example, there are like Microsoft certificates there, they signed
various shim versions, which load pretty much everything out there. So just by the fact that you
have secure boot enabled, you are not going to gain that much, unless you also measure basically
you remember which certificates you used in the bootchain. This is not what we call measured boot,
and that then you need a trusted device for that. You are going to record this like say
hashes of their certificates, which you used in the chain, or all the binaries, which you used in
bootchain, you are going to record them somewhere, so you are able to check it later, like using
evidence to prove that you are good. Where this place can be. Normally, normally you can use a
TPM device, like on hardware, but in the arena VM, so it's likely going to be a virtual TPM.
There is another mechanism, which I wanted to mention, that TDX, of course, you these similar
registers, which like RTMRs, have only four of them, right? But think is there not that
well supported across the stack, and they have very well limited use, so I'm not
convinced that this is going to get a wide adoption when TDX finally rolls out.
So VTPM, you measure something in a VTPM, and then you trust it. Well, in that case,
you want to be sure that the VTPM is trusted, right? Because if it's controlled by say
fully controlled by the host, well, then the host can change this measurement, right? You measure
something in it, and it will just swap this measurement with something else for you. The idea is for
7th and P, for AMD, that they have this privilege levels, and you can implement your VTPM
as part of firmware. You don't always know how close do they? Some of them
claim that they implement this in particular right Microsoft. If you run on premise, there is
SDSM project which implements VTPM as part of the firmware inside the guest, and the
beauty of it that it gets in the launch measurements, right? If it's part of the firmware, then you measure it.
For TDX, as I remember, like two architectures were suggested initially, they were suggested
you run, they don't have this VMP else, so they were suggested that you can run VTPM and separate
TD, basically a separate VMP, and connect them somehow, right? Which creates its own set of
complexities. The new architecture is called like TD partition, which is basically let's run
an asset hypervisor, right? We have create a one, which is going to be like privileged,
it will contain your VTPM, then you will put your Linux in L2. You will see, as I said,
like TDX and the implement is just blinding cloud providers, which enable it Azure. I haven't seen
any publicly available claim on how they do TDX, on how they do TPM on their TDX instances,
but it's like they have it in preview, so maybe by the time they go to G, they will say something
about it. Google as far as understand, never claimed that their TPM is isolated from the host.
So basically, you have to trust that it's, you have to trust it.
Yeah, yesterday in the Coca Devroom, there was a talk by Microsoft, where they described how they
do this like VTPM based booting for their Azure Linux. I did a similar talk in this Devroom
a year ago, which was more technical than this one, so I'm encouraged you to
watch it if you are interested. Okay, so you have your boot chain validated right, and then it
comes to storage. Think is that, again, storage is not covered by any launch measurements.
You may have parts of storage, which are say persistent. Think about like image-based
operating system, where you don't really want to hide it from the whole world. For example,
it's like an open source operating system. There is nothing to hide, you may say,
but you're still interested that the host doesn't temper or somebody else, and a attacker doesn't
temper with this static parts of your storage. You may use something like
deembarity. You can say sign a hash, what you produce very much. You can inject the key which
signed the hash in your secure boot DB, and then you can check that you are using it. Normally,
this works, right? But again, you will have to make sure that it's all measured, and again,
we go to the point where your VTP must be trusted. With a volatile storage, where you want to
save data, you have to encrypt, right? There is no other protection. The question is,
if you have some encrypted storage, how can you get the key? Again, you don't trust your infrastructure,
so go in, I don't know, a serial consultant during a password on boot is not an option,
because the serial is controlled by the infrastructure you have to do it automatically, and also
it doesn't scale, of course. But even without scaling, it can be done in secure manner.
Normally, people suggest to approach as how you can do it. First, you can use
a policy in your TPM device, and the policy in the TPM device, you basically say,
hey, my PCR register should look like this, and then you can unlock this password and give it
to the operating system, and you can, for example, say, like hashes, again, of the certificates,
which you used in the boot chain previously, or something. This normally works to the point where
you trust your TPM device. If you think that it's implemented right way, or you trust it for some,
other reason, then yes, you can leave the evaluation of the policy to the TPM and use it.
Another idea is to do, like remote attestation, basically,
whether all the evidence we've gathered so far, starting from launch measurements, boot chain,
something, give it to someone who holds our key, and receive a key which decrypts your storage.
Which is, yeah, just good. Only that, that is from the TPM, right?
Sorry, yeah. So, yes. So, what I wanted to say is that if you want to do it, then your
VTPM must be stateful. By stateful, it means that every time you boot VTPM, it has the same
keys in it. Think of VTPM as some like key storage and some very simple processor around it,
which can do some simple actions, like a decrypt seal and seal, verify something.
And if you boot them like public clouds today, again, hyperscalers, I'm using as an example,
they will all give you something which initially looks like a stateful VTPM. It preserves the key,
it's all there. You may wonder where this comes from. So far as I know, only Asia claims that
they do some sort of the under the hood, a attestation, early in firmware, to decrypt the state.
So it basically stored as a file, which can't insert excess, but the file is encrypted.
And to decrypt it early in that estation, they connect to their own Microsoft server, prove that
this is like a genuine CDM and decrypt the state. You can use it, but again, then you trust
these other part of Microsoft infrastructure, if you do so. Is it good or bad? Well, depends on like
what the threat model you use, what you care about. If you don't want to trust your cloud
provider at all, this just shifts your trust from one place to the other inside the cloud provider.
You're not gaining much, right? You may want to say that, okay, I don't want to protect myself
against a completely like evil cloud provider. What I care about is somebody hacking in some
host which runs my CVM and stealing something from it. In that case, you do get isolation from
the host where you run. This procedure allows for debt. What if you run your own infrastructure
and want to build something similar? In SESM projects, there is some work going on, and it's the final
was telling us about it yesterday in the co-curl. It is quite complex because you either need to
have, say, networking early, very early in the boot process, like that firmware is going to talk
to some server way before even your bootloader starts, right? Which is challenging, or we need to
go through the host and have some sort of a process there, but this comes with its own complexes,
especially when you think about this like real world deployments, right? Because you host and
your guest may not even be connected on the same network, right? Your host looks in some network,
like a management network, and your guests are connected with virtual functions to another network.
So, if you go through the host, you cannot reach a server which the guest can also access.
It's still in-gen, but we will have some sort of a solution soon, I guess.
Okay, so last, but not least, is provision in agents. If you run in on something like
a public cloud, you are probably used to think like clouding it, and the thing is about clouding
it, that it's fully controlled by infrastructure. The data source normally can do anything to
your VM, run any script there, and check any key. If you use it, then you trust it.
It's not isolated from provider or anything, you cannot sign it yourself, or at least like
there are no public-level solutions for that. I know that this has been looked at by some cloud providers
who want to provide some sort of trusted data source, so far I haven't seen any implementations.
So, what you have to do is basically avoid any provision in agents, which is hard, because
even if you don't want, for example, your clouding it to inject your SSH key, you still may want to
say let it configure your networking. So, something like a policy-based mechanism would have been
nice to have, just that we don't have one. Okay, so that was basically the state of art as I
see it currently, as you can see in some places we have some fairly good solutions in
other places, not so much, and the work is ongoing. So, thank you, any questions?
Go ahead. I have two questions actually, and if you're possible to embed the encryption key
for the storage and the firmware itself. The question is whether it's possible to embed the key
to the storage in the firmware? The firmware is visible by the host. You don't want to give your
storage key to your host, in no way you want to do that, right, because the host will be able to
then do something with your storage. So, no, you cannot do that.
No, even if you are bringing your own firmware, you cannot give your giving away this firmware
to the host. The host is going to use it to build your VM. It sees it completely.
In my other question is, what do you want to do, computational dedicated servers,
or computational containers, is it for you? The question is if we want to do
like, confidential containers, whether it's different or not, it's slightly different, but
normally you still, if you want to do it, say again, like public cloud infrastructure,
this underlying VM which runs these containers must also be trusted by you.
Because think about this, if your cloud provider can swap your kernel in your VM with something
different and use your kernel for your container isolation, can you guarantee anything about
this confidentiality of these containers while they're processing? Probably not, right? So, it's a
building block, right? Then you will come to get in this container securely, get in data for it,
you know, making sure that only like these trusted assets VMs can do it. Yes.
Dedicated servers, I mean, CVM technology is better than bare metal in this regard, right?
Because if you have your server somewhere, right, then normally somebody can have like physical
access, for example, to the server, can like inspect your memory, for example, or something, CVM,
kind of protects you against that, right? So, well, out of time, if you have more questions
please talk to us outside the room. Yeah, thank you again, I'm around.