Cool. Hi everyone. Hey guys doing. My name is Aurelian Bumble or Aurelian. If you speak French,
I'm just actually my first time at Fazlem and it's kind of a funny feeling for me being here
because I'm born and raised here in Belgium and you don't have to screw up and move on with
a chicago abroad for work and now I'm back here for work, right?
Workwise, I work with the Confidential Containers Project, Lately I've been looking
at storage. Otherwise, it's just helped out with different aspects of the project, no
to be the CI and then I also work with the Cadillac Containers Project where I'm a part of
the Architecture Committee there which is a group that, you know, steers a project and has
a final CM decisions and in a Microsoft, my Simon really is, the Linux Confidential
Platform as part of the Azure Linux team. I work with our channel and Kita, that presented
an earlier, I work with a lot of time, right? So I know some people in this room.
So when you talk about, you know, Confidential Computing, a lot of the time, people focus
on the compute part, right? So weapons in the memory, not so much about networking or
storage, right? And so today I'm going to share with you folks a little bit about what we've
been working on with the Confidential Containers Community to implement secure storage from
the perspective of, you know, containers in a Confidential setting, right? And so mind
you, a lot of this stuff is still very much a work in progress, but at least this will
give you a good overview on the current tenant things. And so I'll talk about both the
implementation itself. We'll also have a ties into the broader ecosystem, right? And
by the way, we have a PR, that's not for review. I can share with you folks if you want
to, you know, criticize my code, please feel free.
Now, first and foremost, what is the Confidential Container, right? So when you think
containers, you think typically your standard Docker runsee containers, right? Where you
share the whole kernel with the containers. Now you can do better isolation than that using
the kind of containers you put the containers in a virtual machine, right? That still doesn't
prevent you from a potentially malicious host that could attack your containers, right? And
so to address this problem, we have the Confidential Containers Project or Cocoa, that's
the, that's the term that we're using on the project to refer to it, right, Cocoa. And
we build on type of cata to leverage a trusted execution environment or a T, to guarantee
the Confidentiality of that VM, right? For example, by encrypting the VM memory, right? And
even though the workloads, the containers themselves, can use that to run a process called
remote anestation, that don't give you essentially a cryptographic proof of the contents
of the VM, right? And then later on, I also want you to use the concept of container security
policy that we use to secure the VM boundary. So, you know, the interface between the VM and
outside of the VM, right, which is entrusted by by designing a threat mark, right? So now,
these three are container run times, right? Be address a question of, how do I run a container
on the single machine, right? Now, when you're talking about the point of scale, when you
have loads of containers that you want to deploy to a cluster of machines, right? You want
to use something like Kubernetes to handle what we call the orchestration of your containers,
and so that's going to handle the deployment, you know, scaling, you know, balancing,
services coverage, fault tolerance, all that good stuff, right? And so, it's very important
for the Coco Project to work well with Kubernetes, because that's what people use, right,
to deploy to scale. So if you want people to use Coco, you want to work well with Kubernetes.
And with this, get us one of the aspects that's managed through Kubernetes, old storage.
So it's going to be, you know, pretty important for us here to have some understanding
that Kubernetes is here in the picture, and that we have to work with it.
Is there a little bit more order?
I just, I think I'm being sold out more about it. I agree with a lot of it. So before I dive
into the, before I dive into the details of the storage implementation itself, I'm going
to quickly run through this already simplified diagram of the life cycle of a confidential
container, right? And so we'll build on top of this diagram to understand the storage
later on, right? And so this is the view from the host. And so when you want to create
a container, you're going to create this container spec here, which is a YAMO file that you sent
to your cluster, right, to Kubernetes. And then on every host-oriented cluster is going to be
a component called the Q-blood from Kubernetes. And the YAMO file is going to reach that Q-blood,
the Q-blood is going to talk to the Cateron time. And then the Cateron time will trigger
the virtual machine manager, right, the VMM, to create this confidential VM right here, right?
And in this confidential VM will be installed a few components, including the Rust Cater agent.
And this Cateron agent will be responsible for creating the individual containers inside your VM,
right? And then it's going to talk to the Cateron time interface with it over our PC, right? And
so one key aspect to understand is that anything that is outside this, you know, this green box,
the confidential VM, right, is not trusted from the perspective of the VM components, right?
Now, when you talk about storage, broadly there's two types of storage that you want to consider.
There's a firmware storage and then persistent storage. And so I'll start with a firmware storage here.
Hopefully, you have some time to say a few words about, you know, persistent storage before,
you know, I get kicked out of the stage. So let me dive right into it. So
even if in most storage, right, there's a pretty crucial use case to support,
because every container that's on point is going to need to store data that is not going to
live the container, but that is also not going to fit into memory, right? And so enabling that,
you unlock use cases like sharing data between the containers in UVM. I don't know,
having temporary lock storage before sending those lives over to logging service, right, in the cloud,
or caching and check points that's not charging so far, right? And so always the first goal here,
right, is going to be security and especially confidentiality, because even if the storage is
a firmware, you don't want it to be accessible to the trusted host or to any other
untrusted control plan components, right? And then we want to be careful of how we integrate
with Kubernetes, right, because Kubernetes already has storage features that people are leveraging,
right? And so we want to make the transition to Cocoa as smooth as possible, right? So we want to be careful
of how we integrate with Kubernetes. And so with this, the key ideas in the design are going to be
from the host itself, right? We're going to create a temporary block device. We're going to pass
this block device into the VM. And then from inside of the VM, we're going to encrypt and format
this device, right? And so here, one important challenge is going to be securing this interface
between the VM and, you know, outside of the VM, right, with the, with the kind of runtime,
because we're injecting a VM inside that devices at that VM, right? And so we want to make
sure that the interfaces is protected, right? And we'll do this through the container security policy.
Now this is the same diagram that I showed two slides ago, so we have to basic
container lifecycle, right? And so again, you started by creating this container spec, right?
And so here, with our confidential storage type, it's a new storage type, right,
that we're introducing into the control plane, right? And to implement a new storage type
in Kubernetes, you want to implement something that's called a container storage interface driver.
So it's CSI driver, right? And so, you create a container spec. You put a reference to your CSI
div in it, right, for your storage. Then you send the spec to your keyboard, the queue is going to
see that reference to your CSI driver, right? It's going to call it to your driver. In this case,
our driver is, you know, create the temporary block device, right? Send our block device to the
counter runtime. And then the counter runtime will virtualize that device as a photo block device
into the VM, right? And then also send some metadata about it down to the catty agent over our PC,
right? Now, when the catty agent sees your block device with that, you know,
custom reference, it knows it's a confidential storage, right? It's going to call it to this component
called the confidential data hub, the CDH. And it's in this CDH that we're going to first generate
a random encryption key. We're going to use that key to encrypt the device, right? And then we're
going to format that device and finally mount it into the container, right? And so,
one property here is that, you know, the key is generated inside of the VM, the VM's encrypted,
right? The key does not lead to VM. And the key gets destroyed with the VM, right? So without this,
we can guarantee that the data is only, you know, accessible to the confidential VM here, right?
Now, this is the last, the last time this is done, right? So it's not going to get any more
complicated in this. The last aspect is going to be the security policy. So, you know, to
security, the interface right between the VM and the outside. So, one key thing that you have
to understand about Kubernetes, right? So, when you create this container spec, when you send
your YAMO file over to the cubelet, the cubelet is not actually going to send the YAMO spec
as it is to the kind of agent. It's actually going to transform augment that that spec into
a much lower level spec, right? And it's this lower level spec that will actually be executed
by the, by the kind of agent. And so, in the kind of agent, when you receive that spec,
you know, you're going to execute it like the lower level spec, but because remember that
all those components outside of the VM are entrusted, they can still temper with the spec,
right? Between the moment where you create a container spec and the moment where it reaches the
the kind of agent, right? And so, you want to protect this lower level spec, you want to guard it, right?
And we do this with a tool that we call the policy generator. And so, this policy generator is going
to take your YAMO spec. It's going to generate a security policy that maps one to one to your lower
level spec, right? Re-inject this policy into the YAMO spec. And then we're going to send both
the YAMO spec and the security policy into the VM, right? And then once it reaches the
the kind of agent, we're going to enforce that the, that the YAMO spec and the security policy match,
right? And if they don't, we're going to reject the container creation request, right? And so,
this is pretty important in our design because, as I was saying before, we're injecting a device
into the VM, right? And so, we also want to make sure that that device, the metadata, even included
without device is not temperate with, right? And so, we're going to include the device as part of
the security policy as well by modifying the policy generator, right? And so, this was one half of it.
So, if you notice here, the, you know, we've enforced the policy inside a kind of agent, right?
But the policy is still not trusted at this stage, right? Because it's still coming from inside the VM,
right? So now, we want to ensure the trustworthiness of the policy itself, right? And we do this
through the outestation process. And so, here, the way that it's going to work is that after the
kind of agent has enforced the policy, it can, you know, connect this outestation agent here, right?
And we will have a remote and trusted outestation service on the right here. And this outestation
service is going to have a reference security policy. So, the ground truth for your policy, right?
And so, then the, the outestation agent will send the container security policy, right?
Fetch it from the T, send it over to the outestation service, the outestation service is going to,
you know, compare the policy that receives and the policy that it stores, I'd refer to security
policy, right? And it's going to send that result over to the, to the outestation agent, right?
And if that process passes successfully, then you can guarantee that your policy is trusted,
hence your container is trusted, and hence your device is trusted, right? And so, that closes
the loop here on a confidential referral storage implementation, right? Because we've, you know,
ensure trust readiness and to end here, right? So, that was it on the, that the only part, so,
you know, now we can take a breath a little bit. I'm going to show a very quick demo in a shell here,
where we're going to deploy a spec, so I'll go look at the spec, we'll deploy it and we'll
play with it a little bit, right? So, let's look at the spec we have, you know, the Yama spec,
right, a bunch of metadata up top, the name of the container, you're going to create my app,
then I, you know, generated the policy before, the, the talk here, I'd just have some time.
So, it's basically to phone code it, it's pretty large, so most of it is going to be truncated,
right? But that's how it appears in the spec as in the annotation. And then we make a
reference to our custom storage site by the CSI driver, which is called Coco, local CSI here,
we clean about 10 gigs of storage, and then we mount this storage inside a container on slash
mount slash encrypted, right? Now, we can deploy this into the cluster with acute cutout
apply, the container was created. Now, because this is a test environment, right? We can
execute it into the container, and then we can list the faster than that we just mounted.
So, from the right to left here, it's not in on slash management and such encrypted, like I said before,
we have about 10 gigs of storage, it's an x-hole fast system, and then here we have a virtual
device, right? Because the original host device was, was encrypted, right? And so here,
we used a DM quit and DM integrity, which will create a virtual device, they're not
part of the device, right? And then we can see the into this folder, it's going to be empty
at first, right? Because we just created it, which is for metadata, right? And then you can write
some very sensitive and complex payload into that storage, and then you can read back from
it, right? And so here, I wanted to show that from the perspective of the container, right?
For you as a container developer, this encryption layer is totally transparent, right? So,
the container does not have to do any setup, you need to configure the encryption
settings and so forth, right? Only have to do is specify that CSI driver in your container spec, right?
Now, really quickly on persistent storage, there's, I'm going to show one design, you know,
there's a few designs that I'm thinking about, that fulfill different goals, right? The good thing
is that the design, if you understand the firmware storage, the design is very close, there's only
two components that change the CSI driver, and then the, those are the old slides, never mind,
the CSI driver, and then the QR code service. So, this was the firmware storage, this is
persistent storage, right? So, two key changes here, right? And the CSI driver, we're not any
more going to create a new block device, right? We're actually going to get some storage that's
pre-provision, right, from somewhere in the cloud most likely, right? And so, this bug device is
going to be pre-provision, it's already going to be encrypted, and it's already going to have
data in it, right? And so, now when we get into the, the confidential data hub, we're not going
to generate a random key anymore, we need to get a key from somewhere, right, to decrypt,
forgive me, the storage I was already created, right? And so, we're going to get a key from
this key blocker service here, and the adecision clearance is very similar to before, right? So,
we triggered the adecision agent, the adecision agent connects to the key blocker service,
the KBS here, right? And then the KBS itself is going to perform the, the adecision, right? And then
if adecision passes, the key blocker service will release a key to the adecision agent,
and then the CDH can decrypt the storage, and then, you know, expose it to the container.
And now, yeah, that was that was it folks, so the next step, so it didn't be for me to
kind of merge that, if anyone wants to HPR, trying to figure out this persistent storage stuff,
and if you have some ideas, you're going to come and talk to me after this, like, please,
for, for, yeah, I'm going to open to you any ideas. Thank you so much, folks. Thank you.
Thank you, thank you, thank you. So, we have two minutes, we'll take one question, I think, one, it's
a question. So, you're asking this for a fan of storage, you're often putting that in that stuff that is known.
Right, no, no, by the way, I think that's the assumption. So, you're saying,
for a fan of storage? Yeah, okay, this is why it's the case, what I said it.
So, I ask you, is that a statement? That part is the statement, the question is, how do you
move together to get a fan of storage? Okay, so let me tell you another question. So, the question
is, for a fan of storage, you, when you know that there is something that is the most
noise that you know about. Yeah. So, you have a non-linked space, for instance, let's say,
container, you may show something like that. Right. How will you mitigate it, you're going to
integrate the text and the text? So, for example, container images are not stored in
this a formal storage, right? This is just for data storage, right? So, when you're saying,
if we have more content in that, if we're more storage, I guess I do still want to send a
question because, well, that's storage going to be encrypted, right? Right. If you know what is inside,
then what are you talking about? We play the text, there is an evite. No text, no text, no text, no text.
Yeah, I'm not sure, I know that there is, you know, for example, we use a gene integrity, right?
And I know that's still available to replay the text, right? That's something we were about.
That's something that we're working on right and we're evaluating different solutions, but yeah,
right now, you know, we're just getting started on this, right? So, we don't really have a,
I don't really have a good answer to give you, if you will, can you? Okay, that was it folks.
Thank you so much, thank you for your time.