Okay, hello, so quickly switch back, so happy to be here for the opportunity to talk a bit about what we are doing in the confidential containers project in terms of attestation and our challenges that we have in this particular environment.
I'm working at Microsoft in the Azure Collinux organization and I'm involved in the CNCF confidential containers project as a contributor and the context quickly so confidential containers is an effort to introduce confidential computing into the container ecosystem.
And confidential computing today is mostly a VM technology, so this is a bit of a friction because containers usually are not launched in VMs.
There's like different boundaries, namespaces that are being used to isolate containers, but if you want to kind of retrofit confidentiality to containers then we have to at VMs.
So luckily, this is not something that we have to start from scratch because there are like prior efforts to do exactly this for maybe using containers in my detention situations.
You want to have a strong virtualization boundaries across in an environment and cataccontainers is one of those projects that we're existing and a good fit for the project to integrate into.
Yeah and quickly I think this is simplified but it's it's important to understand the context what it's actually going on today when you launch a container so it's.
Usually he would do this on a Kubernetes deployment Kubernetes cluster a user with right there their spec.
The container spec and when we talk about container spec often it's it's actually multiple containers that are logically grouped into a pot which is like a collection of collocated containers we can think of maybe a local cash or a message queue like.
It's not necessarily just one process and this is what we call pot and those specs we deployed to API server and the API server kind of dispatches this to the notes where they actually work load is running and on the notes we have.
The component that's called a Q blood and that Q blood itself then starts interfacing with the actual container layer so it used to be Docker under there today it's a bit more abstracted container runtime interface.
There's different container runtime's most popular ones probably container D today and this one is again translating it into OCI runtime course and you have a part of the runtime that is really doing the.
C groups the name spacing and spawning the processes that are that we defined in our pot spec so this is how it looks in a more or less vanilla set up for containers today.
And yeah this like I mentioned the sandbox that you see below this terminology in the container runtime space where we basically hosted the pots like this deployment item I was talking about the collocated processes and.
This is more or less what people work with and it's a good interface for confidentiality to to to kind of build upon.
Yeah, you can also build for example say we will introduce the confidentiality reality boundary on the on the note.
There's also options to do this but this has several problems because.
There might be untrusted code that is deployed on the note I know a lock scraper or other.
Like administrative processes and the confidential containers project settled on the on the pot as an extraction and that introduces some challenges.
So as we see here like the confidentially m boundary is sitting between the container runtime and if you will the container runtime back and in our case it's.
The cutter agent that's spawning processes and kind of being the watchdog.
And.
This can actually also be the case that it doesn't it's not necessarily a VM that is on the running on the note this architecture allows you also to.
Have a confidential VM that is remote so this is one flexibility of this cutter architecture that there's also remote hypervisor there is not actually local.
But can just talk to for example cloud APIs and in this case it works.
Pretty well because anyway for confidential VM so you have very strong isolation between the host and the VM so it's not like you can do a lot of research sharing anyway.
Yeah, the at the station architecture for confidential containers is not very exotic I would say like you have a workload that is measured that is included in evidence and there's an.
At the station service in a key broker service that's called trustee that's part of the project that is kind of very close to confidential containers but it can also be.
But I think there's nothing really special in this like in terms of the.
Yeah, the ceremonies of the station that confidential containers does.
What's more exotic is the fact that we have to deal with a lot of dynamism.
And this is something that we need to measure like this is sandbox that we see in earlier.
Is one part of what you call workload like that is static components we see below it the cutter agent and the base OS and this is all you can measure you can use measured boot for that so this is I think.
Because.
Less challenging than than managing measuring the dynamic sandbox and since focus on.
Yeah, so generally maybe as a as it would be marked like the Docker OCI container themselves are already have some.
Convenient way of of measuring them built in so they're the content that restable if you reference them by the digest.
You get and you can verify that is actually what you're pulling for an OCI registry is what you but you intend.
To what you specified.
The problem is that sandboxes are controlled empirically.
So what we're seeing is like this is a lock from the cutter RPC.
Basically and but what we get from from this earlier sketch from the Kubernetes.
Specks I distraint slides into a list of of calls in this case.
It's really great to sandbox and there's some exchange.
But then there's a container being created that's like the first thing that happens is actually two containers so one is a pause container that is kind of an implementation detail of Kubernetes.
But there's as you see like those processes are not something that is very declarative it's pretty imperative and it is also not arbitrarily how those are executed like the pause container needs to be first or in a container needs to be launched first and this is critical.
So I mean the objective stays the same for for confidential containers and the end we want to ensure that the only the operations that are being executed are the ones that we intend to run before we release a secret it has to be verified in remote attestation.
And this means like we have to somehow get to a notion of a container workload how we can measure this and.
Yeah, as I mentioned before so this is we have to deal with the dynamic nature and it's not just that.
Yeah, that the commands are the imperative but there's also inherent dynamisms in Kubernetes that.
For example, services that are in the cluster are injected as environment variables for service discovery and an environment variable can can if you just without very fine it can support confidentiality.
Yeah, that just seemed like someone changes the red house very heavily and talking to a local red house and more but to the remote red house to this pretty critical.
And also often there's a governance built into Kubernetes services so in the user wants to deploy container a version one but the platform team decides that you cannot do this and they will always rewrite your container to version two transparently so this is something that is also a challenge in those environments.
And there's like different options to deal with this one of the option that's being discussed is locking the control plan in a way that you really only.
To identify the trustable with predictable operations but it's very hard.
For my perspective it's very hard to do this because the Kubernetes API surface is pretty large and it's always increasing and it would rely on which require a lot of.
And we keep up with this and have a providing experience that is not completely foreign to users of Kubernetes but there's an effort on the way it's called split API that tries to do this.
I don't really know where we are with this but I think people are actively working on this.
We also have the option of keeping a log a bit like like IMA.
The problem that we have in confidential computing is that not all T is at the moment provide registers that we can extend at runtime.
And as I mentioned like some payloads are not really predictable because environment controls them so the verification of such a log is also not something this previous but I wouldn't rule this out in the end.
We have to get to architecture that is somewhat similar.
So finally the option that we ended up with is having a policy in the TE.
So we kind of described the invariance of deployment like something that we really want to know.
Earlier like we want the image digest for example the OCI image digest.
We define or we say like there's acceptable environment variables that can be injected.
We're seeing starting with service.
We check like all cutter RPCs like for example exec into a container.
Loving by default and have users allow list the option in the policy and be a cherry pick what's required.
And this is something that is currently implemented in the cutter agent.
The implemented on using regal which is in Kubernetes land familiar tool.
And cutter also has to generate policy like saying default policy from a pot spec.
And you can see here like how this blocks into the whole architecture.
So in front of the cutter RPC we have a policy engine that is part of the measured static container runtime if you will and this evaluates like every call.
So we have the call and the payload and we check whether there was in a lot parameters.
So you see here like this is an example policy how it looks like you see here we allow a few.
We request we define commands that are allowed and this is something that a user would have to do or has to do in this case.
But those policies can be rather large.
So it's maybe something that we need to automate or abstract a bit.
Yeah so this policy also needs to somehow get provisioned to the TE.
So this is instead of the generic runtime which is static and we can measure and there's a predictive measurement.
The policy is specific for the workload.
And we need to provide it as measured configuration at launch and we also need to link it to the TE.
And this is like one way of doing this I think is pretty typical is somewhere hashing this dynamic configuration.
And store the hashing to S&P's host data or MR config ID for TDX.
Or extend it into VTPM because the very fire can then check that as host data for example is part of the exercise report.
The very fire like this is not changeable or anything.
So if you launch VM and you specify host data you know at verification time that this VM has been launched with this configuration.
So this is how we bind it to the TE.
So it's not just a policy, it's a bit more but we call this init data specification.
For example also to set KDS certificates etc.
But it's rather straightforward.
It's essentially just a dictionary of past and fire content that we provision on the host.
And this is currently being implemented.
It's not available for all TEs but the architecture has been settled and it's probably will be in the next release.
You can see here from the user site it's embedded into the pot spec as an annotation.
We just have I think some mutations here but in essence you run.
You do some basics before I think encoding and then you will provide it in the pot spec that is sent to the API server.
Yeah this would be like a truncated example of the init data that we use.
So there's a hashing algorithm so that you can tell how I find it in host data for example.
Where it is it which I would not have to use to identify this.
Yeah now the challenges that we have with this architecture from my perspective I think the problem is that the policy state left.
State left and it tries to to kind of map it onto an imperative set of calls.
And there's like complex orchestration so if you just watch it like every RPC call individually.
We cannot really express stuff like the init container has to run first.
And if the init container runs second then this is not what we intend.
So now we have to kind of introduce a notion of state in this and this actually I think actually being implemented or there's an open.
We have a set of practical problems that are kind of solvable.
I think the size of policy is Kubernetes has limits in terms of how much you can fit into annotations.
And yeah we have to come up with schemes like compression is probably workable.
But in the end I think the user experience is not really great.
If you have to write this yourself so we have to really build tooling that makes it easy for users to deploy their products into a TV.
And I think the conceptual problem I'm seeing is like if you're really tracking the RPC interface closely in those policies.
It's very easy to introduce like exploit vectors unintentionally because you have to basically model the behavior or have to keep track of the behavior of those APIs also.
And for the people who work on Qatar but not cocoa they would have to cover this very closely to make sure that is also like in the default policy you have.
You have this managed.
And I think on all like there's more challenges like runtime measurement that is coming up now.
I think there's compatibility is with confidential GPUs.
But I think yeah we will address them when the users demand them but I think those are all solvable.
And if you want to recap quickly.
As in the at the station for containers and boxes I think it's the tricky topic.
And the solution right now that we do that we offload the verification to a policy is like a workable mitigation for the time being but maybe not.
The solution that we that we cover all cases.
In the future and I think that would be the talk from my side thanks for listening.
Thank you.
Yes.
So complexity in general doesn't help with security a lot.
So I'm wondering how far we stretch this case in terms of complexity like the chances that people get this right.
Yeah absolutely.
I mean that that's really also my impression that there's two complex I think.
And we have to somehow get a grip like what.
Can we guarantee how do we model this but in the end we have to we have to like yes.
Few we can do about.
The container ecosystem.
So we have to like confidential containers has.
Yeah wants to pretty much pick up users where they are and they want to have essentially just a small annotation in their pot that says confidential through.
And that's all they should do and this is this like the north star then you have to kind of deal.
With what's given to you but we have to somehow I think find.
Yeah a way to reduce this complexity I agree so this is very easy to to inadvertently or maliciously introduce.
Problems in this architecture.
Oh yeah the problem.
The question was the question was like this looks all very complex and it's very easy to like security and complexities don't mix well.
And what can we do to limit those?
Yeah I wrong if we have.
First the top very apart.
Like reducing like the project old like reducing the.
The.
Yeah it's like very.
I think a user centric project that like where I think the applicability.
For the solution was supposed to live there from from the beginning like we have to start.
See like what what can we do that people can actually start using this and and then.
As we go we figure out problems with this architecture and have to see how we can address them.
I mean.
Yeah I mean the.
T is that we mostly work with our.
So as we.
So we have run time but there but.
The idea was to abstract like T is.
To it with minimum and so there's no notion of runtime measurements yet.
Confidential containers.
But this I think will be a problem very soon because we're like they're looking at very long running training.
Machine learning jobs rather than engine X containers that people spawn up.
So we have to probably to think about continuous.
Measurement.
And we have to see like how do we.
How can we fit this like Azure has this.
Soft is in a way that there's a BTPM.
That is running on a higher privileged level that is isolated.
As a confidential VM from the host, but also from the guest.
So you have.
Kind of this confidential playground extended it's it's a bit more complex.
It's but it's linked to the TE.
Cryptographically.
And there you can it have like.
Use other the runtime tools to do measurements and into PCRs.
So maybe if this architecture as in coconut.
It's also providing a VTPM for for S&P and we have RTM R's and TDX.
So maybe there's a assumption that is reason for the future.
Yeah.
You mentioned trustee and trustee.
So you have to encrypt it.
And then you need to get the encryption key first.
But you want to get that encryption key.
When you run it in a container, you first need to get it.
Of course, through the application.
Then.
So when you're blocked by it, then there you should get it.
Yeah.
So the question was what about like the encrypted images.
There were missing there but the encrypted images are essentially.
So they're not you don't need encrypted images.
And encrypted images like at the moment images are pulled in the guest.
For the for the upstream architecture.
And so what you can do, you can.
The guest OS starts and it does performs like at the stations.
The ceremony and you get.
Yeah.
You're able to to to to to to get the encryption key before you launch the container.
This just part of the pre container.
That's something you need to get.
Yeah, but it's.
Yeah.
Thank you.
Thanks.
Thank you.