Hi, thanks everyone for coming to the talk.
So what we're going to discuss today, we're going to discuss about containers, why does
the attack surface of containers?
It's probably going to be a rehash of some ideas we have prepared in the previous talk,
so that's very interesting.
We're going to discuss about some known attacks and mitigations.
We're going to see danger zone and divisor, danger zone, will be like a reference project
in order to understand how containers work and can be attacked, and we're going to see
how we integrated those projects.
Quick word about me, I'm a software engineer for freedom of the press foundation, and I'm
a maintainer of the danger zone project.
And quick word about the freedom of the press foundation, we are a US nonprofit, we do
several things regarding press rights, we train journalists, you know, to keep them safe.
We do some advocacy for press freedom rights, we track press freedom violations, probably
we're going to track a bit more in the coming years.
And we build security sensitive tools for journalists, like SecureDrop, I don't know if everyone
knows SecureDrop, and danger zone, which is the other tool, we're having.
So I keep talking about danger zone, let's see what that is.
You can see on the left side live, in practice, it's an open source tool that was written
in 2020 by Michael Lee.
If you don't know the name, it's the guy who did Opsack for the Snowden Licks, so it's
pretty cool.
It's been maintained since 2022 by final press foundation.
It was a written, in order to help journalists on the intercept, which is a US publication,
to sanitize some documents that they got from the sources.
It's basically a desktop application, I transfer my queries, Windows, Linux, because that's
what journalists sent to use.
And it basically takes a suspicious PDF, it passes it through a sandbox, and then it gets
back, let's say, a safe PDF.
This is not a novel idea by Ann Means, it's been used, well, in the army, but besides the army,
it's been used in CubeSOS, which is a very interesting operating system, it has this
trusted PDF feature, and the main logic behind danger zone is there.
Unlike CubeSOS, which uses Zen, the Zen have a visor and disposable VMs, we are using
containers, you know, it will chill out.
I'll briefly explain how the danger zone works, so you have some suspicious dogs, you
pass them to the danger zone, which is the application, but that application does not open
the dogs, and that's important, because they may be malicious, what it does, you know,
is that it opens, it's spawns a container, it passes the dogs through the HDD of that container,
and it gets back, not a safe PDF, but it gets back pixels.
And that's because you don't know what's going on on that sandbox, it may have been compromised
due to the dogs, so what you want to have is like a very damp format of that original
document, and that's just RGB pixels, three genres of color and all that.
We then recreate the original document from set pixels, page by page, and by doing that,
there's this analog call, where essentially we believe that every malware and every metadata
has been raised.
The result in documents is not that is the work with, so we do some OCR extra, so that you
can select text and all that.
As you can see, the containers and their security properties are really important for us.
Else, our users are in danger, and we're talking about users who are being targeted by some
serious people.
So the question here, some will say here, and that was, again, the case of the previous
SOC, that containers are good, but for security sensitive workloads, they may not contain
as well.
And that's true, I agree with that, but we have to think of the alternative, the alternative
is that our users would just open the document in the laptop, and we don't want that.
So unfortunately, or fortunately, we are in a position where we have to do some harm reduction.
You're going to open that file in any case, so try to open it by us, where we use container
so it's easier to install in your macOS Windows laptop.
But at the same time, we'll try to keep you as safe as possible by doing what, by trying
to identify what is the attack service of a container, and trying to figure out ways to reduce
this attack service.
We cannot complete null fight, we are well aware of that, but we at least can reduce it.
So the very first thing we have in mind, and I'm going to talk about mitigations here as
well, is that you cannot have arbitrary code execution in a container if there are no
software bugs in your product.
Does this happen?
No.
What the first mitigation or comes to mind is that you need to constantly scan your container
image and regularly update it.
On the bottom left, you can see an exploit that could affect a LibreOffice at some point,
that was a specific CV for that.
We got that first security scanning, and we should update it very quickly.
Now a second vector is network access, and usually most containers are connected to the
internet, so it doesn't apply to most people, but here's something interesting.
There's this concept called kind of a tokens.
I'm not sure if you're aware of that, but there are, let's say, silent alarms that you can
put in documents.
Usually they are triggered by AcrobotBDF, so not by other views, but in principle, you
can open document.
The document tries to get a fetcher resource from the internet, and that resource is controlled
by the owner of the document.
This way, people can know if what you are viewing right now, if you are viewing actually
a sensitive document, if they're sending an authorized view of the document.
If you are a journalist, you won't really even know traces of such a thing.
So mitigation here, disabling it, working if your container does not need that.
Okay, so the first things were pretty simple.
Let's go to some more juice stuff, third attack vector, it's your shy runtime itself.
The your shy runtime are usually in kind of a privileged position in the sense that they
are written in memory safe languages, go rough, rough, because you know, ready.
And also, they tend to set the states, they tend to create the namespaces, create the
cgroups, fork, and then let the links kernel do the rest of the protections.
That's the theory in practice, the habit bugs, you know, shy runtime's, either letting
the user, the containerized process, take advantage, they exploit the actual runtime or access
resources while being containerized.
Like the host file system, when it should.
So in order to protect the OSHA runtime, well, of course you can run rootless Docker
podman user netters, I'm not sure probably no one does that, but there's this option.
But in practice, for end user laptops and all that, even rootless podman, you're still
the UID 1000, you're the 1000, you can practically read your Firefox history, passwords,
whatever, so it's not a super good mitigation.
Fun fact, for people who are using podman, just so you know, a good thing to do is to disable
logging if you haven't done already for sensitive workloads, because if that workload is
sending out but to STD out, it's logged in your general city.
Now what happens if the process somehow escapes or not actually escapes, but sees the
view of the host file system, there are some mitigations for that as well.
Of course, SLINUPS is a very good mitigation, having the process not gain any new privileges
it's also important as well.
What's also really important is to note that the UID zero root within the namespace usually
maps to the user who created the namespace outside that container, so you have to restrict
it somehow.
It's best to create containers using a different user than UID zero within that, so that's
you, this one thing, and podman for and later has this very cool user and S-NOMAP of
lag in which there's no user from outside the container, from inside the namespace that
is mapped within that container, so practically there is no correlation between use outside
the container and within it, and that's very good.
And of course, dropping all capabilities, you'd be surprised at how many CVs can't
be mitigated by no capabilities in the container.
Of course, your workload may need those capabilities, in our case the engine needed none,
so that was interesting.
We're talking about the OSHA runtime, that's an attack vector, but let's face it, the
biggest attack vector, it's a Linux kernel, and you can't do much there, if there's
a buffer with lower something in the Linux kernel, then it's kind of a game over, and
there's bound to be, because it has like a very vast attack service area, what you can
do here is that there are ways to profile and trace your workload, which will create
a second filter, which will have a more restricted set of Cisco's only the ones that your
workload has used in real time.
This may not be accurate, but it's a good starting point to fine tune some stuff.
The less Cisco's your workload is allowed to do in theory, basically reduces the attack
service of the Linux kernel, so that's a big thing.
And by default, the default second filter has quite a lot of Cisco's enabled, because
you need to, it needs to adapt to most workloads out there.
And finally, there's the vector of hardware bugs, spectrum, meltdown, and all those types
of vulnerabilities.
Historically, VMs were not able to isolate the workload and not exploit those bugs.
So mitigation, either burn your laptop or use a burn your laptop, pick your poison, whatever,
but there's not a good mitigation here.
We know we want to use containers because it's easier for your users, and we know some
ways to reduce the attack service, but still, there is the link to the kernel out there.
There is your share on time.
And ideally, indeed, there's a lot we would like to isolate the containerized process from
those things.
We will slip better at night, let's say.
So can we do better?
This is where the advisor comes into play.
Now, some talks have talked about before, I'll mention a few things real quick.
The advisor is a Google project, it was written in 2014, open source team May 2018.
It's written in Go.
It's available only on Linux, and it has an OSHA runtime run in C, which can take a root
surface path and contain very much and create a container out of it.
Now, what's different from the typical container run times is that it's ocean application
kernel.
What it does, it's very interesting, it intercepts system calls, and it re-implimentant
or it re-implimentant, and does that in Go, and therefore gives the appearance of a
kernel API when there is no such thing to the containerized process.
Explaining what the advisor does is probably requires a talk of its own, so I'm going to do something
different.
I don't have the time to explain, and I don't have the expertise to explain, because
it's a very complicated project, and I'm trying to show you what I mean.
Let's take a simple thing that most container run times do.
Let's say you have a shell in a container, and you want to open a document and read
its contents.
That's a pretty simple thing, you do podman exec, took exec, and you do it all the time.
You assume, and this app does happen with the advisor, that you'll see the file system
hierarchy, and you read the contents of the document, and all that.
What's interesting here is that every word that you see here, every graphic is incorrect.
It doesn't up to the reality, and let's start with the on the file system part.
There is actually nothing that the advisor sandbox can access.
Practically, there is only a slush prok there, it's not even a full slush prok, there are
some things that are missing, so the containerized process.
How can it open a document, that breaks the question, how can it do that?
Well, opening is also like, that does not happen, there is actually a second filter
that this allows the open system code.
What does happen is that the container is a divider, intercepts the open system code.
It re-implements it with an IPC code to a different sandbox, where a go-for process run
with a part of divider, and in that sandbox, it opens a document, and, in fact, it cannot
even read it or write it, it can just open it, it can pass back the file descriptor to
the caller, and that caller can read or write it.
So in practice, if you manage to point any of those two sandboxes, you cannot interact
in the file system, you have to point both, let's say.
In just for kicks, you're seeing a process here, that's also like, that's not the case,
the divider sandbox, there's not even have exactly, it's a knight actually, the exact
system code.
It can only fork stuff, so basically there's a goal, there's a goal team, which kind
of runs the code of the cell, what we're seeing here, using long jump and all that.
So yeah, you can see that divider takes some serious steps in order to try to isolate
stuff as much as possible, and we're very interested in trying that out.
Problem is, we are users, or journalists, or activists, and they are on macOS and Windows
predominantly, there's no that much Linux involved, and also even on Linux, using divider
requires some manual installation steps, and we can't have any of that, we can't have
extra installation steps for our users.
So we want to use divider, it's available on Linux, if only there was a way to use divider
on Windows and macOS, and that got us thinking, what if we could containerize divider?
If we could containerize divider, we could run it in our Docker desktop VM that are users
run, it's a prerequisite, notice you run our, in order to run the divider zone.
And that, in turn, this container could create a nested container, which could run the
actual code.
And that's what we did.
Don't mind not the right side of these slides, it's just the code for those who want
to try at some point.
So essentially, what I wanted to show you here is how easy it kind of was to do it,
well, easy, painless, let's say.
We, what we did, is that created, that we created two container images, basically.
We took the original container image from our Docker file with no modification, and we added
it as a multi-stage build to a new layer.
In other end, in that layer, we installed divider.
So there's just divider and that container image there.
We did this, in order to create two container images with two different purposes.
The outer container image will be the one that contains our logic and contains divider
as well.
It's basically the possibility layer in order to provision divider to our Windows and
Mac OS users.
The inner container image is where the actual fun happens.
The divider runs this container, and it's where our logic sanitizes the documents.
Now, how do we run the inner container image?
We have an entry point script, which takes the command that you want to run, and creates
generates an oceanic config on the fly.
It drops everything.
It drops our ability, it drops, it runs as an previous user, it masks some system
paths.
If you want to, it drops resource limits, it makes it make the root of the first
re-dominant node up.
And then we run divider, rootless, within our outer container image.
And finally, how do we run all those things?
How can the danger of an application run this call, bag of things?
So we have, we run, we run Python.
So what we do is that we call Docker.
And we call Docker.
We follow the mitigations I have mentioned before.
So we drop privileges.
We have, we drop abilities and all that.
However, in order to run divider, one would expect that we will need to add a few extra
things.
It turns out, we need to add just a tiny bit more.
We need to add the sisterhood capability, which is added by default, actually, in most
containers.
The P3 system called the second profile, which is again added by default nowadays, it
was only for like four years ago, it was not there.
And we set the container identity as a Linux label.
Fun fact, this means that this whole, crazy thing, this whole nest, the container thing, actually
runs its systems with S. Linux and forced, divider cannot run in S. Linux and forced systems.
They do not have support for that.
But by doing this thing, we can essentially do it.
And that's the final architecture.
We now have an isolation layer between our application and the kernel.
Now, if an attacker wants to escape to the host, they need to find a flow in divider and
they need to find a flow in the Linux kernel.
So they can do that, sure.
But it raises significantly the barrier to do that.
It raises the cost to do that.
You have to burn the exploits that are pretty huge, because currently there's no divider
expert out there.
So yeah, thanks for listening.
On the right there are ways to get involved, if you want to, divider actually integration
was from a user of ours, and yeah, that's all.
I'm happy to hear any questions as you may have.
Thank you, any questions?
Hello.
Thanks for your presentation.
You said it was a desktop application, so you need to manage the UI, how do you manage
easily the UI with the, as it's containers?
Our users, if you haven't heard the question, it's how you have a way to manage documents.
We have the UI, how do you see the documents?
The containers, we have the UI, I think that's a question, right?
So again, our users are not technical, so we do not show anything about the containers.
We hide this part that they're even used, actually.
What they do know is that they need to open Docker desktop and have it sit there, and
we do everything other food, so they are not aware of something like that.
Another question?
A question that I'm having is, you said that you parse the documents to take it there,
malicious, and then you return them as pixels, and you transfer them back to documents.
How do you not intercept with the original bytes that were, how do you turn them back to
the original bytes?
Or you're just transfer anything to optical for the original documents?
Yeah, good question, so the question is, by doing that by making a printout, let's say
of the document, and then scanning it, how can you return to the original format?
The answer is that we can't, this is a destructive process, we cannot trust anything
in the document, and there's no easy way to reconstruct that format aside for the OCR that
I mentioned, so yeah, we have still time for more questions if they're running.
So you said this is as close as possible to VM level parity for security, why not just use
a VM, like what advantage does do it all this stuff give you?
Good day, question, I was hoping for that, actually.
Kind of use a micro VM, that would be my question.
First of all, we do use VMs, Docker desktop is a VM, let's say.
So we do have that.
Docker desktop does not contain that well, because it has access to the network by default
and it has access to the host file system, it needs to be in order to mount files.
So we don't consider it a VM for the isolation pair, which is what we care about.
So why not a micro VM?
Personally, I had searched about micro VMs that could be used in Windows, in MacOS, and various
flavors of Linux, and to my experience, micro VMs are some of those, I think one, pretty
no one was available in MacOS, but none that was available in all those operating systems
sent, supporting that many flavors of Linux, some micro VMs, and some other micro VMs
that would be very difficult for us.
We are very small team, you know, to do that, so that's the answer.
If someone knows some platform agnostic, and promoted platform micro VM, that could help us,
I'll be your, I am all ears.
Yes, is it possible to find the slides online?
That's my first question.
Yes, my second question, I didn't really catch how the user get back the output of the, let's
say the pixels or the image, do they get it as a file back, or like, is it shown in the application,
itself in the UI, or yes, okay, so two questions for those in the stream, if we can get the slides
online, I will upload them soon after the talk, and add the Google talk link soon.
As for the output, the user gets back, they do not see pixels, of course, it will make sense,
but the user gets back a PDF in the file system, they choose a suffix, and the PDF will be created there,
it will be called a safe PDF, and that PDF will be the optical thing of the original PDF.
The original PDF in order to not accidentally click on it, because we have to think,
security is very good, now that we can add the output as much as we want, but people will click on
the thing by mistake, so we move it to an unsafe file, we call it unsafe, that's the answer.
I have a question, if I understand the workflow correctly, your UI application ensures that the
Docker pulls the current image of your container and for processing the documents, how often do
you release those containers when there is some bug?
Right, so currently, the question is, if there is a Docker pool, we have an image out,
how often do we do these releases? It's worse than that, there's no Docker pool in both
actually. That's a baguette from the first versions of the engine zone. The container
image is actually baked within the application, so we don't only have to create a container image,
we also have to see a whole new application to everyone. How often do we do it? We try to do it
every two months or something, and if there's a CV that we detect, as we have done in 24th of December,
previous year, we will do a release then on 24th of December for crying out loud.
Okay, thank you, we're out of time, and thank you very much. Thanks a lot.
No, thanks a lot.