Alex, are you ready?
Yeah, I'm ready.
OK.
Our next talk is by Alexa.
It's right.
And it's about half safety in the trenches.
Yes.
So hello, everyone.
It's lovely to be back at Falston.
I haven't been for quite a few years, because I'm stuck on the walls.
Pinocola, you may know, called Australia.
But yeah, I'll be talking quickly about, relatively quickly, about some work we did in
on C recently.
But it affects a lot of projects, but it's kind of, particularly, an ACN4
currency.
And I'm also at this new company that we've learned, called Amidall.
So run C, I'm sure all of you know, what run C is.
This is going to be a very, very quick battle for these slides.
So run C, as you may sure you all know, is sort of like the entry point
to doing most operations that are capable of specific on the next.
And basically the entire OCA system runs to run C or run C like programs.
And run C has this, you know, CI.
We have this specification, which defines this particular JSON structure, which you specify
what kind of container options you like.
I'm sure all of you are saying it's before.
And with the JSON file, you do run C, run and you can run containers.
All of a reason.
So the thing I'm going to talk about today is path safety.
And so I suspect most of you have actually already run into what I call regular
path safety in this talk with some interesting bits.
But the bit about street path safety is something which I've been working.
I've been working for quite a few years and is one something that is not obvious
when you first run into it.
And it's very, very frustrating when you have to fix bugs related to it.
So regular path safety is like the very classic time of check time.
I've used a tack that you have with any kind of file system operation.
Paths are just strings.
So when you operate on a path, the path can get swapped out from underneath you.
This is the classic.
I want to sanitize the path problem and you can't sanitize paths because it is
not as the path.
So I'm going to change it after you sanitize it.
And so like the classic example, there are lots of classic examples you can have.
But you know, you have some slash root of first, which is owned by an untrusted process.
They can mess around with any of the files in there.
So if you have an administrative process on the host that is like a container
on time, that is trying to operate on these things.
You can get tricky and opening random files that doesn't expect.
And I want to point out a lot of people think this own-o-follow thing protects you.
It doesn't own-o-follow is extremely limited in whatever texture against.
And in a particular in this case, if Etsy is a similar to somewhere on the host,
it doesn't matter if host is not a similar because own-o-follow only stops host
from being a similar, not anything else.
And basically, almost all of the classic VFS APIs have this problem.
So you would think this problem has existed since you existed before.
You know, this is a problem that doesn't happen very often.
So this is a list of CVs, which are in the ones that I could think of when I was making this slide.
You could easily make five slides or six slides that are just lists of CVs or this kind of problem.
It is an incredibly, incredibly common problem even today.
And so this is really not a new problem, but we still keep running into it.
So in 2019, I worked on this thing called Open Add 2, which is a system called
on Linux.
It's no longer new.
It's quite old now.
But what this is allowed you to do is in addition to the regular Open Add 2 phases,
which lets you operate on a file repository, like directory files purpose.
You also have this resolution flag thing, so the result thing, which lets you control
resolution.
And so what this allows is it allows you to control aspects of how the entire path is
resolved.
And so I would say the vast majority of programs and Linux should be using at least one of these options.
So either you want to use Resolve InRoot, which is a kind of like CHRoot, but it only applies
for that particular Open Call, and it adds some additional protection to CHRoot doesn't have.
Resolve Beneath is kind of like CHRoot, except if you try to jump out of the route you get
an error rather than it being scope, like a CHRoot or scoped.
This is a free PSD kind of API that we, that actually found enough.
We, I took inspiration from the previous API, and then they took inspiration from the Linux API,
for this.
But yeah.
And then Resolve No Simlings is basically better owner-follow, so it actually blocks
Simlings for any path component.
You can still escape with absolute parts, which is why you want to do more than that.
But these are incredibly useful things, and I find myself using them all the time now.
The one thing though is that OpenAt only solve one problem, which is opening files.
Now, in terms of this, it's actually sufficient for most programs, because Linux has these,
and not just Linux, they can cause X2, has these app system calls like
make, make, make, direct, and, you know, link at and so on and so on,
which lets you operate relative to a file descriptor.
And so, as long as programs are written to primarily operate on file descriptors,
not paths, you can actually protect against these issues, basically entirely.
So, here's the example, one of the examples from before, but using the OpenAt2 and the path
based off.
So, for Open Operations, you just, rather than using, so you open the root of the first.
This is a trusted direct, it's directly constructed, because it's in,
the whole, it's wrong by the host, and then you use Resolve in root,
and then you just use regular OpenFlags, like you would have anything else,
and this protects you against these kinds of race attacks.
The more complicated case, this is just one example.
The more complicated cases, if you wanted something like make, dear,
because make did doesn't take results, what you need to do is you need to open the parent directory,
and then underneath it, you just open underneath it.
This is safe, because make their act does not follow trail links and links,
and because there's no slashes in the path, this is safe through.
And you then do this for every single, you know, act system pull,
and every VFS thing you're interested in.
For all the kernels, which when I first started working on this,
this was a more pressing issue.
These days, all the kernels means like, you know, seven-year-old kernel,
which, some of you were running, but less than nowadays.
Though, it also should be noted, because OpenAt2 used this structure thing.
Some systems block it with second system, D blocks it with second for its own usage.
And some people block it with second, because second currently can't filter
arguments, so effectively, they don't trust that we might add a backdoor to it.
Like, not backdoor, but like, a problem with it in the future that they couldn't block.
Anyway, point being that there is this old school way of doing it,
which is rather than using the kernels facility for doing these saved lookups,
you can use Opaf to like emulate lookups in user space.
This is quite finicky and a little bit annoying to get right.
But it is something you can do.
Some systems are done this in the past.
And the key thing is that this is basically swapping out the OpenAt2 bits.
The rest of your program still needs to use Fabricus for things.
And usually you end up having to use this Proxile FD thing,
which lets you, which is this thing called a magic link,
which I'll talk about in the little bit later,
but it depends on this for validating that you are out there as well as you want.
Yeah, system D does this in Chaser, for its own operations,
go last year added OS.ruth, which does this in a more complicated way,
and has a couple of limitations that I'm not a huge fan of.
And the Potheres, which is a library I wrote.
So the Potheres is a rust library I wrote,
which basically provides all of the main file system of file APIs,
but withness, within a root operation stuff.
And it's written in rust, but it has CFFI interface that you can use from going Python.
I had Python bindings.
And now the work I'm talking about here last year,
you may have heard we have some security issues in on C.
And so this work I've been working on for a while,
but you do the security update we couldn't just require
to use rust to fix the problem.
So I ended up porting the code, the code, the code you needed to go.
It's kind of a little bit funny because Potheres,
the Potheres started as a go library that I was going to use from C.
And then I thought, oh, other programs I need it, so I read it
to rust and I read it to go for this Potheres lighting.
So that was one of the way.
And the nice thing is that this library, it supports OpenI2 and OPoff,
and it's what's a bunch of other new kernel features,
but it transparently doesn't use them.
If it's not supported.
So this is the C API example, where it's rather than using OpenI2.
OpenI2, or whatever, you just use OpenRoot to get a root of date.
You can also use it.
It's all file, it's going to basically, you can just use a regular file,
because if you like.
And then yeah, Potheres in root do whatever.
Yeah, it's pretty, pretty easy to use.
And then in rust, I'm not going to go through something
to do it's all because I don't have a time,
but the rust API uses actual structures in rust,
but like the basic API is the exact same idea.
You have a handle, which is wrapped in a structure,
and then you can operate on it individually.
Yeah, if you want more information,
you can look at the docs for the Potheres.
Yeah.
So that was regular Potheres.
If the strict Poth, safety is more annoying.
So with previously, when we're looking at operating on random files,
you don't actually care.
As long as the file you end up operating on is within the root of us,
you don't really care which file it is.
Because an attacker can also swap the files around,
which we're assuming the person has privilege over the entire directory.
So they could replace the file with it of a content that they wanted anyway.
So it doesn't really matter.
As long as the files inside the container, you don't really matter.
It doesn't really matter to you which file you get.
Because you can't trust it.
It wasn't the anyway.
Because the attacker can write 25.
Unfortunately, this is not the case for strict Poth's safety.
So in on Linux, a lot of operations,
go through PROCFS, which is a file system.
And so the problem is with this,
that if you want to operate, for instance,
PROCFS will add our exact as how you specify what LSM labels apply to you.
So what Apple, I'm going to explain to the labels apply to you.
If you're writing to that file, you need to be really, really sure you're writing to that exact file.
You don't write to other PROCFS files, or then you're writing to some dummy file.
That doesn't do anything.
Because if you don't write to the right file, then you won't actually get LSM label applied to you.
And the same thing applies to most PROCFS files.
So yeah.
PROCFS is the most critical one.
And the key thing here is that if someone makes a PROCFS that is fake,
so it's a TEMPFS that has fake files in it,
or they mount on top of PROCFS from somewhere else.
They bind them out a different bit of PROCFS on top of it.
The PROCFS you care about.
You kind of operating on the wrong files.
And it's a bit, it's a big nightmare.
So an example of something which would be a strict Poth unsafety would be,
oh, I just opened PROCFS that are exact.
And I just write to it.
And so on as on.
So actually, every single one of these examples,
could both a security issue if someone have it no amount.
And I'll give it some examples in a little bit later of how that happens.
Now you think surely not that this is like not a real issue,
whatever.
But here are CVs that we had in run C for this.
There was a 21.
And actually this was what initially inspired me to work on the Pothress
and all this other work.
Because we had this issue and then I realized,
oh, this is actually much bigger issue than it seemed at first glance.
And then we had the issue from November last year that we also fixed.
So the core issue here is that now you would think,
okay, what can I do to fix this problem?
What I'll do is I'll open slash PROC.
And because PROCFS provides you the guarantee that the root I know of PROC
has a fixed I know number.
It's one.
And you can check with statuses that it is actually PROCFS.
Okay, fine, that's fine.
I will open slash PROC.
And I will check this.
And then I'll use OpenOut2 to make sure that I go open
that I go to self header exact.
And this result, noxative here is protecting you from Overmout.
So result, noxative rejects going through mount points.
So if you do this for regular files, this is all you need.
So this problem is basically already solved.
Very few programs do this properly, but this is the problem that's already solved.
So if self was mounted over, or add it was mounted over,
or exactly not over, you would reject the operation.
Unfortunately, this is not all you need.
You have these things called magic links.
So magic links are, they look like siblings.
And if you have a laptop in front of you, you can open,
look at PROC self-exy.
And it shows me a link to the binary.
But these are actually not siblings.
Similar links are dealt with lexically.
So in other words, you can imagine, it's as if you take the link,
and you replace it with a content, and then you continue to look up.
That's not how magic links work.
Magic links are a kernel concept, where basically all of them are in PROCFS.
Where when you do the open, rather than being resolved lexically,
the kernel knows what's kernel object this link prefers to.
And it will pipe you through to the underlying object without going through regular path look up.
And this has been very, this is very useful, because it allows you to make quite useful
security hardening for various things.
But I have a hand, it also lets you do container breakouts and stuff like that,
because this stuff does not care about main spaces.
So you can open anything with the magic link.
So to give an example, I want to open, reopen a file,
which can perhaps have open.
So I want to open PROC, Fred, self-FT, once and three.
And you would think, oh, this is totally fine.
I can do it because this is the exact same example as the regular file example before.
But unfortunately, resolve the external block this,
because when you try to open the magic link,
because this is in a different mount, most likely,
then it, because you go for PROCFS, some other mount,
resolve the external blocket, which is doing exactly what you wanted to do.
So what you might then say is, oh, okay, right,
I will open the FD directory, no X-dev,
and then I will validate there's nothing mounted on top of the sibling,
because if you didn't know this, you can actually mount top of siblings and Linux,
which just makes my life very fun.
And so you're like, oh, okay, that's fine.
I'll just validate there's no, there's no of amounts,
I'll just open the file.
Unfortunately, it's valid, no of amounts problem is actually unsafe,
because if you are dealing with an attacker,
they can change the mount table from underneath you,
then you can check it, nothing,
and it's, well, back to classic time, check time with you stacks.
Now, I will say, by the way, if there's example,
since Linux X-12 Christian, I mentioned to convince them to block this,
so you actually can't do this over a mount anymore,
for this particular case, but there are other cases where you can do it.
And besides me, you just solve these fold kernels anyway.
So, the solution comes in the form of the new mount API,
which was merged seven, eight years ago now.
And however, I will say, it has nem pages now.
For those of people who have dealt with using the mount mount API,
you had to like read this kernel sources
to figure out what the API looks like.
Last year, I wrote nem pages for it, so finally,
you can actually read the documentation for how to use this thing.
So hopefully, more people will get to use it.
But the key thing about this is that the new mount API
allows you to create file systems without having to mount them to a bigger directory.
And because it's a file descriptor, nothing else can mess with it.
And actually, there are some nice protections that protect other people from
like propagating mounts into it and so on and so on.
So once you have one of these, once you create a progress handle this way,
nothing else can interfere with it.
So, for this example before, we swap out the first bit with the open proc,
to you create a new progress instance.
This is the simple example in the more complicated example,
you need to have a full back for certain cases,
but this is like the basic example, where you create a proc of a sample,
which is a file descriptor to a directory like anything else.
But the nice thing is I think I'm overmounted.
And now this value, no overmount stuff is actually totally safe through
because nothing can overmount it other than your own process.
And so, this is the way you would do it.
And there are other things you have to do that are also kind of annoying,
but I made an API for this in Lepathras, so in Lepathras,
if you just say I want to open this proc of a file,
it will do all this awful stuff for you,
and you just get to open a proc of as the way you like.
And it works on new end old kernels,
and it works with additional protections for a bunch of other stuff
that I don't have time to get into.
And yeah, it also has a Rust API, which again,
I'm not going to need to know, but yeah, it's kind of really similar idea.
The one thing to notice is that,
yeah, you can make a proc of a handle that you can cache
for some usage and so on and so on.
Well, yes, so now some of you are probably thinking,
and I've been asked this every time I give a talk to talk to another,
this is like hang on.
Why do I care about any of this, right?
Like only root can mount, like why on earth,
would you care about someone mounting a top of a proc?
Well, unfortunately we're dealing with runc here.
And one of the cool problems we have with runc is that runc
actually does not currently have a threat model.
So, one of the problems we run into is that Kubernetes,
and Docker, and community, and podmen, etc., etc.,
they all like to provide users with lots of flexibility
but what they can configure.
And because we don't have a fixed threat model,
sometimes they allow users to configure things you wouldn't,
you wish they wouldn't configure.
And that's includes mounts,
because if you do a volume configuration for a container,
you've just configured a mount.
And so every single time, and we have to provide volumes to people.
So, this means that mounts are very much in the picture
of things that attack a control.
So, and also some people allow people to run containers,
you know, I'm probably going to be able to run containers
in their systems because they assume that the container
on time will block malicious things,
so we have to protect against these things.
This is something I want to work on,
and I've been speaking with some runc containers about doing
a threat model.
But at the moment there's no proper threat model,
and everything will vulnerability we have to do
on a case by case basis.
And the key thing to actually notice that,
because of this, a lot of vulnerabilities in the past
decade with runc, most of them have been
basically misconfiguration bugs.
But the problem is that is that a regular user
on a system using Docker, or community, or podmen,
or Kubernetes could misconfigure that way.
So, even though it was strictly speaking,
misconfiguration, we have to treat a security bug.
And one thing that really helps me
is that people still don't use this name spaces.
However, Kubernetes, I want to thank Rodrigo.
Kubernetes, in the next release of Kubernetes,
we'll have use name spaces like fully supportive
thing for containers.
So, hopefully soon, we will live in a world
who will ever use these name spaces
because it's crazy to be not used as an engineer
as containers, like it's completely insecure.
Anyway, I'm going to do very quick pop quiz.
This will have to be lightning run
because we're running out of time.
So, I want you to put your hands up.
Okay, how about this?
Put your hands up if you think that one
about to show you is or is not a vulnerability.
No, is a vulnerability.
So, is this a vulnerability?
Where I disable all the name spaces for a container?
This is the easy one.
The answer is no.
Because if you disable all security options,
obviously it's not a vulnerability.
Okay, I bind mount slash to my host file system
into the container.
Is this a vulnerability?
Yeah, some people say yes?
Okay, so the thing is that if Docker allows you,
so the problem is that for some people,
I mean, this is a bad idea.
I'm not saying this is a good thing to do, right?
But the problem is that this is a management decision
to make, or Kubernetes makes about what you can do.
From runcase perspective, this is just a misconfiguration.
Like, you can take any amount you like.
I can't tell us runcase what permissions you have,
what are the dimensions you have in Kubernetes,
or what are the dimensions you have in Kubernetes.
So, from runcase perspective, this is not a vulnerability.
Okay, how about this?
I have a bind mount to some sub-director.
This is a vulnerability.
It is a vulnerability.
And in fact, this is the two CDs we had in November.
That what I'm not telling you is that,
so, I just make sure we clear what's happening here
and that I'm buying something some directory
into some path slash vulnerability.
How can that be a vulnerability you ask?
Well, the problem is that volume is actually
a sibling to slash dev.
And there, you actually have two processes.
You have one process that is supporting this.
And then another process is now creating fun files
in some volume path.
For instance, you make dev null,
or dev PTSD, a zero, a sibling
to process core kernel pattern.
And what happens is that in runcase,
we have to mask certain things in progress
that people don't use this namespace.
So, you need to mask them and we mask them.
So, the first example, we mask them with dev null.
So, we bind now dev null on top of the thing.
But we don't use dev null from the host,
because there are security issues
with taking photos taken from the host.
So, we use dev null on the container.
We have done that in the container.
It is a similar to proxy kernel core pattern.
I've now bind mounted proxy kernel core pattern on top of
the files you like.
And proxy kernel core pattern lets you break out of containers
because what you can do is you can specify
a core dump helper, which is a binary,
that the kernel will then spawn
with full privileges on the host completely unnamed space.
And so, the similar problem applies to dev console
and dev PTSD as zero.
It's basically the exact same problem.
So, that's fun.
We did fix this, obviously.
So, one of the things we did was we added
much strength of validation of the ioners we use.
So, we now validate that dev null is actually dev null
in order to take things.
And it uses photosquefers to do the mouse.
So, it's safe against time-checking our use.
And it's basically all the mount points
creation code in runc.
It was moved over to lipothress or rather
pathress light, which is the go version.
And now, most of the, almost all of the path stuff
in runc is now a file is going to be based,
which fixes this and a bunch of other possible issues.
And yeah, it turns out, well,
you maybe you should care about regular path safety.
And for consoles, this is thing that I worked
on many years ago called GetPier.
I often older actually makes lot things a lot more safe.
If you're making consoles.
So, now, next one, it is vulnerability.
So, what I'm doing is I'm mounting some fake files
system or surface slash product.
This is vulnerability.
It is not a vulnerability.
And because, well, it's not vulnerability
because, well, in this case now that we've done this,
we actually blocked this.
But, well, it used to be, you know.
I would say if you had asked me two years ago
whether I would consider this a vulnerability,
I would say no.
I would say that Docker should block you from doing this.
In one scene, we have code to block
matching on Proc but like here.
It's, okay, maybe half, yes, half knows.
If you want to be technical about it.
And you would think, okay, is this vulnerability.
So, this is the exact.
So, I bind mount Proc one shed over Proc at or off.
I'm going to exact.
Okay, now Proc, for a bit of background,
Proc, pitch, a Proc any pitch shed is a file
which is just shows you like random information about a process.
But, you can write to it and it does nothing.
So, if you bind mount this file on top of Proc's of error,
it's a ProcFS file and actually in 2019,
the fix we had for, as I should say.
In 2019, when we had this security,
it looks last time.
The preliminary fix we had because I knew
that we didn't have this infrastructure yet.
The preliminary fix was, oh, I opened the file.
Is it a ProcFS file?
If it is a ProcFS file, then I will write to it.
The problem with this is that Proc1 shed is a ProcFS file.
It's just not the one you wanted, right?
And so, you end up writing sign to it.
It does nothing, and now the container starts with no app on it.
However, this is not a vulnerability,
because Runcy blocks this,
because we have a restriction of what you can mount at the top of Proc.
For LXC reasons, there are some Procs things you can mount on top of.
But you can't mount on top of what anything you like.
However, let's look at the slightly more complicated examples.
The key thing here is that you have two bind mounts,
where you have a bind mount of a food,
and you have a bind mount on top of a food,
with a very suspicious looking path name.
Link threads up that far, right?
Does anyone think this is a vulnerability?
Yes, this is another vulnerability.
Basically, what's happening here is that you have some cache directory,
which is shared between two containers.
It's the exact same attack as before.
Oh, yeah, he thinks this is a vulnerability.
Oh, no.
Five minutes, okay.
Okay, okay.
Yeah, so anyway, this is a vulnerability,
because you can have this Proc 10th cache,
one of the things is a fun simling to another target,
and you're going to make it to Proc's shed.
The best part is you can use a simling this to Proc's circuit trigger,
and turns out that XZEC, Docker default,
has a lot of fun characters to put into a circuit trigger,
including basically this crash is a machine if you do this.
So yeah, basically the fact that we were swapping between
this link and Proc is going to allow
to bypass this check, and as before,
Copa had an allows you to escape from container.
So the solution we came with was doing a showing the mounts.
Again, we should have switched the mount API earlier.
It's something I'm working on, but it's somewhat done yet.
And yeah, the live pathress, basically, for all Proc effects rights in the pathress,
we now use, sorry, all Proc effects rights in runc.
We now use the pathress for them, because it protects against these attacks completely.
And it's, yeah, and also we went through every right path in runc to make sure
that there was nothing you could put missed directs for a right, which you couldn't.
Okay, and runc, the things we need to do is we need to,
we really need to move our mount infrastructure to using the new out API.
It's really quite insane, we don't use the yet.
And yeah, there's somewhat what it's I want to do.
And also, live pathress exists in runc, because I use the Rust thing.
Today, you can actually build runc using live pathress,
because pathress light has live pathress back end.
But I think we, I want to get rid of this, like, intermediate go thing.
I just want to use the Rust thing.
Does then run and see, can use it in other programs, can use it.
And it's much more robust than the go version, because the go version is kind of
like to be a minimal thing.
The kernel stuff, we, I can skip over, because it doesn't matter.
There's some kind of stuff I want to do, not too important.
And the one thing I want to say if everyone in the room is, if you are writing any kind of program that
has to deal with paths, which is probably any kind of program, you should, at least,
use open-out to a live pathress for regular path operations.
And you definitely, if you're writing new programs, you should definitely write them in
a, you know, part of the previous way, because playing around with paths is just
waiting for a trouble.
And if you need to have any such prompt, you really, really, really really need to use the
address with the prox stuff, or make infinity yourself, you like, but you have to really do that.
And yeah, the key thing to keep in mind is that any support takes a path name is potentially dangerous,
especially if you're dealing with entrusted directories.
Okay.
Questions?
Three minute or questions.
Oh, thank you.
My apologies if I talked to you fast.
I think the smoke needs to clear from the speed.
I think there's our questions.
How do you feel about other projects, like the do most projects are a lot of projects
that deal with paths are aware of this?
Like, not just the no.
No.
So the problem is that most projects, so what I've defined is that most projects,
to be fair, run see was kind of in the same camp, but camp for this, is that they
write a version that works fine, and they've thought of like a couple of lossable attacks.
And then each time they run into one of these problems, they end up
hacking around that one particular instance rather than looking back and saying,
actually, we really need to rethink how we do this sort of stuff.
Yeah.
I will say one nice thing about that's in recent years is that the Linux
kernel has actually moved a lot more to file script based APIs.
And this is a huge, huge improvement to the way things work before.
I mean, everything is a file, everything is a file descriptor.
Yes.
All right.
Okay.
All right.
I think there's a few so much.
Thanks so much.
Thank you.