All right, time to go.
All right.
Next up, graph vision runner, talking to us about the VFS.
Hello.
Yeah.
It's a microphone zone.
I'm Christian.
I'm one of the VFS maintainers.
And, uh,
I don't know.
James?
No.
No more fips, please.
And I'm one of the VFS maintainers.
And I want to talk a bit about what we've done.
I think it's just a few selected updates of what we have done in the last year, I think.
And this really just a small portion.
And just deals with, like, I guess, a subset of stuff that we have done that is UAPI visible.
Um, there is obviously, as we just heard in a talk before, there's always a ton of changes.
Internal changes going on to make stuff faster.
We work locking and so on.
But that's just not what we're going to be concerned about with yet.
So, I have a, I really have a bunch of topics.
It's not possible to do it.
It's all in 30 minutes.
So, if you have specific things that you are interested in, um, we can also skip over stuff and go, um,
and talk about other things.
And you should also feel free to ask questions if you have any while I'm talking.
It's usually more interactive that way.
By the way, one of the big things is obviously, uh, we have a new Mount API.
It's new in the sense that it came out in seven or eight years ago.
And, uh, parts of user space of switch to it.
Other parts haven't, uh, but it's seeing more and more adoption.
And the really nice thing is that it's not path based.
It's purely file descriptive based, which is kind of neat because paths aren't safe.
Um, and, uh, this way you don't have to have anything visible in the file system.
You can just interact with mounts via file descriptors.
They don't need to be anywhere in your current mount name space.
Um, one thing that we did, that's a while ago.
We introduced, uh, the concept of a 64-bit mount ID.
So that you can uniquely identify, um, each mount.
So before that, we had a 32-bit mount ID and on a busy system where you have lots and lots of containers.
Uh, these mount IDs can quickly be recycled, not just because, like, eventually, you will wrap around.
That's one possible concern.
The other thing is that the way these mount IDs were allocated was like,
if you unmounted a mount that is a specific mount ID, you created another mount.
And it was very likely that it had the same mount ID that the previous mount.
Because, like, the allocator quickly recycled.
For example, the pit number allocator is such that it wraps around, like, before it actually starts recycling.
And this old mount ID allocator always, like, quickly recycled mount IDs.
We changed that.
We gave it a 64-bit mount ID to make it stable and more predictable.
Um, and, uh, there is now an API that is built around this.
Another thing is if you wanted to watch for mount notifications,
you had to pass a specific file.
You had to get to that as well.
We have replaced that part of the API in recent years as well.
So, okay, what have we done?
Um, we added a new system call called step-mount.
And you can think of it as, like, a companion to static in a way,
only that a it doesn't work on file descript as well.
That's technically not true anymore.
It works on 64-bit mount IDs, but for various reasons,
we also made it possible now that you can use FDs with these system calls as well.
Because on the certain circumstances, you cannot look up amount anymore.
I won't go into details, but, like, an unmounted mount,
you can't find by its mount ID anymore,
because it has been wiped from all mount name spaces.
But in any case, 64-bit mount IDs.
So, you call step-mount on a mount ID that you can retrieve that via
statics.
That's where we expose the mount ID.
You can call step-mount and then you get a whole range of information
about this mount.
There's a ton of stuff in there.
You get, like, the super block mount options.
You get a mount option string, which is, like,
all the file system specific mount options.
You can retrieve, like, the generic mount flags.
Obviously, you get the mount points.
You get the root of the mount, which can be different.
If you have a bind mount, for example, and the ton of other stuff as well.
We also enhance the mount name spaces themselves.
They also have a 64-bit mount ID, which means you step-mount
also works across mount name spaces.
So, if you have, if you retrieve the mount ID of a specific mount name space,
in the container, you don't need to be located in a mount name space
of that specific container.
You can call step-mount in a particular mount name space from outside.
Which is quite powerful.
So, because, like, traditionally, interactions with the mount name space
is specifically always required you to be, within that namespace,
to be able to see mount properties and so on.
We don't require that anymore with the new mount API.
The companions system call.
So, here you can see, hopefully, somewhat readable.
Probably not in the back, but you will have to slide to available.
What kind of options you can retrieve by other set mount system call.
It's really extensive.
Like, everything that you could possibly want.
I think it's nowadays even more than what you can see in the mount info file.
You get to that in a second.
The second system call that we added is a list mount,
because step mount is obviously useful.
Like, you can set a specific mount ID,
but obviously you want the ability to iterate through a list of mounts efficiently.
If you implement something like that, you have different ways of doing this, right?
So, you could, for example, envision a way of implementing something like list mount.
You take a mount ID, you call a system call that gives you back the next ID,
if it's sequential somehow, and then you keep iterating.
The problem is that doesn't scale, right?
Because the kernel needs to do a lot of work.
It has to allocate a file struct in the kernel.
It has to install that if the file is scripted or table.
It has to return it if the user space.
Then you can call a step mount on it.
Then you close it, file is scripted.
Then you get the next one out and the next one out.
You can make this more efficient by returning an array of file descriptors.
That's kind of wacky in my opinion.
You can do this, get-then-smodel.
The get-then-smodel needs to keep kernel internal state.
So, you retrieve names, which for mounts also doesn't make a lot of sense.
You could probably come up with an API of that, but I don't like the fact that you have to have,
like, need to have a cursor and a kernel, and to remember where you actually were.
With 64 bit non-id is pretty trivial, pretty trivial.
You just give it like the 64-but non-id of the mount that you want,
and then it gives you the children of that mount.
Specifically, the next mount number, because in the kernel internal,
it's kept in an RB tree in a red-backed directory.
Anyway, the gist is via this mechanism.
You can very quickly iterate through a list of mounts in a mount namespace.
By default, in the mount namespace that you currently located in,
so if you're in the initial mount namespace, you lock them to your computer,
you list it, you get, you can iterate through the mount table of the system.
But, same as with set mount, we made it possible so that you can take the idea of a mount namespace,
and then you can say, give me the mounts in that mount namespace of that container,
and then you can iterate from outside through all of the mounts inside of a container.
The really nice thing about this is, because you can also iterate through all mount namespaces,
that's a separate API.
We're not going to be able to cover here, but essentially,
via these two mechanisms, it's possible that you can iterate through all mounts on the system,
like literally all mounts.
I mean, if you wanted to do this right now, what you have to do is like,
iterate through Procophus, I guess.
Look at all of the processes that are running on the system.
Look at the mount info file, then figure out whether it is a separate mount namespace,
and then you can start listing all of them on super inefficient.
With this, you can just iterate through all of the mounts on the whole system.
You can literally dump everything.
It wasn't efficiently possible before.
I've linked to a program that does this.
Yeah, I think it's in the kernel tree.
You can look at this, it's not, let's say.
I'm very rigid with my kernel code.
I'm sloppy with my testing code.
It's not pretty, but it gets a job done.
You can look at it, and then you can implement this for yourself, for example.
But I think, eventually, the mount binary that you use to mount stuff on your system,
nowadays uses the new mount API, and it's very quick at adapting new features,
and eventually you will have the same thing that you can just use that binary to list all
mounts of the system.
But why did we do this?
Like, okay, the one motivation for this was obviously you wanted to make it easy
to retrieve information about mount, without relying on mount info,
but you could make the argument why the hell did you invent like this whole new API?
Why did you just use the mount info file?
You extended it if you wanted to display more information on it.
Well, okay.
System D people are here.
They can probably tell you a thing or two about the mount info file,
but it's actually just doesn't scale.
Like, okay, text parsing user space, you can do that.
That's probably fine.
But for example, you cannot have difts.
Let's say you have a mount or a U mount event, then the mount info file will generate an
notification that user space can listen on, but it's not specific.
It just informs you that something has changed, which means user space is responsible
for passing that whole file and figuring out what exactly changed
with regards to the previous version.
And changes like nowadays, people run 10,000s of containers on a system sometimes.
So, and some container, not runtime, some container,
I guess orchestrators do stuff like, for example,
where they stuff all of the mount in the initial mount name space,
meaning they all show up in the host mount name space,
which means system V on a system constantly gets spawned with
a spanned with mount events.
Technically, it's not very interested in.
But the mount table constantly sees updates updates,
updates, system V is just busy catching up or whatever mount tool you use.
So, that was something that we wanted to get away from.
We wanted to have a mechanism where you can catch up in case you missed events.
You get an indication when you actually missed events,
and it should be way more efficient.
Then what you currently have and not be based on a mount info file.
So, what we used to do or what we have implemented nowadays
is an extension to phenodify.
Because phenodify has basically an notification mechanism.
It's nowadays also available on privilege.
It has all of the really nice handling of dealing with overflows of the queue.
It has like a reliable reporting mechanism of events
and so on.
And so we didn't want to reinvent everything.
I think it was Army's idea back in the days that we used phenodify for this.
And so nowadays, what you can do with phenodify,
you can take a mount namespace file descriptor.
It's easy to get as exposed in Proc.
You can get it via a pity nowadays.
We get to that in a second as well.
Depending on how much time we have.
And then you can use that mount namespace file descriptor.
Registrate with phenodify and then you can via the phenodify API.
Listen for events.
And every time a mount event changes,
you get an event for example for a specific mount.
Sorry for a specific mount namespace ID.
Then you can use that mount to get information about that mount.
What exactly changed for example.
And then you can also use the list mount system called to figure out if any of the child
mounts have changed.
You get an indication if you missed any events.
For example, the queue that overflowed because there were too many mount events.
In that case, you just need to restart.
And like we passed a whole mount name mount namespace table.
But this is way more efficient.
I think the initial testing that we saw is like orders of magnitude.
Faster and more reliable than the mount info file.
So I'm not quite sure if the system is already switched to that,
but that's going to be a future way of it.
You probably wait on lip mount, right?
Lip mount is ready.
We just have a new lazy process.
Anyway, that should get rid of a bunch of bottlenecks in system,
do you hopefully.
But obviously you can also make use of this for example.
And if you have a container manager that manages a single container,
it's usually how they implement it with run C,
but with other container run times.
Then that could register the mount namespace as well.
And register for mount events inside of that mount namespace as well.
If you wanted to get really, really crazy,
there was one of the original ideas that we once had is you could build,
like use a mechanism like this to supervise mount events,
but probably not something that we're going to need anymore in the future.
But anyway, mount notifications and the mount may be API.
So system B and due to the length of probably going to be the first users
of this lip mount is ready.
It already uses this.
System B is being lazy.
But hopefully we will get them very soon.
So in the interest of time,
because I talked about mount for quite a bit.
So I have a bunch of mount related topics that could be interesting.
I have a bunch of pitifice related topics, so pitifties.
And API did we build as well.
And I have some very new recent features that we've been developing just for this cycle.
So is there any preference on what we want to talk about pitifties for the mount stuff?
Anywhere stuff?
Okay.
So we'll skip over a bunch of stuff.
Okay.
So this is very recent.
When you create mount name spaces in a container,
I think this is going to be pretty interesting.
I actually could do a demo for this.
But one of the problems is nowadays when you spawn a container
and you create a new mount name space,
it's probably the first thing that you need to do.
Like if you don't use any other main spaces at all,
you're probably going to use the mount name space simply because like you don't want all of those
mounts to leak onto the host, right?
It should be live in it's separate world.
The problem with this is every time you do the system call is called
Unshare Clone USS.
It's called Clone UNS.
And there is a binary that you probably use in a shell.
Unshare dash dash mount for example.
The problem with this is it clones the whole mount table.
Like everything that you have.
If on your host system you have 30 mounts,
probably not that big of a deal.
If you have 1,000 or 2,000 or 3,000 mounts,
then the kernel copies 3,000 mounts into a new mount name space.
And then when you spawn a container,
you usually don't want to lose these 3,000 mounts on the host, right?
The point is that you isolate the container from the host.
You don't want it access to all of the crap the host has available.
So what you usually do is you set up a new root of est
for the container somehow,
barit container something usually.
And then you switch out the current root of est in that name space
that new root of est.
The container is supposed to use.
Container D dust is.
And run C dust is.
Inca dust is.
The community dust is like everything.
Everything dust is the thing is.
It gets rid of all of the mounts that you copied before.
So you copied 2000 mounts for completely no reason at all.
The problem with this is not just that.
This is in performant.
The problem is that the way the kernel is constructed currently.
Because like we have relationships between different mount names spaces,
they can propagate mounts to each other.
So that you can share stuff that is one global,
one specific variable block that we can't easily get rid of.
So that means you keep blocking on that log,
waiting for someone to unshare mount names space with like thousands of mounts
that you're never going to use completely useless, right?
So we thought, okay, you idea how to get this,
how to get this done.
So we don't have to do this anymore.
And this is the new API for this.
It's like open tree name space and FSMount name space.
That works with two new system calls we have in the kernel.
And what they effectively do is you can specify a path.
You can specify a path or a file descriptor.
I should say.
And when you specify that for open tree,
the name space, if you specify that file descriptor or that flag,
you pass that flag.
Then the kernel will create a copy of that specific mount or mount tree.
And then create a new mount name space and mount it.
Place it as the root of the root of that mount name space,
which means you just copy exactly the mount or the number of mounts
that you need for the container.
And FSMount is a similar concept.
Only that FSMount deals with actually mounting a file system.
So open tree clones.
It's like binding mounting.
It clones an existing mount tree.
And FSMount and the associated API with this.
It creates a new file system mount.
So the stuff that you do when you say, I want to mount a new XFS file system.
Or a new uRof file system or whatever.
And it's the same concept.
Like you mount that.
And then it becomes the root of a new mount name space.
I can try and show you what I mean.
Unprepared life.
I will say me prepared life then also.
Which may look.
There it is.
Okay, you can somewhat see this.
I'll give you a second.
I'm sorry.
Okay.
This is just a VM.
Actually, I already created the demo folder.
And here I have a static busy box binary in busy box.
Nonodynamic executable.
So if you do a traditional way, right?
Unsharing a mount name space.
You do pseudo, unshared, dashed, shmount.
You can see if you do cat prox, self mount info.
All the stuff that you had on the host has been copied into that mount name space.
And now you can switch out a new root of S in here.
And then you could technically start your container.
So I have a program here.
There is also not very pretty.
I'm going to be sure if I have a winner on here.
But what exactly does is just what I talked about like open free namespace.
Here you can see it.
It's like a new flag.
It defines the system call.
You can give it a path.
It calls open free, which creates a detached mount.
So it creates effectively a bind mount.
Not visible anywhere in your current mount name space.
Then it returns a file descriptor to a mount name space.
You set a nest into that mount name space and then it tries to execute a busy box binary.
That's all that it does.
So we can create a directory somewhere.
Let's say here, for the sake of this like a demo.
And then I do a bin busy box.
And move that into demo.
And then I do ignoring a user nest from now demo.
Okay, there's nothing in there.
Other than it.
Busy box binary.
I could do make the a proc.
Oh god.
A busy box mount, right?
A proc.
Proc.
Sush.
Proc.
Okay.
And then I can do cat proc.
Not info.
Ah.
What's all you have in there?
Thank you.
If we port various thank you, I'm surprised this worked.
But okay, I'll take it.
And we have a few more minutes, actually.
So yeah, if once you've switched your container runtime over to this, there's some limits to this.
That we don't need to get attacked.
Don't have time to get into.
But if you switch your container runtime over to this, you will probably get a lot of performance benefits out of it in terms of scaling.
Because you're not wasting cyclists anymore for no good reason.
Yes.
So to make use of this in your actual container runtime, you're going to have to switch your entire
mount setup.
Yes.
To have these, right?
Yes.
Because otherwise you have to shut up my namespace before.
If you want the goodies, if you want the goodies, then you need to do some work.
So the thing is that's what I said before, then the mount API is completely FD based.
And traditionally like you had a single system called mount, right?
And mount took paths.
You could technically use file descriptors.
If you use the trick where you did proc self FD and then past it, the FD this way to mount.
But the new mount API only gives you the option to work with file descriptors.
I've got maybe a dumb question.
But why wasn't that at its clone 3?
So if anything, I would like to clone 3, by the way, this is the mount.
The mountable I would like to have as part of the clone.
Yeah.
You could see that I could just API wise.
I find it nice in the actual FS APIs.
But you could actually do it twice.
And you're not in the first need to set up the mount namespace.
And then still do your clone for all of the other ones.
All do all of the engineers separately.
You could extend clone 3 with that as well.
Like if where it makes sense.
But most of the setups, most of the setups that I've seen.
Even sometimes in Lexi, we get away with unsharing, with cloning it right away.
But like, for example, most stuff that I've seen.
Obviously I have a system D centric eye in a sense, oftentimes.
You're just unsure. You're just always unsure because it gives you more control.
What is owned by what and the ordering.
But there is certainly not necessarily something that would prevent us from doing something like that.
Cloning 3 is accessible.
So sure.
Because I could see that that's being a good security measure.
Some people too.
Particularly, you know, clones 3.
And by the way, I don't need to fix these them.
So don't bother.
So one of the last things that I'm going to cover.
Very quickly or not.
Is I want to say that the nice thing about this was clone 3 is that it would allow you.
We would need to do more extensions for it.
We'd cost it before.
I can't hear you.
We would need to do more extensions for this.
But the nice thing about doing it with the amount of these is that if you can.
If you can mount on supplement method, you can construct the entire amount tree as a thing.
And then it's heading us into it.
And with the clone 3 thing.
You're stuck in the situation where you don't have extra amount.
Which means you have to come work around for getting stuff like you can't mount proc.
Because no proc is visible.
And you have to deal with the crap, which is like.
I'm sure that it would make sense to do it in the case where you know you can figure that out.
But for.
I'd like this API.
I would buy it.
I'd prefer this one.
So last piece.
Is nullFS.
So when you boot on a.
When you boot on a regular kernel, then the first thing that the kernel has is like the real rootFS.
You would like to call it like the terminology is confusing.
Real rootFS.
I mean the thing, the rootFS that you see when you do LSAL is mounted on.
That serves as the parent mount.
The kernel like uses this in dual ways though.
It's first of all.
It uses it to handle the inner RAMFS case.
So when you specify an inner RAMFS, the kernel either creates.
A mount.
Real rootFS that is a tempFS or a RAMFS.
But itself that mount doesn't have a parent meaning you cannot unmount it.
Meaning you unpack the inner RAMFS into that tempFS or RAMFS.
You then have a bunch of crap in there.
And so now you have two options.
You mount the rootFS.
You pivot root into the rootFS and you leave everything.
That is in the inner RAMFS under there.
That's performance wise, kind of nice.
The problem is anyone who somehow manages to get the hands on the real rootFS will see whatever you unpacked in there.
So if you have any specific sensitive binary or something, then that will remain there.
So usually what any system implementations do is that they do an RMRF and empty everything from the inner RAMFS, then mount the real rootFS and then fire away.
Obviously it doesn't make it just wasted effort, right?
So it also means you can pivot root because like, so pivot root is the system also which is memory.
You're also wasting memory.
But you're also wasting memory by doing that.
Yeah, you're wasting memory.
But anyway, the pivot root system called switches, which is a rootFS out underneath you, right?
And so you can do this because it requires that the thing is already a mount point.
Even if you bind the inner RAMFS onto itself, it still wouldn't help you because you'd still leave everything in that directory.
So nullFS essentially is a concept to say, do away with this, all that we really want is something that serves as a mount point later for the real rootFS.
And nullFS is like completely catatonic, catatonic in a sense that you can create anything in there.
You can create anything in there, you can change mount, options you can do anything meaningful with it all.
It's only purpose is to serve as a mount point, which also means the count on mount a tempFS, a slash RAMFS on there and you unpack a bunch of crap in there that you need.
And then pivot root suddenly works.
You can just switch it out, you don't need to do RMRF anymore.
The kernel will just do away with everything that is in that tempFS when you own mount it.
You have your real rootFS, you have your rootFS on there, and then you just boot.
Also works if you have an inner ID, but inner ID systems are fairly rare nowadays.
The other nice thing is that nowadays we have the concept in mount name spaces where you lock the rootFS mount on top of the real rootFS simply because of the thing that I said before.
There could be stuff in there because it's a tempFS, or people could create stuff in there if they got access to it.
So it's locked, you can unmount it, it makes a couple of things more brittle.
If you have anotherFS, you know always empty, nothing in there doesn't really matter if you can unmount it.
So you can unlock it, it makes a bunch of handling in the kernel easier as well.
Okay, so there's a bunch more I could talk about.
I'm hoping hopefully I was understandable, and it was interesting.
And I hope you have some interesting questions.
How do you get this slide?
I'm uploading them after the talk, I'm sorry, I probably forgot this.
So usually on first them, you can get this slide for any of the talks directly from schedule.
Another fast stuff is by default, right? I don't have to open.
And so the preference was to make it, people were to make it unconditional.
I had the option, I also added the option to make it conditional.
You can use it like this causes any issues, and we could make it a command line option.
But we don't need to do anything in system need to make.
No, system need does everything correctly, it pivot rules.
It tries pivot rules first, and if that doesn't work, it just moves once, but even if you don't do that,
even if you say you always rely on the fact that you don't need anything at all,
and you said I can't pivot rule, then all that happens is that your root of this has another parent mount,
but it doesn't matter, you can't unmount it, you can't get to it.
It's like irrelevant.
Thank you.
You're doing like I choose your own adventure talk.
I'll take the mic.
No, I know.
I know.
Let's start.