Background, I've mostly done file system work in recent years, but I've done some other
kernel stuff before.
So let's start by discussing why it is, we're building this thing to begin with.
The company will work for diversity, we provide a software stack that helps people manage
some of the biggest archives in the world, and it's an interesting space to be in.
We're going to focus on a post-exfiles system here, but we need to talk about how this
file systems use to understand what engine a file is trying to do because these file systems
are very weird.
So the general idea is that they treat this whole software stack as sort of a archive that's
after their main large data stores, right?
This is where all the files go to just sit at idle mostly, there's a lot of motion in and out.
But I've tried to lay out the sort of workflow here that shows why these file systems become
a challenge.
So the general model is they'll use the Pozix file system as just a buffer for files that
are going into and out of this archive media that's very irritating to work with.
So they'll have a lot of files they want to archive, and they'll dump it all into the Pozix
file system, which acts as sort of a cache, and they'll do this in these big batches.
What's interesting is that the files accumulate in the Pozix file system what we're going
to be talking about, and they sit there and they sort of, there's a policy engine that
will look at all those files and say, okay, these have all arrived, where do they go?
We'll put them off on these tapes, we'll talk about tape, or we'll put them off on these
object stores or whatever, but what's interesting for this file system is that once the
file data has been put off on the archive, we truncate all the file contents.
It has a slightly different meaning here, I've put this little offline flag here, if there
are file system people here, when you have unwritten extents, there's no file data, but
there's a logical block allocated, we do the same thing except that marking on the extent
of the data block says, I have this on an archive, but it's not here in this file system
cache, all that's to say that the data at rest, there's no file contents.
This is a file system with billions and billions of files with no data, it's just the metadata,
and that's mostly what makes these systems so weird, but so to go through this archive flow,
they'll dump all the files in this Pozix file system, you know, through our sink or an
FS or Samba or whatever, they have their own homegrown data motion scripts or whatever,
our little archive agent will go and get all those files read them again,
write them off to the archive and now they're at rest, and then when they later want to recall
them, they'll try and fetch them from the Pozix file system again via whatever means,
and the kernel sees that the extents are offline and it goes and asks the archive agent to go
get it from the archive and it brings it over, writes it into the file system, and then the
reads are satisfied and off it goes out through the services. So the data has this flow in and out
of the system, and we have a bunch of tasks reading and writing all the time and all these little
files sometimes, and then we have our software agent that's behind the scenes also messing around
the files, and what we're to understand from all this is that we can have very weird access patterns
that are, you know, defined by the site's access patterns, and then we also have our little archive
agent walking all the metadata, so we have a lot of opportunities for contention and for reads and
writes to land on each other and a lot of things, and that's what the talk is mostly going to talk about
that, how engine affects addresses that problem. After those two workflow topics here,
we're describing the environment just to give you an idea, very big namespaces, that's another part
of how these systems are interesting. The reason this thing is multi-node because is because of these
archive tiers are big enough that they have to have a lot of nodes to get aggregate bandwidth
to it. These two charming photos on the right. That's the same manufacturer, it's the same drive.
The top one is an awesome glamorous shot, right? The bottom one is one of us walking to a
data center taking a photo of the phone. It's the exact same thing. Those are racks and racks of tape
drives and robots and actual cartridges. Often, these are a challenge because that little third
bullet there drives are not uniformly accessible from any node. This gets to the contention
problem again. We can have files arrive on one node for whatever reason, but they need to be read
from a different node because they have to be written to a specific tape drive because the
policy engine said it has to be in this set of tapes. It means we don't have, this is where the
multi-node kind of pozics name space comes from. The services need to be able to read and write
from any node and our archive agent has to be able to read and write from any node because
files have to flow through specific nodes to get to some archive media sometimes.
I just threw in a little example there is that's operational in the field. This class of machine,
a bunch of network out the top, a bunch of, well, the storage fabric in this example is
in fit a band, but that's a thing and then a bunch of fiber channel for the tape drives. This is
what's deployed today to give you an idea of the scale of the thing. The billions of files in
particular, sometimes we have, you need to do operations on a lot of those files and doing
anything of billion times is really upsetting. So, in comes engine a fest. This is how
I like to sort of understand the different design choices you can make to get to this coherent
pozics name space and what we're doing differently in engine FS. So, we can start with that column
on the far left to sort of frame the rest of them. To me, this is like the minimal job of the file
system is to go between processes that are calling a system causes or the P's at the top and the
devices that are doing the actual IO. That's the stuff at the bottom. Anything in the middle is
irritating and we wish we could get rid of it. But we have to have the processes doing the work
and we have to have the devices actually working with persistence. So, in the local file system case,
you construct a coherent pozics name space with a bunch of kernel software runtime constructs.
You have caches and you have spin locks and you have mutexes and all that stuff. That's what
makes it safe to have a bunch of processes working on one name space. If you try to have two tasks
delete the same file, it'll be in the VFS in one of these locking constructs that makes that
operation safe. That's where the consistency comes from. So, that's sort of how we start the
conversation. That's in your head. That's how local file systems are. X3, XD4, XFS. All that stuff.
That's the world there. But that's a single node. We want to be able to get this coherent
name space across multiple nodes. How do we do that? The next column over is NFS, which is one of the
first and easiest to understand. We have the exact same model where we have tasks calling into the
VFS. But now instead of what comes out is just raw block IO. What we do instead is we just package
up those calls again. If you did an unlink of a file, it goes through the VFS. We just send the same
message that says, and just unlock that file and we send it off to another VFS. And it's down in
that other VFS in the server where the safety comes from. If you get two tasks on different nodes
trying to unlink the same file, they don't know about each other at all. And it's in the server where
those two things are resolved. So, that's where the safety lives in the NFS model. And that's a
problem because it means safety is only resolved off at the server. If you want to unlink a whole
bunch of files, you're sending a whole bunch of messages to that remote server. You can't make
local cash decisions because you don't know if they're safe until the server has checked with
everybody else. So, you get these per file round trip RPC costs to get safety. So, that's the sort
of quote unquote network model. These little quoted terms at the top are just how I think of this
in my head. The third column, what we'll call shared block, fellow dinosaurs might think of these
is custard file systems. That's my brain. I have the example here is scout FS, which is a system
we have. There are others, GFS to OCFS to all these class of systems. There's a bunch of proprietary
ones work in the same way. This is more of a hybrid between the local file system and the network
of FS a little bit. So, we start again with all the processes calling into the VFS. But in this
case, what comes out of the bottom of the VFS is block IO again. It happens to go to a, you know,
a raid head, a shared cluster block storage thing. But what's interesting is the safety, even
though the VFS is a remitting block IO, they don't know that it's safe unless they've talked to
this external lock service. So, if we go all the way back to the one on the left, the local model,
the way the kernel made that safe was you get a software mutex and you do your work and you unlock
the mutex. In this shared block model, you sort of take that programming concept, but now instead
of a local kernel software lock, you have a remote RPC lock and unlock. So, it's the same kind
of system where in these shared block file systems, you have your metadata processing code path
and instead of mutex on lock and lock. You send these RPCs off to a remote thing and there's
caching and stuff, it's a little more complicated than that. But the basic idea is you're doing
a lot of remote messages out of band to get these locks that make your operation safe. That's where
the safety comes from. And again, kind of like the POSIX RPC case, this is a lot of extra messaging.
Like, if you were to compare the two flows, the local file system compared to these shared block
file systems, there'd be a lot of sort of latency hiccups in the shared block one where it's talking
to its remote locks, or as it does all its work. And so, this is where engine effects comes in.
A brief little fun anecdote. The name came from me being so frustrated at naming and wanting to
start messing around with prototyping. So, I just did NGN as next gen and then realized it could be
pronounced engine. So, that's engine effects. Never get it naming. But in this model, we've made it look
a little more like CPU cache coherency protocols. What's happening here is we have the same kind of
model where the VFSs are emitting block IO again. So, they look a lot like the fast local
file system case. But there's metadata along with the IO's that manages cache coherence. So,
instead of for every operation sending a remote call to a server like we did in NFS, we do the same
exact kind of IO as you'd see with the XC4 or whatever. But we have these metadata tags flowing
along with the IO. And these little devd boxes, the main design element here is that in
NGNFS, there's a user space process sitting in front of every single device. And that user space
process offers a network protocol that looks like an IO protocol. It has reads and writes and all that.
But it has a few more messages for cache coherence. And in the reads and writes, are
some more metadata to describe cache intent. And that's the biggest takeaway here. What we're doing is,
instead of having a VFS talk to devices with just block IO and VME, SAS, whatever,
we have the VFS doing a network protocol to little purdevice servers that are then doing the IO to the
devices. So, to make that make a little more sense, we have this little timeline that is,
this is the heart of the cache coherence model model. So, down that middle spine,
this is all from the perspective of one block, the system is all block IO. The middle spine is the
state of one of these device servers that's responding to all these network requests. On the
left and right side, we have the VFS agents that are doing an operation. The operation can be darn
near anything. But we start this timeline time flows down. On the left, we have a VFS that wants to,
wants to update a block. Whatever it is, it's going to touch an I now, it's going to remove a
directory entry, who knows. It has to first get the current version of the block to be able to modify it.
So, it does one of these network protocol messages to the device server to say, I'd like to read this
block, please. But I'm reading this because I want to write. I send a little right intent tag that tells
the metadata server, the devd, that I would like to get this block because I'm going to modify it.
So, the moment the device server responds with the actual contents of the block,
in its state, it has a little database for the state of all the blocks. It remembers that that block,
someone's writing it. The moment it sends the actual current version of the block to the writer,
when the writer, the column on the left, gets the block. In memory, it can now consider that cash
block rightable. It can actually dirty it in memory. This is the usual file system path you'd
see. And so, now, before we consider the messaging from the other VFS who's trying to read the
block, we have a state where the device server remembered that it gave it a block for right,
and we have a VFS agent who is currently writing it in however it wants to. But for whatever reason,
say another VFS wants to read this block. If the first VFS was, you know, removing an entry,
say the other VFS wants to read the directory, right? For whatever reason, it wants to read this
block now. So it sends the same kind of read message, but it's a little metadata says,
I'm reading this block, actually because I want to read it, I don't want to write it. When the device
server gets that read message, it needs to resolve that right state, right? It can't just give
whatever version of the block it may have because that other note has written a more recent version,
right? So it sends a message to that dev the on the left to say,
and as somebody's trying to read this thing, give me back the block, please force your cash state
into read mode. So when the VFS on the left gets that message, it moves its version of its cash
block into only read mode, which because it had a dirty before, means it has to write the current
version out, right? So it sends that right command the back of the dev, these say, hey, I'm
modified this block here, it is. And as I send you this right message, I tell you,
and my cash is read only, you don't have to worry about me modifying this thing anymore,
I've made sure through my local software constructs that no one's going to modify that thing.
In the middle when the device server now gets that right command, it knows the block is readable
everywhere. It doesn't have to worry about writers, so it can send the block contents back to the reader.
So this looks a lot like people are familiar, this looks a lot like, you know,
CPU cash protocols, it's peer-to-peer instead of snooping all that stuff,
but blocks are in states and different actors, right? Some people have it for read, some people have it
for write, and there's that middle-devd process that's in charge of doing the messaging to
coordinate all this access. And what I think that the real
wind behind engine effects, if you look at this diagram,
if there were no cash coherence here, no multiple actors, a lot of these messages would be there anyway,
right? In fact, if you have to read a block and then you have to write it back out,
all that's present here, the only thing that's extra is that forced read mode message,
right? We don't have these external lock calls to make stuff safe. All we have is sort of
annotating the block since we trade them around. So this is kind of the core of the thing.
And if anyone has questions throughout this, please holler. I like interaction. It's good.
So this is the core of it. This is the heart of what makes this engine investing interesting.
And this is what the talk is about, just this little cash coherency protocol.
10 minutes, Jesus. Okay, but we can talk about, we can have a specific use case, as a file
system designer, why is this interesting? That comment is in the C file that handles this stuff
in the local VFS, and I think a few of you know who wrote that. So let's take this problem case,
as kind of an interesting example of if we're one of these local VFSs, and we're implementing
a POSIX file system with this read right, but cash aware network protocol. What does it do for us?
How does it save us work? We can look at this case because it's relatively easy to understand,
but this pattern applies to a lot of shared structures and file systems. There's a lot of stuff
under the hood, like allocators and orphan lists, and all sorts of nonsense that has the same
sort of pattern, but this one's really easy to understand, I think. So we have a problem in a
rename where you can't arbitrarily rename directories anywhere, because you can create loops.
That's what these first two little blocks are showing us, right? We make a directory link a chain,
and then if we were to try and move that A directory to be a sub directory of C, A would still have
B as an entry, right, and you get a loop out of that, and if you try to do that, it yells at you.
You cannot move a director under itself. There's a lot of other cases like this, but this is the one
that's easy to understand. The way it returned to that error message is what we're talking about.
If you look at it, it's not exactly obvious because, well, maybe I don't know.
The rename call on the kernel only takes two arguments. It takes the two directories that are being named.
So it doesn't really know that they're related to each other, and doesn't know that they're forming a cycle
initially. So it has to figure that out, and what it does internally is it walks from both directory arguments,
back up to the root, it just walks all the parent directories, and if it finds a relationship
between them that it's not comfortable with, it returns this error. That's what happens in this case, right?
It starts at C, and it's walking parents, and it sees A and says, hold on.
I walked from one of my operands, and I hit the other one, and it's bad news. That's what creates
the cycle, so it's returns the error. But you have to think that while you're walking these parents,
the tests you're doing only make any sense if they don't change while you're walking,
right? And this is the heart of the problem. In the kernel, we do this with a per-name space,
that's that DSB is super-block. There's a mutex for the entire name space. Any time you rename between
directories, you're globally serializing all renames between directories to this day.
That's all well and good when you're on a laptop. When your, you know, a bunch of nodes
doing this weird service on behalf of this like archival stack, serializing across the entire cluster,
across all of the machines, as a real problem. So it'd be nice to address this. And that's where
this sort of test, this per-block has to go in and see things comes in really powerful, because
by just implementing this naturally, we solved this problem, because as we saw in that sort of
cascoherency, message exchange chart, as we're reading, we're getting a cash state. As we're
walking those parents, we can say, I'm walking these parent blocks to test, you know, these
ancestor relationships. But I'm holding the blocks in Reed State, no one can write them until I've
finished my search. So by doing just using the block IOS as you would, you solve this global
rename problem and you don't have to have the global new text. I've made three little cases here
just to sort of demonstrate the different possibilities, right? The one we started with, we're
here return an error, we only get read references to the block, can be fully concurrent. There's
no writers, there's no serialization. All nodes can be doing that at all times. But even if the
rename does succeed, the points for contention where you'd have to wait for each other are
only the directories being modified. Right, this example on the lower left is sort of a silly
one where say you had a really deep directorie and you moved it all the way back to the front.
That's what this pattern would look like. It would have read references in the middle because
it was walking the parents to find out it was safe. It was safe and it modified the
directorie at the very top and the director is the very end. In the example in the lower right,
which is more common, you rename between sub directories that aren't ancestors of each other at all.
It'll walk to the parents so they'll get read references on all the blocks and then only modify
the leaf directories. So you can have as many of those you want concurrently all sharing read
references on the parents and what I like about this that's so interesting is we didn't have to
do any special engineering other than write a final system to avoid this global serialization
problem. Right, and that's what make this idea so powerful. And this is rename is just one example.
This happens with file reading, write all sorts of things and it's it's why as a file system designer
it's so interesting to work on this stuff because we don't have to spin up all these
you know extra locking relationships to make everything safe like we did in the rename case.
Just by doing your work by assembling the blocks you want to work with, modifying them or not,
and then letting go you get concurrency that's safe. And that's it. That's the talk.
It's just a little design overview of what makes this thing interesting. So if there's any questions,
I'd be happy to answer them.
We had deferred to the microphone here.
Speak very close to the microphone. Hi, great talk. One question. Partition tolerance?
What what? Partition tolerance? I didn't I'd understand. What do you do in the event of a network
Oh boy, that's a whole other layer. There's a core and set. There's maps of things. There's yeah, yeah,
all sorts of stuff. Partition. Yeah, that's what I didn't hear.
Need to somehow be able to throw it across the room, right? Yeah, we need to catch boxes like that.
Yeah, yeah, yeah.
What happens if dev decrashes?
So there's redundancy between them. You do rate on top of them.
And that's a hot mess, but that's part of it.
A little bit. There's some things inside post-expile systems. There's recovery protocols.
It's stuff like if you had a file descriptor open, there's a few tiny bits of memory that
is need to be known all over the place.
What if two nodes have the same read cache over the same region? And then somebody else
does to write and that on those blocks. Very similar thing as we saw that force mode read message.
The devity would send lots of messages that force them to be invalidated. So if a write request
came in, it would send invalidations to all the readers. They'd all drop their reads. And then
it keeps a list of all the addresses that are in these block numbers, yeah.
Can you say a bit about the state of production readiness of engine effects?
It's early days. Part of what's so interesting about this attempt. We've done some other file
systems. This one we are developing the open very early. A lot of the design is solid and we're
going to get this thing going. But if you were to look at that get-tree, you'd see that it's slim
pick-ins, but you'll see a lot of activity. It's early. Still.
A lot of fish done. Do you have an idea when you would be, you would be willing to
obscure this, I guess, right? Yeah, the client. What's interesting about a lot of this is
it's all in user space, but there's a VFS client. And yeah, that's pretty cool.
All right, thanks a lot. Thank you.