Before we dive into the talk, I would first like to ask how many of you have ever worked
on a document and you've ignored the battery warnings for the past hour when suddenly your
battery dies and you realize the last time you saved was about an hour ago and the hour
is gone.
Well, it's happened to me and it's very annoying so I hope this work can help alleviate this
issue and it can prove useful not just for programmers but for regular computer users in the future.
With that out of the way, let's officially begin.
Right, so first let's get our definitions out of the way. What is persistence?
Academically, it's defined as the period of time during which an object is usable and since we're
dealing with operating systems and kernels here, what we mean by object can basically be any
state of a program or like for example variables, expressions, etc. And what we want to
persist is to persist exactly what I mentioned in my example, system crashes, system outages that
happen unexpectedly. In short, your programs should not fear death.
Now, this is not a new topic, it's been around in academia for the past couple decades and there
are some traditional ways that traditional operating systems such as Linux and Windows for example,
they do some have some persistence mechanisms and I'm sure you're all familiar with them,
files. The problem with files is that the persistence mechanism is, the user space application is
responsible for the persistence mechanism. It has to serialize the data once to persist,
it has to create the file, write to it, update to it, etc. etc.
Persistence is in the hand of the application, right? So let me introduce to you something you may
not be familiar with Phantom OS. Is it a non-linear operating system offering orthogonal persistence?
Actually, Phantom OS has a new methodology in mind, they want to do a way with files as a
persistence mechanisms. A big caveat here in terms of the user space programs, you'll see why in a bit.
But it wants to do a way with files in short. And yeah, I'll explain how it does so, but as you can see here,
I've highlighted a term, orthogonality, which you may not be familiar with, but you will be shortly.
So orthogonality refers to these three concepts in dependence, which states that all data should
be treated the same, no matter how long the data lives for. The data type should not
infer anything about the persistence either, which is what call them to call them three
state, which you can read here. But in short, orthogonality, orthogonal persistence refers to the
fact that persistence is in the hand of the kernel, not in the hands of the user space program.
The programmers should only worry about getting the business logic out, no need to worry about saving
data. In short, in traditional systems, system outages are a full state. In Phantom OS,
it's just regular day life. Now, I'm sure you're curious how does Phantom OS achieve this
new type of persistence? Well, if you've played a video game in the past 20 or so years,
you know that most games have some sort of checkpoint mechanism, and it's much the same here.
Instead, that instead of calling it checkpoints, we call them snapshots.
So, okay, let's take a pause on that for a second. All user space programs are compiled to run on the
Phantom virtual machine, which is a language virtual machine, much like JVM, the Java virtual machine,
and it has strong tipization, which is backed up by memory protections. Now, when the different
user space programs run, they all run inside one singular Azure space, which is shared by all
programs. So now, the snapshot mechanism, please note that this is all done asynchronously to the running
programs. So, the user space programs are interrupted very minimally. Yep, for performance reasons.
So, first, we want to freeze the persistent Azure space, and to enable this to be done asynchronously
we use a copy on the right mechanism. Once we have frozen the Azure space, we copy all the data to
disk in what's known as a superblock, and then we unfreeze the persistent Azure space.
And of course, we can then do the inverse to recover the data back from the superblock after the
system outage. Now, this way of doing the snapshot is fast, and it's very easy to implement
from a developer perspective. However, what happens if your disk storage gets corrupted?
Well, you're cooked, right? You don't have any way to recover the data, it's corrupted.
And I hear you thinking, wouldn't it be nice to have an abstraction layer such as a file system
that can handle this sort of recovery? Well, that's exactly what the new snapshot mechanism aims to
do by integrating a file system as part of the snapshot mechanism. Now, I know how in the first part
of the presentation, I was bashing against files. However, this file system is just a way to manage
the data storage. It has no implications for the user space programs.
Yeah, and by using the file system, we can have retention and redundancy policies which I'll go
after in a bit. We added possible snapshots and multiple generations of snapshots, which I'll
cover later. Right, a brief caveat. Phantom OS was ported to the Genode framework by a student
working on this before me. This enabled us to use a Genode ecosystem of drivers and also to utilize
the micro-corner approach and configurable security policies and capabilities.
And of course, one such project in the Genode ecosystem is LWX-T4 developed by Joseph,
which proved the invaluable in this work. So, how does this just give you a brief overview of
what's going on under the hood? We have user land, which lives on top of the persistent
address space, then we have Genode running back and everything up. And we have the new snapshot
mechanism running as a separate Genode component. Now, this has isolation advantages and it also allows
us to manage snapshots from many different components and to also use the snapshot mechanisms
in other Genode projects. Right, so how does this new snapshot mechanism work? Well, let's start
from the basics again. You have your one singular persistent address space on which every single
program lives. And of course, it's split up into virtual memory pages. So, the first step would
be to label all of these pages. I will refer to these labels as snapper identifiers.
And so, let's take a snapshot of the persistent address space, but instead of saving the raw
data into disk, let's just save it into files into a file system and let's give it some sort of
direct directory hierarchy. The snapshot data will be stored in the black files, which I will refer to
as snapshot files. And please note that the labels on the pictures here do not necessarily correspond
to snapper identifiers. They're just locations in this directory hierarchy and the mapping
between the snapper identifiers and the paths are managed by the white files, which you can see over
there by the archive file. Now, okay, this is all well and good. We take one snapshot, we take another
snapshot. However, while this may be good in terms of redundancy, let's say, yes, we took a snapshot here,
we have some data stored here. And then maybe we have the same data because it didn't change
between the snapshots, it's still saved here. We have the same file storing the same data. Now,
this, again, it's good for redundancy, but in terms of disk storage efficiency, it's suboptimal.
So, what do we do when we have the same file? Well, we create a link, right? We build the new generations
on top of previous ones. And we just link back to older versions that contain the same snapshot data.
You can think of this link as a hard link. However, it's not implemented as a hard link,
despite my best efforts, because it's not supported in genode. So, this linking capability
is done via mappings in the archive file and reference counting in the file themselves.
This is the data layout of the snapshot file. We have some version of the snapper mechanism,
which stores the integrity of the version and the data. And the reference count to manage links,
linking. And the archive file, much very similar. We have a version, a hash, which has
the two here, the end of the data, the end refers to how many mappings we have in each archive file.
And data are the actual key value pairs of snapper identifiers and snapshot file paths.
All right, so benchmarks are a favorite topic. Let's get into it. Here are the boring specs.
Ooh, okay, so it doesn't do very well. And no, this is not a mistake in the graph. This is the original
super block approach. As you can see, it's quite fast compared to our brute implementation.
Yeah, it takes 43 seconds. Now, it's important to note that these 43 seconds are not interrupting
the user space programs. They are running just fine. This 43 seconds basically stops us from taking
a new snapshot within this time frame. So, we have to wait until this is over until we can take
the snapshot itself is synchronous between different snapshot. And also, we have a wider
range where we can have a power outage during the snapshot. And we can lose the data that we're trying
to save, which is not good. So, let's optimize. We have our snapshot file, which stores some data,
which is referred to by the snapper identifier. One to one mapping. Why don't we squeeze more
the virtual memory pages into this? In order to reduce the IO workload. So, that's what we did.
We grouped a bunch of memory pages and made it so that all the data is backed into one snapshot file.
Just a quick note. This optimization is done on Phantom OS. It's not managed by the snapper mechanism.
Since we want to keep the snapshot mechanism flexible to capture whatever the client component wants to
give it. So, again, this is on Phantom OS. And with this, we can see drastic performance
improvements, which level out about 128 pages per snapshot file. And it takes around one to seconds.
Again, much slower than the original, but we have four tolerance. So,
yes. And one quick thing to mention here, since we are grouping a lot of pages into one snapshot
file, if we do change the virtual memory pages changes, it's content. Then we have to update.
We can't just link to a previous snapshot file, right, in the previous generation. We have to
basically create a new snapshot file. And that's where you can see here. For example, in the last
column, we are changing 25 per almost 25% of our snapshot files every new generation.
And if we have one virtual memory page, we change only the ones that we need to change effectively.
And how you can use and configure this, you can configure the redundancy levels,
which refer to after how many generations do you want to start making redundant copies?
Because if you just have links back to other generations and you something happens to that
all generation, you lose data. So, this redundancy basically creates copies after certain amount of
generations. And you can configure the amount of snapshots you want to have at all times
the expiration date, some other stuff, look at the GitHub for more.
And some four tolerance frequent ask questions. What happens if the system shuts down during
the snapshot process and it corrupts the file system? Well, since we are using the file system,
we can benefit from the utilities around it. So, you can just use FSCK to just fix your system.
However, this leads to the second question. After using this tool, we had corrupted data.
Okay, we recovered the file system, but that data is still corrupted and it invalidates the hashes.
So, unfortunately, you will have to go into the file system, remove the problematic files to just
get rid of the warnings. But then your system should recover the non-problematic pages.
Now, towards the end of the talk, I would like to mention some other developments in the Phantom OS world,
especially these two projects done by students prior to me. We have the introduction of
a WASM runtime in order to increase support for more programming languages. And we also have
a way to restore connections. For example, TCP, if one machine dies, if it's before a certain
timeout, you can still recover the connection, however, with this working work,
you can restore the connection if both machines die and, yeah, you can restore the connection.
Now, let's see demo of Phantom OS using this new mechanism.
So, here we start the operating system. As you can see, we have a weatherup displaying some sort of
states, it's making some progress. And we started the snapshot, it committed here. So, now,
if the system crashes, just remember where the graph is. Okay, the system crashed.
Yep. So, let's see if we recover the state. Here we reboot.
Recovering. And there you go. Right? Like, yeah, you didn't lose anything. No files needed
that you know of. It's so managed by the kernel. They have. You didn't lose any of your work.
So, to conclude, what did we recover the question of traditional persistence mechanisms and
whether they're outdated or not? Of course, we covered how Phantom OS used to take snapshots
of the data, as well as the newer alternative. Whether it's better or not, I leave that up to you.
I do want to mention that this work is experimental and not ready for production use,
but I think it's a cool concept. And of course, here are some references in case you're interested
or yeah, also Phantom OS was present in, I had a talk in 2020. So, if you want to go check
that out, please do. And I leave the floor up to any questions that you may have.
Thank you.
Yep. Right. Talk in one of your slides. You mentioned that your Heshes are just four
fights, why isn't that a little bit narrow? Yes. It's a 32-bit Heshes that does need to be
a question theory. Yes. So, that's a great question. So, to restate the question,
I am using Heshes that are, let's say, not the best for error detection,
since they're not really collision-free. And it's restricted to four bites. Of course,
this is very, I'll just go back to the slide.
Yeah, for instance, this. This is still in development. So, yes, it could be expanded to larger,
but I prioritized a quickness. So, those were the trade-offs I had in mind. And also, the Heshes
mainly to keep track of whether the state has changed, if that makes sense. So, a force.
What happens if you actually run into a collision? Would it ruin your snapshot?
Yeah, it will just create a new snapshot file. Because once,
this is the text, once we detect an erroneous hash, because the data doesn't match, or if it matches,
then yes, yes. Okay, I see which, yes, you would reuse some state, which is not the best, but yeah,
this should probably be changed, yeah. Yeah. Thanks, want to talk about how does your work
relate to some of the hardware advances like non-volatile or in the access memory?
That might actually sort of render your work obsolete, I don't know.
I haven't looked into that, but it's basically a RAM that stays state-spot-system, even if you
don't need that. Ah, okay, yeah, this works. This works for traditional RAM. Obviously,
non-volatile memory would be great to have, and you can buy it even nowadays.
Yes, yes, but you can actually, yeah, I mean, just for that, you might still use it to use your mechanism
for any multiple stack shelves, and basically use the NVRIM as an accelerator, so, so, yeah,
perhaps, yeah, it's, I'll look into it, but of course, yeah, this, if it's non-volatile,
but then we don't need to manage the snapshots, right? Like, if it's non-volatile, we'll just persist.
So, maybe, yeah, definitely, I mean, yeah, just look into it and see, see, if it does not
like your work, so, okay, okay. Another question very quickly, so, how does this, what's the
line, actually, the originals, call them, call them, call them, call them, and this one, it's really
just a continuation, recrystallment, how would you call it? It's mainly worked on by students,
so it's just exploring a new direction, building on top of the microchernal capabilities that
you know, it provides, so, I mean, in my opinion, it would be great if that would be the main
stream, but the developer, there's also a separate Phantom OS branch, which is,
just purely Phantom OS, however, this has the benefits of, like, using the networking stack of
genomes, so, yeah, so the originals are still being done, yes, yes, yes, yes, yes, yes, let's go,
question from the back. Yeah, so I remember the original Phantom OS thought, okay, not what the
address space is like, 62 bits, right? Yep, it's you, but, but here you show that you could run both
Phantom OS instances, right? So, I'm not going to confirm what is kind of the granularity of
Phantom OS, it's like with you, but a brown run and one thing, the last one is one,
other applications, the differences, the one and one. Okay, so the question was, could we have
more than one Phantom OS instances, and would we use the different
instances for different apps, is that correct? From what I've read and what I've worked on,
it's under the assumption that it's one Phantom OS instance, and wow, I guess, I don't know,
it seems like you promote like having different containers for different, different protection
domains for different applications on Phantom OS, you really want to, but it's designed to run
all your programs for this instance. Wow, yes, maybe you could find the use case for having
different Phantom OS instances, they wouldn't be able to communicate within one, with each other.
So, they'll have to be managed separately. But I guess you could then link it back to this
snap or mechanism, which is running as it's own server, to manage all of your instances. So,
I could see it being used like you suggested. Yep.
So, if your premise is to get rid of the file system for applications, right? Yeah.
And the file system is not only used for participants, but also for communication in a sense, right?
Well, you open an editor, write a C file, and compile it with a different program, so how does that fit into your
concept, basically, that you also use in your file system for purposes, other than just, you know,
Yeah. So, the question was, how do we, files are not only used for persistence, they're also used for
message passing, and how does Phantom OS accommodate for that? Well, Phantom OS doesn't, it still
wants to support files. However, it doesn't want to, it promotes writing programs that don't
depend on files for persistence. Mainly, that's, that's, it's mean like, right, with it. I guess
it still supports files for, you can still use files for persistence, but that's not what it's
promotes. So, you could use it for message passing. The files, yeah. Yeah.
Yep. Yeah.
So, the question was, can you take snapshots of just certain processes that you're interested in?
That's a very interesting topic, one which I really want to look into.
However, I haven't looked into it yet since this is all running into all the processes are running
on a single persistent Azure space, which is, again, it uses strong tipization by the virtual machine.
So, I like looking through the code base, I couldn't really discern which, which data was used
for which file. Again, more research is needed in that area, but it's a really cool concept,
because currently, when it freezes, the snapshot mechanism has to go through all of the pages,
and it would be really cool if you could go through only a certain region, let's say, and then
pause that and pause another one. So, yeah. Do you have a connection with me,
what's in your process ID and application, because then if you don't have what's in the context,
and you see, yeah, this is positive. So, it cannot kind of simply kill the app or remove the app from
their context, because as to make you need to store all kind of information about the processes as well.
So, yeah, that was when I was going to say, yeah, I think it's doable. So, and that's it.
Yeah. I haven't looked into the, yeah, well,
because if you have the different types, not sure that you're going to store them, then you
will have a good application, you'll look at the trace, what would you do? Yeah, that is it,
that also looks bad. Excuse me, can you probably read this? That's it, that's it, right?
Yeah. Yeah. Yeah. Yeah.
You used what? The system address best for all applications. Yeah. Is it all sorts of
the case for the virtual memory address space? Yeah. Well, the browser process could
exist memory from the web app, in that example. There are some memory protections in place.
I haven't looked into the code of the pvm. So, I can't answer in a detailed way, but I've read
that there are memory protections, and that's as far as my knowledge extends in that area.
If you would, yeah, memory protections, you could use a communication mode on just like a smart
of VM dip back in the day. The eight is where you could have the web objects that could be
accessible for each other objects. Yeah. I assume there is some sort of protection, but again,
I can't answer you with 100% certainty. Yeah. It sounds like when you take a snapshot,
you have to freeze everything and scan every page. Have you experimented this right protecting
pages so that you can actually see which ones are being modified? That's exactly what it does,
because it's a copy on the right. So, it sets the all the pages to read only.
And then all the new pages that happen during the snapshot process. Yeah. Yeah.
Yes. Okay. Thank you. Thank you.