All right, next time we're going to have Luca talking to us about a reproducible
XFS file systems.
Hi, everyone.
I'm Luca.
Today we will talk about reproducible file system with XFS.
A little thing about me.
I'm a software engineering singer.
I know an open software enthusiast.
You can find me everywhere with 89, Luca 89, that's my mail.
Why are we interested in reproducible file systems?
We build stuff and we want that stuff to be very viable by the end user.
The user has access to the source code, so they should be able to reproduce our own
artefacts and be sure that we are not missing stuff up.
It's also useful for us for debugging, for auditability and so on, so it's very important.
Working in Schengard with our product of VMs, we wanted to be transfer for our users
and have reproducible images for them.
How do we use one usually approach publicizing the file system, MKFS, that file system,
you mount it, you copy stuff inside, and you are mounted, but this is actually very much
not reproducible.
We have some caveats, we need route privileges to do mounts, and we need a kernel for
that, which is not, I mean, it's very important.
Time stamps, obviously vary, you write different times, you have different time stamps, and
we have different random seeds that lead to different high-note generation, and so many
moving parts.
My first attempt was using a little effect time if someone doesn't know what it is, it's
a library where you basically use an LD-prilode attack, let's say, it will, in IJAC, most
of the random functions, like get time of day, get random, get from 32 and stuff like
that.
So, if we try to use a lib-fect time and create an empty MKFS file system, these are
really works.
We have an empty file system, which is reproducible, but in the moment that we mount it to
populate it, this will break reproducibility, even if we use no time, no time, no this
stuff.
There are many reasons for that.
First of all, LD-prilode, he's only on user space, and mount is kernel space, so that
won't work, simply.
Also, the creation time is set by the kernel, lot by the lib-c, so you cannot intercept
that.
Also, it's a very complex file system.
It has allocation groups, which has some eristic of, to spread the load, as parallel
and as in KO, which is non-deterministic, because it's not, and non-deterministic order
of allocation changes the B plus 3 shape, so that's also not reproducible.
So I went out and checked on reproducible.org.
There is a system images section that shows you how some file system are actually B can
actually be reproducible, like all the XTE family, SquashFS, EFS, and the ISOs.
The common part is that you don't mount it to populate it, you populate it while you do
it.
So let's do it.
I went, the plan was first, let's work on the hardest part, which is populating the
directory while we do MKFS, then we can do the deterministic parts, so block the fix the
time and fix the other random points, and then attest, because we always are to attest.
First step was populating, I went out and searched, and actually XFS has a way to, already
had a way to populate a file system, which is called Portofile, it's a very, very old specification,
which basically is like a text file that represents the file system in our text manner,
and as you can see, you have a reference to the original file that we want to copy, only
for regular files, all the other files like SimLink's chart devices, I don't know, block
devices and stuff like that, are actually created and they do not refer to the original
file, so we have limited extended attribute supports, because we lose them for directories
for SimLink's and stuff like that, we don't have actually control on timestamps,
because we could copy the timestamps only for regular files, all the other files are set
to current time, which we can control, but we lose the information from the original
source. At the beginning, I proposed to change this, but this was rejected because it's
a very old and let's say stable, you know. Yeah, and specification, so we decided, they decided
no, no, don't touch it, so they suggested me, we should actually create a populate
from directory functionality, and I tried, so basically what it does is we use the Portofile
infrastructure that was already there, and just branch out in the case that we are pointing
to directory to populate from, instead of file to populate from, so it will, the recursively
traverse all that are actually in that source directory, and then if it's a directory
recurs, it's a file, if it's a file created, copying from the source, timestamps, which
the modification time is what we usually care about. Access time usually is just noise, so
by default, it's not copied and set to current time. Creation time is set always to current
time, we will control current time afterwards. So yeah, usually in the reproducible builds
in the world, access time is stripped anyway, so by default, it's not preserved. All the
time stamps we need attributes, one caveat was that when we, for regular files, this was pretty
easy, the words are, they were already lots of function are pre-made to do that. For
the same links, sockets in feeforce, you cannot actually use the regular FGAT, get X attributes,
because we will not open them regularly, we use them with all parts and with no follow,
so we don't, you know, have loops, so we have to handle the bedfile descriptor error in
the world back to, I'll get X attributes and so on, because siblings and habits extended
up attributes that are important to, and if we just skip them, we might not have a function
of file system actually. The other part was tracking card links from the source directory
in the destination file system. The implementation is added, concentrate more on correctness
than performance, so it's linear array, it's a static global array, and then we just registered
all the various card links and refer to that, with some logic for the growth of the array.
Actually, I didn't find it a problem performance wise, because we are, so MKFS is IO bounded
and it's sequential, no parallelism, and I tested it on my flatback folder, which is full
of our links, and it behave good enough. Anyway, this is, maybe for the future, we can improve
that and do it and ask for something like that. We have populated the file system, now we
have to add the determinism to it, so we can try with, again, with leapfake time, and this
actually works, but we want to get away from leapfake time, because first of all, we
all deeper load type of things is not ideal, it doesn't always work, static linking is a
thing, this doesn't follow the reproducible build standards for source data, so, and basically
will depend on external tooling, instead of just doing MKFS, right? So, I went away ahead and
implement the source data epoch thing, so we get current fixed time, so if source data epoch
is specifying it's a valid number, because it's a unique timestamp actually, so seconds from
1970s, instead of get time of day, we just return the fixed value, this is used for creation
time, change time for I-nodes, and the other part is having the get random part, I was suggested
to do a very simple thing about just, is it deterministic mode and return a fixed value instead
of just passing the value that you want to be return it, because anyway, get random in MKFS time,
it's only used for the I-node generation number, and at MKFS time, you don't reuse it, I-nodes, so
it's not a problem if every I-node has the same generation number, so this doesn't break stuff,
and then we have the testing, so in the XFS tests, I added a test where we create a five-system
three times in a row, and they should always have the same harsh, very simple test, the five-system,
sorry, the source directory is generated with FFS stress, so it has all type of files, links,
and so on, so it should be quite representative, we can see it in action, so we can have a very
basic XFS file system that is probably populated from our root of FFS directory, we can preserve
access time with eight-time equals one, maybe it can be useful for someone, we can have an
actually reproducible file system, an empty file system without using blip effect time by using the
environment variables for source data book, and the deterministic seed, so this works,
be my full, we still need to fix the U-wide of the file system because it's random else, but yeah,
that's that works, and we can do a little test with make, or FI, to create a root of FFS for
Debian, and create multiple, sorry, images for it, so this is a script that just uses MK also
OSI to create our root of FFS, create an image file, and then MKFS it to impopulated, so if we
build it the first time, this will go, and sorry, create the root of FFS, which is the slow part,
actually, then after this it will automatically, you will see the familiar output of MKFS, which
is one, and it automatically also populated the file system, you can see the root of FFS, we save it,
in another name, and delete the root of FFS, and regenerate it again, this will rebuild the root of
FFS directory, which is reproducible because MK also OSI supports reproducible builds,
after that it will create a new image file and a new MKFS comment as you can see,
if we file the images, they are both valid, XFS file season, and both of them have the same hash.
This is very useful for us, we use it to build our images, so we know when an image actually
change it or not, before maybe shipping them to clouds and stuff like that, because many other
steps are not reproducible like converting our role, I don't know, VHD, VNDK, for BIS file format,
cloud format, so we know if we have to fire up our CI or not from this, for example,
so it's important for distributions, maybe also for embedded system, this can be integrated with
MKOS OSI, as you see, it's important for security, for compliance, you need reproducible builds and for testing,
these are the links to the patches, the first two are the MKFS patches, the test patches,
and then the documentation are reproducible BIS.org. Thank you.
My question would be, do you keep some artifacts of what was the environment you used to generate
the image, because if you want to, do you keep metadata files or artifacts that traces,
software you used on the version you used to obtain a particular file system?
For the MKOSI part, you mean? So this one is just for the demo, in Enchenga, we use
a upcode to build our root of the images, which is also the tool that we build for our container images,
and that one is also fully reproducible and we have a very historic part of how it was composed,
it's rendered, it's rendered, so it's fit the package versions, and it's pinned to snapshots
of the repositories.
All right, thanks a lot. Well, we're going to set the out of time, sorry. Okay, thank you.