I'm Alex. I'm a software engineer for freedom of the presentation. I'm going to talk
today about the subject of reproducible images. I want to do something before I even sign
up a thing of the sort. How many people here have built irreproducible
e-mots ever? Can I see a raise of hands? Yeah, and keep your hands up if you keep
monitoring them ever since. Yeah, you're going to do it. Okay. So this talk will be about
like why the rest of the people who didn't raise the hands should care about the subject
of reproducible images. We're going to discuss how you can build one. Maybe we'll also
offer some tools around that. And then we'll see, we'll go just a step beyond and we'll see
what do we need besides building this image and reproducing it once? So I assume that if you're
interested in reproducible builds, you have some sort of trust issues. I might trust issues
are the fact that we're building an application in a freedom of the presentation. It's called
Danger Zone. And this application is used to sanitize and trust documents with containers.
This application is supposed to be used by an experienced users, journalists, activists,
whistleblowers, and therefore it has to be auto updated. So we just point out later. There's
no way to point out a specific digest. And both officially and unofficially, we kind of know
that our application is used in within T-plines of major newsrooms. So you can understand that
we are a bit frightened of what we serve to the users. What are our concerns? We continuously
build our images in GitHub actions. GitHub is a US company. I'm not sure if we will trust
this type of thing for activists and whistleblowers at this time and age. And even if we did,
the rest of the users need to be able to verify. But what they get is what is what's
currently built by the team of our main brands. So why we care that much about reproducible
and why you make care as well? Because A people can audit the end result easily. Because people can
know that the end result, people can also debug those can be detected very easily as well.
And we do make some checks out of band too. And because those things happen, then my laptop,
my hardware keys that I'm using to shine the images, the sci-fi plans that we're using,
are not a viable target. So this gives us a piece of mind. That's why we have taken the last
year to make our container image a bit for a bit reproducible. For those who don't know what
we produce to build in means in this list with regards to container images, it's basically
if you have like the build environment with all the files that you need to build the image,
if you have a Docker file and if you have a build command, it should be able to create a bit
for the same image if you run it now, five minutes later, one hour later, one year later.
So let's see how we can create reproducible image. Basically, if you take a look at the
documentation, you'll see that you need to set a source date epoch and pass some sort of
rewrite timestamps flag, which we'll be used to set the timestamps of AIMAS layers to that epoch
specified. And you can see here that this is a way Docker and behind the scenes build it and
podman do this. Now, once you have, as you know, the commands you need to use, let's assume that
you try to create an image that is reproducible. Is this similar to reproducible by the way?
No way at all. And it's easy to see if you just rebuild it twice. But that's not the
interesting thing. Most of you probably know why it's not reproducible. What's interesting is
how can you assess, how can you understand why those two differences differ? What's the difference
between those cases? You do see there's a difference, but why is that? And whenever I
am faced with something like this, there is a very helpful project that offers some tooling around
reproducibility and containers. It's reproducible containers project. I'm not affiliated with them.
But it has been really cool to use. And I'm going to show you some tools from this project.
First is the DFOCI tool, which you can pass it to images and you can also pass that you want to
check just the file contents. And then you want to, for instance, dump just the differences in the file.
And if you run this, for this example that I gave you, you will see that the differences are just
some apt files. And we can further dig into that by, because we've done the contents,
by doing the adiv of those files. And seeing that the difference here is that the
somehow in the apt file and the log file, the host log has been captured and it. So the source
date epoch that we used before is not a panache for everything. You need, you still need to do
some stuff in order to make your image reproducible. And one may not even say, okay, let's just
remove the logs. But that's not the full story, because we, where did we install this issue from?
We sold it actually, even though it was a timestamp, there was a tag for this container image.
We installed it from the main debutant repos. And this means that it can change at any time.
So the solution, the solution to create a reproducible at least debutant image is to use
the snapshot repos. Point them to something that aligns with your source date epoch.
And then remove classes. And that's the second helper from this reproducible containers project that
is very useful, which does this boilerplate stuff for us. And because you get stuff from the snapshot
repos and you know that it's sometimes easier to to cache and reduce the loading snapshot
service because they need the bandwidth. So on the left, we can see the original image we proposed,
which is not reproducible. And here's the a bit more variable sting unfortunately,
but this is the more reproducible thing on the right. And if you run this, the digest should always be
that one. So we saw how we can build a reproducible image. But what can we do after that?
And also, that image that I show you is like a two-line Dockerfile for production images. Most likely
it's a much, much more complex image. And here are many more social from other terminism.
We have been beaten when converting our own image from very stuff. We in order to make it fully
reproducible, we have to build it and across different run science, we have to build it and across
different days. And while I'm using the days thing, because while failing and debugging and
failing and debugging and finding stuff, I came across like a pretty simple example to understand
if like if the Dockerfile and sing can't be reproducible. And here's a pretty simple command.
It's just lots of Dockerfiles have it. You are the user and in order to run in a less
privileged user your application. So this, whichever Dockerfile has something like this, unless
the person who wrote that really knows what they're doing, will be reproducible in five minutes
from now. It will be one hour from now. It won't be in one day from now. And why is that? Because
whenever you create a user in Etisado, the unique step of since you create a user is added there.
So the reproducible variant is to do this add user chasing, at least that's what we found.
I have seen very little examples of people doing that, which gives me the impression that
there are very, very few images that are actually reproducible out there. So we have been
it by so many things. We wanted some scientific checks, in order to feel that we are in the ground
above us won't give in. So what we did is that we had this Dockerfile, they produced it all
when we showed before. And we just created a CI job which will run a build command every night
and will make sure that the image would always match the digest and at some point we were
kind of bored with it because nothing failed. And then something did. And imagine our surprise,
it is not supposed to happen. It's mostly a scientific check. And a couple of weeks in,
you just stumbled on this thing. What was it? At that time when our CI test run built it,
0200, had a regression, it added a small variant filled in the manifest. And it's a very
very sequential change, but it changed a bit for a bit reproducibility that we strive for.
We filed immediately a bug reports and the built-in maintainers fixed it very fast. And we thank
you for that. But if we weren't reproducing this very image and this was out to the longer
and people didn't notice, there will be images that could be reproducible with that particular
version of build kit. And like once that bug was fixed, there wouldn't be able to be reproducible
again. There would have to be the built version, there would have to be pins to that buggy,
let's say, version. So we have collected lots of tooling and some workflows over the last year
where we're doing this. And I wanted to show you here some, like, we have a repo,
which you can do it, which you're going to say below, where with basically what we did is that
we created scripts for all the incoming mistakes that we did over this last year. In order
and to be able to do some stuff more easily, to verify images more easily, etc. So that people
don't need to encounter them again and again again. And the main thing that this repo offers
is a Python script. It doesn't have any dependencies, it pins build kit to a specific version
solely to make sure that nothing changes and also compressive the latest we'll see that later.
In a form some strict build parameters and it's supposed to be used both with Docker and
Perman, CLI's and I'm using. I'm not using the cooler variants that are out there,
because at the end of the day, what's the goal is not for us developers only to reproduce
elements. The goal is to make it easier for any semi-technical person or a person who is not
that well-versed in the container ecosystem to just have a way to reproduce an image. And those people
may be Windows laptop and Mac OS laptop, etc. We also bundle this as a GitHub action and
we have made it so that it's a drop-in replacement for the Docker build push-action,
at least for simple builds, a drop-in as in the need to also define a source-day repo
but I know that the interface is the same. And what's interesting and hopefully we'll be very helpful
is that we also have a workflow, an action again, which does the following. It can build your image,
it can build your project, an image from your project locally. And then you can pass it either
a target image that you have push-summer, even the latest brands of that image so that you can
check that the tip of your main branch matches to the latest sag of whatever register you're using.
Or you can just paint it to a specific register and make sure that that's the exact same thing
all the time. And that thing is something can easily run nightling. It should be a small change.
We are also eating our own dog food in this repo and we are building nightling,
bookroom and tricksy images and we offer them so that you can use them,
using for testing purposes ideally because it's not super tested but the point is that you should
be able to reproduce them independently. And we still reproduce this very little digest that
helped us find the bug in the first place. Without said, there are still some stuff to do.
While working on this, we realized that if people want to reproduce your image,
they need to know what was the environment like a year after this image is out. They need to know
what's the environment and they need to know what were the instructions you used in order to build
that image. How many people will know that you use this repo build script? So ideally,
it should be said described. What's also, what we also believe that is very critical in order to
have a healthy ecosystem and be safer is to have something similar that Debian is doing and
the rest of the reproducible build project, which have like some sort of registered packages in
their case images could be in our case, where they are continuously monitored for aggressions.
And we can see if there are the reproducibility presented false or increases. And finally,
we would like some improved ergonomics for this thing, for multi-arch and complex build pipelines.
So that's all for me. Thanks a lot for listening. And here's the links for the project,
and the brief summary, what is discussed. Thank you.
Questions?
Hi. Thank you for the talk. This is exactly what there's a report.
My question is, do you know, do you know, if there's a new report, the Docker file, if you want it to be
reproducible, half of our file is now just this enabled and snap for repo, we're moving logs, etc.
Is there, like, any base image that would have this old pre enabled or make it easier?
And an image that has this pouch.
An image that has those things, there's not such reports, etc.
Back then, you said, right? Yeah, an image that would pre enabled the snap for repo, get to
not log anything like we'd have installed, etc. Yeah, it doesn't make sense. That's the images
that we are offering. The ones we see that we showed before pre-enable,
the snaps at repo, they do not do something about logging, so that's something we need to
remove on your own. But to be fair, for my experience, this is like just a tip of the iceberg.
Even if this existed, there are so many things other than that that you need to figure out at some
point because you're bound to hit into other limitations. So what I want to stress here is that
it's good to have an image which has this reconfigured, but it's much more important to make it
easy to do this build, check, beef, and then have this thing. So yeah.
How slow is it for non-representable responses? What's the performance? How slow it is for the performance?
Yeah, like non-representable. How fast they are. Because we rewrite the layers in order to set
the source date Apple, this creates definitely an overhead. This creates an overhead when you
rewrite the layers for the Apple. It's not as fast as regular images that for sure, but there are
other reasons you're doing though, in theory. Thank you.