Hello everybody. Good morning. Thank you for your help. Hello. So, we'll talk about
reproducible bits. We are first to brief history and then we have some status reports about
various distros. But first about you, who knows about reproducible bits? Okay.
Then we can go home. Great. Who contributed or contributes? Yeah. Thank you.
Who knows? We produce a bits. It has been around for more than 10 years. Probably there was
to no. For 30 years. But I don't explain what it's talk. And you know, an S-bomb is basically
built in for files as we designed them in 2014. About us, K-P. Hello. My name is K-P-CYD. I contribute
to multiple distributions, like little dust for deviant, alpine and actually looks. That's
been most active. I've been doing security research for 10 years and I created like a website
to index source code in response to this exact incident last year. There you can compare
tab or file there on different distros, what source. Yeah. So, professionally, I work for
Red Hatz. In my hobby, I do art stuff. I'm a holder. I'm old. I installed deviant for the first
time 30 years ago. And I work on reproducible. It's in 10 years. And we try to make all
software reproducible, all free software. And it's not only us, but many people have been
involved in this. And yesterday, 11 years ago, Lunar gave a presentation here about reproducible
deviant back then. Then 10 years ago, also yesterday, Lunar and me gave this talk. I had lots
of fun. And then sadly, last year, November, Lunar died. Yeah. Lunar made a big difference, as you
will see. And we are here for many projects. I'm not going to list them all here now. But
so the introduction, probably all know, source code is free available. Most people installed
here in store binaries. And no one really knows whether they really correspond to each other.
And as a result, they are very supply chain detects. And if documented this by now very well,
always a bit livable. It will be reproducible given the same source code bit environment and
bit instructions. Any party can recreate bit by bit identical copies of the specified artifact.
And this is a slide from 10 years ago. And we already had the same thing.
And our mission is to enable anyone to independently verify that a given source and produce
bit by bit identical results. That's what we do. And that's easy to explain, right?
And they are important building block that don't make software more secure, but they make the supply chain more secure.
And unsecure software still is unsecure, it just changes that. And as this is very difficult to
understand, we came up, what does that even mean? Can you explain this to your co-workers who are not
involved with this? So our new slogan is just enabling supply chain security. That's what we do.
Everybody gets this. So we produce a bit by 24 or 25. And we have been widely understood.
We have all these resources. We have the documentation. We have scientific publications.
We can look up this because the stock is very brief. And even the White House has understood
reproducer this. They made a memo in 2021 about supply chain security. And in the
additional measurements, they also explain a bit by bit of abusive bits. So how did we get there?
Money and what's no?
Why money? In 2011, Bitcoin made their client reproducer work. That time Bitcoin was
worth $4 billion and they were afraid. If some backdoor client would steal the money,
they would be blamed. And so they made it reproducible. And that was for the first time
we learned that software can be made reproducible. And Snowden was a huge motivation in
many things. And set Perry, Mike Perry, and set Shuren. And others made the tour brother reproducible.
And that's firefox. And firefox at that time was 50 mega by binary or something.
And every bit reproducible was pretty neat. And how did we really get there?
Lots of work by many, many people. Yeah, it was the reproducible. Some of the 2019 in Maracas.
And so in 2013, Lunar hosted a brainstorming meeting at Debt Com 13. And another Debt Com 14.
The Debt Conference. And there were 30 people there spontaneously planned. And that's how it started.
Then Lunar gave this talk at first time. And then Lunar and others, the first patch is for deep
decades. Sorting fixes that the contents of the Debt Com 14. The Debt Com 14.
The Debt Com 14. Sorted correctly. And we designed bit in four files where the building
for files store the sources, the defense and the output. And in 2014, I started to rebuild
100 Debt Com 14. And compare this and that became the setup where we test all the
packages later. And Mike Perry and Citroen gave this presentation at the Chaos Communication
Congress in 2014, explaining the problem space very well. Back then, there were not solutions,
but rather it describing the problem. And this is the presentation. And most people think
they are trustworthy people. And why should somebody not trust them? And I am the developer
I built it myself. I am stored it. I know what I am doing. But developers are really good targets.
They explain very well that if nation states have an interest to get the benefit of attacking
developers is very high. And they also showed this vector like this was a CV and
SSH. And one bit was different in the code. It was greater or greater equal. And in the binary,
it is a single bit difference. And the binary is 500 kilo wide. And one bit can make this difference.
And they also had a nice proof of concept, the kernel module. So when a source code was
viewed with an editor, it would show this source code. But if the compiler would use it,
that would different source code compiled. They showed that vector also, that's by the motivation.
And then at first, 10 years ago, Lunar and I gave a talk trying to invite the whole free software
work to get involved. And Lunar gave a presentation at camp showing many problems like timestamps
and sorting issues and whatnot. I think 50 issues and 50 solutions for this problem.
And we had the first, we put into the summit in Athens. We are more than debut and people
gathered and that was really the kickoff. And we wrote source state, epoch the spec.
And the Lunar Road, the Bindiff. And the Bindiff is now difficult. And there are more than 80
contributors. So it is really good. Explain this in a moment. And the common reasons for
unreboтоcibility are timestamps. Really timestamps. And build pass and build pass. And for
build pass, we've come to conclude just rebuild in the same build pass. And all the rest. And all
the rest is a lot. But so, and we have some resources. We have this Lunar's talk or general
the documentation and resources has the talks. And it's also easier to show how what makes a
package unreboтоcibility. So we have a Git repository with the unreboтоcibility package
which has many problems and the fixes. And we also counted, we have like over 400 issues,
different types of issues which are all in this Git repository. And so we came up with source
state epoch, build timestamps are meaningless and source that epoch describes the time of the
last modification of the source code in seconds since the Unix epoch. And we did this spec
in 2015. We did one small patch in 2017 and since then it's stable. And it's adopted
as a border by a lot of software like compilers, but also documentations and our rate as
later PDF what not they all. If source that epoch is set, then the current data is replaced with
that. And difficult scope, who uses or has used difficult scope? Are you happy?
So difficult scope tries to get to the bottom of what makes files for packet
a direct list difference and will recursively unpack archives of many kinds and transform various
binary words to what human readable form to compare them. And you can just go to try difficult scope
or upload two things to whatever and it will compare that. That has test text and HTML output
and it supports many five formats. Anything that's way more than by now you can
two directories, two ISOs, two PDF things. I know lawyers who compare PDFs of laws
to see how the law has changed with the first code. There's many use cases and
if you haven't seen it, this is the diff and they're source the diff between two versions.
And then we had these summits, like we had eight I think now. And this year we tried to have a
summit in Vienna, like in September to November something. If you want to attend please contact us
we are happy to have more people there. It's like three days event. And there are many projects
not only distros but also academic researchers companies using it. There are many people
and another picture from Paris summit. And we had funding, we have funded, that's five people
being funded as a soccer freedom conspiracy project. That's Chris Lamp, a very grand cascadian
material. It's a low myself and since November it's KP also is funded.
We need funding for all these work to prepare the summit also. And we are very thankful for
all the funding. And now about rebuilding why we have KP now.
In December 2019, I've been working for about two years in the beginning of the project.
Two of the point of us mostly comparing if we take the source packages and build it twice
in different environments doesn't match, but actually care about the binaries that are in my computer
that actually install through the packet manager. So especially just a scheduler,
there's different projects. One is debris build and for entrepreneurs it's actually
a free throw that Foxbond and Robe. It's been in two or three parts. One is keeps tracks of
research. The other one actually runs the build. And the third one is like conforming it in
current research. For entrepreneurs that's right now about five instances that are run
better than groups. So if each of them kind of confirms the binary, the other did a conspiracy
or it's actually good to use. It's certainly in risk by me and source code is available here.
I've already did a package ring for Arch. You can build it from source code and we're working on a
debut package. This is kind of what it's built around. There's some kind of build input over here
and it's expected to have these two results and then we run this build in our own machine
and we expect to get the same binary. If we don't we throw the diff. It started as a prototype
has been used for five years as a prototype. It's kind of good enough to find issues and
troubles with them. We found some issues that we couldn't find on like the regular CI that
only showed up in the wild. One of them was the group package for example. We didn't
we noticed that if we built on a machine of us booted through like the master boot record
we would get a different binary than when it was booted. That's because the build tested
if there was nothing set it would pick the one that you booted as a default and we found through
as you like in the wild deployment. So this is sort of you of the way of the Arch
in involvement. So we already 10 years ago we started recording created a build info. So
if you build a package we would record the all the installed packages in the build routes
so that we then can recreate this build route and build the package again on a different machine.
After that we still needed to support like source it epoch so we would normalize all the time stamps
and at the time we also made a reproducer tool so this is a small batch break which
allows you to reproduce an existing package in the repository. But for that to be useful we
needed to have an archive of all the past packages we had we have which we normally didn't
yeah we didn't really have the structural archive of all the past packages. So now so that was
some work done so that we were properly organized when you were at it through the repository
and then they could be used to reproduce packages. Then yeah like four years ago we started
this rebuild a build review and since it was a great success so we had I think two B3 machines
with like 100 cores 64g ram and we managed to get at 86% of the core repository I think like 80%
of the actual repository so it was a good first step and in four years later we
KP actually managed to make a minimal container um usually fully reproducible so that I think
that's the base Docker image but now it's no longer reproducible but that's it has progressed
um yeah and now it's a current state we had like 12% so that like the 10% hardest the 10% is
going to take another I know a few years probably to get resolved and one of the big packages which
are like reproducible is like Haskell and the kernel is also a fun topic and yeah oh and I have
some graphs so this is the or the font and I wrote for the Rebuilder D instance so this shows you
all the bad packages to good packages and you can click on the link and get the this was
scope where you can get the build box so it was incredibly useful to treehouse stuff and this is
the recent I think yeah this is a year of the rebuild status and as you can see it has I'm not sure
why recently bumped up to more bad packages but it has been quite stable for some time
now David Kevin yeah we have reproducible potmen or Docker images for that period we have
life images and we have individual packages that was the status like until recently we had the
CI builders where we built twice this trying with maximum variation but we've done the last 10 years
and this is the graph and went up to 96% now and we found lots of bugs related and unrelated
to reproducible which is a nice side effects like really many bugs and we get now testing migration
we fixed this snapshot that we in all which has all the packages was broken for the last
more than four years and was finally fixed last summer and so last September we could
or finally rebuild again David packages but we were able in 2016 already so just I eight years later
and in October we set up reproduced every in that which is a rebuilder the instance
and that is comparing against what David and distributes on FTP debut and all and this is the current
graph now we are there at 86% for AMD 64 i386 is also getting there in the beginning we had
rebuilder D running on i386 but that doesn't work so now rebuilder D is always running on AMD 64
and just the workers on different arts we have arm 64 the hardware has some problems like the
file system is too small the same for arm hf and risk the machines are not powerful enough
and we miss four more release architectures so if you have these hardware or if you have other hardware
please talk to me we are very happy to use the same tool for debut as art Linux it's really great
but the debut service also way more way bigger we have eight architectures instead of one we have
two suites instead of one and we have three times as many binary packages as art so scaling is a bit
more an issue we need more beefy hardware and also there is a package rebuilder from Frederick
P.E which is also exist is written in pison and I would love if somebody sets this up and does
a rebuilder with this software just to test this alternatively if somebody is into pison you might
want to try this more help much work and the difference between theory and plexus for debut
is at the moment 10% we have 96.6 in cia and 86% on reproduced but we only do this in three months
so on how we want to reach 100% in practice 100% is a political decision in nothing technical
because there will always be packages and for debut we can just say this will not get into this
release so we need to change debut in policy we can work around known offenders with a
loudest in the beginning and the goal is still 100% loudest is just to work around to get there
and policy debut in policy was changed in 2017 packages should be built reproducible we hope to
change this year packages must not regress and also new packages must build reproducible
and hopefully in two years must be reproducible then there will be still policy violations
so maybe in 29 or something I don't know and this will be the release team enforcing this and so
I made the past 200% the upcoming debut in releases with whitelist so maybe 40 plus two
in whatever it's eight years I don't know next OS everybody knows next OS is reproducible by design
but that is maybe not true or that is true by the design yes but somebody related reboots
it's a slim Julian he gave a talk here yesterday and published a black block course and he found that
what is it? 91% very reproducible in April 23 so we all use the same software we all get the same
resides because we all have the current problems with the kernel has to be I think it's reproducible if you
build it on one core that's a good work around next OS has also a graph it also goes to 90%
previously there was also a talk at first in 2016 about making freebius D reproducible
the base system we tested on the strength and stability in the stands but it has become
test reproducible and now they tested ports which is packages under the freebius D word
and it's even 80% and then it got stored and just got picked up this month there's now this
zero trust initiative from the freebius D foundation I skip it a little bit where there's other
more than just reproducible there's a five point initiative and they got some funding
so there's looks like this year freebius D will make some huge progress as well
done Benedict Law is testing net BSD and finally net BSD built on Linux is more reproducible
net BSD built on net BSD but the really good part is that 44 to half of the ports a bit less
a bit by bit identical whether you build them on Linux or net BSD the reside is the same which is
pretty neat our BOS is from Bernard Wiederman who normally works on suze and here's been testing
suze since 10 years and it's done in a lot of patches but it's done in a lot of them upstream and he made
a BOS which is a proof of concept where he did lots of hacks like he built Haskell on one core and
he did other stuff that you cannot do in a distro but he achieved a minimum VM images which is
100% reproducible all the packages built from source are reproducible and he did a second thing
which is the DVD with a small graphical UI which is 100% reproducible that is very nice
and Fidora speak enough started to do rebers of Fidora he's also here I'm not sure if he's in the room
and he did mass rebers of Fidora and reached there in 1982 percent reproducible I'm not sure
whether it's CIO rebers but it's a proof of concept at the moment and he also wrote ad determinism
which I didn't mention we have in debut and we have stripped non-determinism where we remove
non-deterministic stuff and he has the opposite they have a matrix standard Fidora so in theory
we are done in practice we have shown that it can be done in theory now we need to close this gap
and those 5% are still crucial or 1% at least 1% for debut it means 370 source packages
so that's a lot because we looked at them for 10 years and didn't find the courses so they're still a bit to do
and many projects support or aim for it to be today that is the huge success
we wanted to change the software and I think we did think that's very good
and the next is finish those last 1% to 5% or 8 and there are some dragons like profile guide
optimizations there's other issues we need more infrastructure and processes and tools
because it's still theory and there's also tool for both arts, Linux and Debian, Debian,
Lepros, Datus, Arts, Depros, Datus which compares your packages but so there's
and so it meets pro level consensus that the project wants Debian has changed policy
much to how the arts processes are what others are, Fidora is a single person that has not
committed so it's not only to happen
yeah and this is again from 2014 it's basically the same today as we have today
but now it's possible back then it was ideas so thank you
do we have time for one or two more questions
yeah thank you for the talk I had a question
you've a closer so what's the reason you're not using QMU or something for
the more out there architectures for the Debian rebuild instead of replying hardware
basically because we started that three months ago and each other added to half the other architectures
two weeks ago and we don't have the powerful hardware to run to Emo and if you have powerful
hardware please stop to me
do the upstream's care I mean can you can you go to I don't know the GHC folks and say hey guys can you
make your stuff reproducible and or will they just like say we don't care most upstream's care
but I know like there has changed yeah there's one two sopah who still owns and there I
know what I'm doing and stuff but most have understood that they probably not completely know what they're
doing for the has got stuff there's been recent patches for the has jealous jealous are set for
the has got stuff there's been recent patches and we've also had cases where projects
announced that they're reproducible like wire got was reproducible on f-roid placed or in
something else and they just announced it on their mailing list but not to us they are just
doing it there is a meta problem of reproducible bits which is now there are containers which
are built on top of debion images and which is still packages and those containers need to be reproducible
as well for various reasons for the same reasons that builds into be reproducible in order to do that
timestamps again coming to play the installation time of a build of a package
messes up with the bit that bit to visibility so other plans for distributions to help with the installation
problem and the time's the thing that partly like this life images usually are installed
packages so there's also work on that yes but it's a different class of problem and so for this
bit out of scope but there's other doing that there's also the related project is good strapping
that the compilers can be good step which is also related but not the same so there's more
projects tackling related problem spaces all right thank you so much
See you