Welcome to my talk about reproducible builds on Android.
I will show you today, let me first introduce myself, my name is Andreas Xtak Raybert.
Going by the Nick Izzy, I started Izzy on Android in 2015.
We are a member of the team for a couple of years.
We provide APKs, built and signed by the corresponding developers.
After performing some additional checks, like checking if there are clean virus total,
what's inside, libraries and so on.
For two years now, we also provide reproducible builds.
Meanwhile, covering about two sets of the apps in our repository to be more precise.
It's going to be 62.4% of our apps which are covered by reproducible builds.
I will today introduce you to our reproducible builds.
How they are set up and also compare with what FDroid is doing is,
I'm a former FDroid main planner as well.
Specific questions go to the end.
If there are some understanding questions, so you lose context,
I hope it's okay if you can ask them in between.
I'll talk to fast, please give me a signal.
I'm going to talk to fast.
I'll try my best to keep down.
Okay.
Today, first of all, overview.
The slide is also available at what is the code.
Pre-tots.
It's also available here for QR code, and there are a few classes at the end
to scan for the slides.
First, item we want to have is what are reproducible builds,
and what are they good for.
How do we approach reproducible?
It's an easy on-droid, and what's the difference for FDroid.
What are the main challenges?
I can't put them up all, it would take too long.
We would need two sessions, at least.
What are the most frequent reasons of failed reproducible builds?
Some of them also can fit.
How we can fix them.
What should Android app developers be aware of and take care
for to give the best chances for their apps to be reproducible?
And then we have a question and answers.
Okay. First, what are reproducible builds?
What does it mean to have a build reproducible?
You as a developer, build an APK, sign it, and provide it for install.
We give the source available.
We take your source, try to build the APK.
And if you come up with a bite identical copy,
that means you reproduced your APK.
We can tell everybody hey, this one is honest.
They really build this APK from their source code.
So that is basically what means reproducible builds.
Sir Party's make sure independently that your software isn't been altered.
Such increasing safety, reliability, and trust.
Because people who trust you or think maybe either can trust you.
Oh, somebody else could produce it, so it's really fine.
We have, we make air, well that's a part of it.
And make sure that the developers code always works the same way.
That is the best case.
Some builds are the smart sound funny, but it is a way.
They are non-deterministic, but reproducible.
How can that be?
We build in the loop and we find the right one.
There are some, how to say some errors in the build frameworks.
That cause different order of bite code, which causing reproducible to fail.
If you try often enough, one time lucky, you find the right one.
So we can prove, okay, it's the code just working the same way if you don't need to loop.
But it makes the software also more consistent because you know it works the same way.
And again, it's a trust case.
If it doesn't sum up and you've got some completely different one, you run a different scope on it and you see,
Well, it's an entire block of code that is not present in the reproducible build.
You know, maybe somebody was speaking into your code, but putting something in and poison supply chain or whatever.
Or you didn't pin the dependencies properly.
There's a difference.
So you see it shows up things where there are improvements.
You might want to add to your process.
A side effect of this is if you have it really matched.
Now, it comes to business case and says, we need to make sure we comply with all eye sensors.
We can, because we know what is inside, it is proven what is inside.
Everything is inside is mapped and we can say, if I say, if I say, if I say, if we meet that, everything is fine.
Nothing was struck in by some dependency of a dependency of a dependency that violates some of the eye sensors.
Everything is fine.
Yeah, binary is matched, so the source code obviously has to match.
I already said that before, protect your software against being compromised.
That is what we had at the start point already.
If the results are identical, it's protecting your software against being compromised because nobody has put something in the middle.
Or taking away.
The increased confidence and trust.
I said that already, but there's one more point to it.
I hope that is in here.
Yep.
It also can be used to confirm ownership in case of signing kiddo.
Because we have the records.
We have the records from the previous both.
It says, this commit, we have both from.
Is this commit, is still in your get repository.
And you didn't overwrite the history so we can match that.
There's an email address to the commit and we can say, yes, okay, that matches.
And we can confirm your key was lost.
You didn't just have somebody introduced his own stuff in materials.
And overtake your repository, but it was really good.
I take it that way.
Okay.
Let's, in overview about what we produce of the builds are.
And what are they good for?
There are many more use cases.
Just the core ones.
So how do we approach the produce of the builds?
That is the Android and on FDroid.
That is the Android and the progressive of the builds.
Run on a complete separate track.
You ship the original of AP case.
Don't sign pass the developers.
And in the second machine, we run the reproduce of the build.
And the clients come over to you two or two to understand.
We can present it to you.
The clients show a green shield to say, yes, confirmed.
That might be delayed.
It comes later.
Because runs on a separate track.
At FDroid, it is integrated with the build system.
FDroid ships their own builds.
And if they are reproducible and not the reproducible,
they ship the AP case of the developer,
but only after the build process.
So the build process takes a bit longer.
There is, if a build fails to be reproducible,
it's the AP case with an FDroid one bishop.
So you get extra security.
It could potentially be a problem.
So we hold it back.
It is the Android.
You get the AP case.
But you see the green shield.
The shield is not there.
Our clients in the future will have a feature that says,
I only want to install it when it's was confirmed.
So we're giving back freedom to the,
because I was using our repository to say,
you can decide what level of security you want for yourself.
We don't tell you how to do things.
We just give you the options.
And you can decide.
It is the Android.
We use a tool called RVT lock.
That's a reproducible transparency lock.
We have the resets checked in the repository.
And we have the locks.
Also checked in is the JSON.
So we can confirm that the build was run.
That's the entire build lock.
And we got a file hash of A.
The original was the file hash of A.
So the bow is met.
It's a char 256 match.
So it's reproducible.
I've tried to use this,
I've tried to offer for the builds and for the reproducible builds.
But they have also second setup.
Complete later.
Both sites using only for tools for that.
That's important for us.
For both sites of us.
And it is the Android.
We use podmen containers with different images.
We have a used flexibility that way.
We can use debion bolts.
The book says you don't want to jump in,
no.
To match environments as possible,
it's którye to simply replace the files.
If you use this varnthus debion Bookwall for the main.
But there are very services are вид vengeance.
For the last year,
the boot will go for the buttons,
Figure out which apps could be reproducible to signal to the developers.
Hey, don't you want to set it up as reproducible build?
It's not easy to switch between the two at F-Droid because if it's not
reproducible, it's signed by F-Droid.
So if you switch to reproducible build, everybody has to answer and reinstall.
Accept yourself as a separate track, but that is not part of my talk today.
Is the Android runs multiple burrows, including independent burrows.
So we run two burrows ourselves.
There's a third burrow run by an independent party to confirm our builds as well.
So we can plug in more.
If anybody of you is interested in, just meet me up at UD2O2 for details.
It's easy to set up a burrow, we have even a tool for that that sets it up in five minutes
and try to build the same business, all tools you need, whatever you want for it.
So we can have independent burrows, proving our builds independently once more.
So the developer can say, I have signed it.
He has proven it was proven by that person that he has proven it right.
And after all, it's currently as far as I know, only after its own builds are
wasn't built, and also their vacation system.
So there's a system you could also set up.
I'm not involved with the details of their build system,
but it's also not part of my talk today.
So we have an independent verifying developer's build.
We always ship the developer's build.
After all, it's independently verifying developers' builds
for apps established as reproducible builds that have tried.
And otherwise, that other check, which could be established as reproducible builds.
And they have their own ones, which they build twice to say, well, okay, it's the same outcome.
So nothing in between our research,
I'd be always shipping the APK from the developers.
And after I'd ship the developer's APK efforts reproducible,
and set up as reproducible builds, if it's not set up as reproducible builds,
they ship their ones.
If it is set up as reproducible, but doesn't reproduce, the ship nothing.
As I said before.
So, what are the challenges with reproducible builds?
I can only stretch on the surface, because there are many challenges.
So I want just to mention the main ones when setting up the first reproducible builds.
I can go, things can go wrong later with the next release,
or next to next release, we can't cover that all.
But I want to mention some, also independent of the build frameworks used.
It's very easy to handle Java and Kotlin,
basically you just need to match the JDK version.
So in most cases, it's a fashion to set up a reproducible build.
Exceptions exist.
More complex, the gets when you have native libraries,
like the Flutter, NodeJS, React, Go, Rust,
do name it, Expoh, whatever, because those frameworks and bet the build pass into the libraries.
So you also need to match the build pass.
You need to match the CMake version.
You need to match a lot of other stuff.
So it gets easier to set up a Java application, maybe in 10 minutes,
15 minutes, depending on the build runs to confirm as it worked out.
If you set up the Flutter version, the Flutter app for the first time takes maybe two hours,
to get it right, to match that.
And then you also have the developer has installed in the NDK.
So libraries get stripped of the symbols, but all builders don't have.
So all our stone strips, we have a different outcome.
So you need to figure which version of the NDK has the developer installed on the laptop to match that.
So that's getting the more complex, the more native stuff you have in.
Build pass and bet it.
Pass from Windowsburg, it's very, but try to match C code on whatever, on a Linux system.
No, we don't put on Windows.
We build on open source.
So no Windows, just to scroll things out, maybe in for some, like to come in on fresh air.
There are ways sometimes to get it matched by saying we want,
not absolute pass, we want relative pass, doesn't always work out, gets difficult.
Different version of different build SDKs produce different output, Flutter.
You say, yeah, we always use latest for reproducerability.
No, latest today is not the same, latest moral.
So somebody, you build today, we build later than you, maybe in between there was another update from Flutter.
Another one wants to check if we are doing right and build the ones later.
There are two versions later already, completely different output.
So you need to pin the versions even of the environments of the Flutter, exactly 3.88.8, whatever.
Not just latest, or greater than between and doesn't work.
Okay, the most, some I mentioned already, the links, if you download the PDF, you can follow the links.
We have the first basic rule that is adding up for 80% of failures, if not kept.
Always build your APK from a clean tree, clean tree, as a commit, your release tag points to.
Which order you do that if you first put and then tag, doesn't matter as long as I match.
And clean tree means no local changes and also no remaining artifacts from previous builds.
Because they cause already different outcome.
That makes up for 80% of the failures.
Some of that are framework and autocorrect, but not all.
Switching between Windows and Linux.
This version is next version, the other one.
We have to correct the linebacks, we can do that. After watching our APK, we say, okay, if we play Windows and other linebacks,
it's identical again.
But it's a changes with each version, because it means every time manual intervention to correct the builders have.
So that causes us headache and unreaded additional time Spanish.
Introducing more or less complex and forced signing.
It's nice if your build setup automatically signs the APK, but we don't have your keys, and we shouldn't have your keys.
We don't have your credentials, and we should never have your credentials.
What we build is an unsigned APK.
It's the APK is complete, we say APK sick copy.
Copy over the same, as other signing stuff from the original APK, which only works if it's otherwise identical.
Or the second trouble fail.
That's how we test if it's reproducible.
Can be copy over the signature and do we get the same head?
So if you introduce complex signing, the build breaks.
Oh, I can't find the signing key.
Please don't present.
Whatever.
Building locally with stuff from setups and specified.
Your build header checked in and to your repository says,
I use Gradle 8.14.
But local you build was at this grade online.
And it doesn't matter.
We get different outcome.
In this case, it's very easy.
It's one file that says what Gradle version was used to be needed to patch that file afterwards.
But it's extra work.
You have to keep it consistent to increase the chances of easy reproducibility.
And third of the build timestamps are ugly, ugly, ugly.
Some frameworks do that like a century, loves to do that.
You build at a different time as we do with those.
And that will always be the case.
We want starts and closely and ends and closely as well.
Can't work out.
So never to build them.
Definitely kills a reproducible build.
Sometimes can be fixed by patching it, but a lot of extra work on this side,
reproducible builders.
Using non-LTS JDKs, like 22 or using 19,
it's not a long time support version.
The versions in between is rather tried to run stuff, which gets three
bulk with the next one and doesn't work 100%.
Yet and so on.
So always stick with LTS versions, like a power way back 11, 17, 21.
And 25, no.
Let's see RTS versions.
L marked RTS.
We only use RTS versions.
So if you come with a 22, it's like playing the lottery.
You can be lucky, but usually you are not.
So part of that, what under the developers
up developers should do, we already had.
We have this page with reproducible buildings for developers,
like the basic first rule we already said,
aforementioned one, and built-time,
related things, like build times, and so on,
because they are usually making it more complicated together reproducible.
For release builds, about canaries, like just talk about the RTS versions,
but you use for test builds, you don't care.
But to release builds should be with a stable framework.
And yeah, basically avoid the problems I just mentioned before,
and that's the basic ones we would like to give.
We have a couple of related links for that.
We have our reproducible build documentation,
including launching what you can do if it fails,
how to look for arrows where you first,
and which order you possess things in what is fixable, and what is not.
And we have our reproducible build framework.
The Arbola set up leads you to the script at set before,
letting you set up a build within five minutes,
with the items to specify, basic only the builder,
or also the tools for checking white failed,
or the repair tools, whatever you want.
You can set it up in the config, and five minutes later,
your builder is running.
Preproducible builds should of course get a reference,
because they are the ones who collect and maintain all the things
around reproducible builds.
Effdroid also has a documentation for reproducible builds.
We want to mention here, and our clients,
draw the fire and near store,
shows the status of reproducible builds up at, is you enjoyed.
OK, I guess I'm in time.
Ah, good.
You can ask any questions you have.
Partly, if you include them as, what is the code?
Pardon?
Ah, OK.
If you cover reproducible dependencies,
that means we reproduced the Arbots off,
but the dependencies, what is about the library's plug-in?
If you have some insight as a good sub, not sub-tree, not sub-tree.
I'll sub-tree, you don't think it.
Sub-tree, of course, we built a sub-tree.
If you have a setup, like, OK, here's a get-clown,
this repository, and build a step, we covered.
If you come and say, well, we have the, whatever,
J&E library in there, no, that depends on the setup.
We try to have everything covered, but other than F-troid,
which says we want to have everything covered, or we don't do it.
We say, you really should, but we don't force you.
But if it's too many of these things, we say, sorry,
it's not all part.
It's about open source, everything should be open.
Answered?
All right, that's the question.
Is it OK?
Yeah?
We used some board-pack numbers of both hardware,
that we need for such a, we need some board-pack numbers
about the hardware that we need for such a builder, if you want, you know?
Do you cover what?
Ballpark numbers.
Or do you use a 10-year-old server, which is for us?
Ah, OK.
No, it was a 15-year-old.
Actually, as I said, we build a supportment containers.
We can name the hardware we have.
It's running on an AMD.
It's hosted by one who provides all servers and everything.
So it's not something you buy at a medium market or something like that.
It's not high-end enterprise hardware.
It's medium software, medium hardware.
But one of our builders runs on custom hardware from a local provider,
just to couple together, something like that.
Right.
Do you do cluster-off?
No, we have plans on this direction, but not yet.
But if your question is, can I set it up on my own cheap laptop?
Yes, you can.
All reproducible based inside the containers.
So the images we use, the containers we fire.
For some stuff, do you need to plan for example?
Yeah, it goes a bit deep inside.
We have to cover CPU count and things like that.
But that is a bit more extended deeper topic.
I can cover that outside, but we have just one minute left also.
Times out?
One minute, okay.
So one more question.
The time stamps show up in the differs group.
Pardon?
Ah.
If you, if you, if you, if you, if you, if you, if you expect hard times from
you, if you expect hard times, then if you extract them from binaries.
Of course, they end up in the binaries.
That's where the differs group shows them.
So we use differs group to discover them and then try to patch them.
But we can't patch them in binaries directly.
It's more complex to explain.
What we basically do is afterwards extract them from the APK,
fix the times, then put them back in the APK,
and check if it's then matching after your real eye
everything.
Can't compress it otherwise.
It's not as sophisticated, really big sophisticated.
It was tricky stuff, but it's just, it's pretty much simple.
Okay, I get the signal times up.
I thank you very much for your interest.
If you have any questions, I'm available to you at UD202.
And see you at our stand.