Hello everyone, it's quite a room.
Thank you for coming, my name is Jean Amarca and Nick Suez contributor, but also a PhD student
and a researcher at Telecom Paris.
Today I'm going to present work that is more research oriented that I did and the supervision
of Teosimirman and Stefano Zaccholi, which is about Nick Suez's reproducibility.
First of all, what are we talking about, what I'm talking about, what I say reproducible
is the conception of reproducibility defined by the reproducibility group.
So basically, build the reproducibility, build by the reproducibility, which is basically
if given the same source code and improved send environments, you can recreate the same
build, exact same bit by bit identical as another person that would do it and another computer
basically.
Build reproducibility is a concept that is historically pretty much associated with Nick Suez.
Maybe if you have heard about Nick Suez before, there's something that comes to mind when we talk about Nick Suez
is it provides reproducible builds.
This was even directly on the front page of Nick Suez website for years.
So this is a snapshot of 2023 when the wording changed a little bit, but it was like a big marketing argument of Nick Suez
for a while, but it actually not something that is through in a general case.
I'm sorry about the display that is being a bit messed up by the display, but you could easily write down an expression.
Oh, the mic is on, you should have known.
Apparently there's issues with the audio on stream, but it's not your fault, sorry about that.
We could easily write an expression that will generate something non-deterministic.
So just for example, this expression that takes value from the random environment viable will generate two different artifacts when run twice.
So here you can see that's actually Nick Suez doesn't give you a guarantee of build reproducibility.
So it's something that has created, I would say, a little difficulties with people from the reproducible builds group, which are actually the people providing the patch and working with upstream project to make them reproducible.
Because like Nick Suez has used it as a marketing argument for a long time.
So it creates the question, how reproducible truly Nick Suez is?
And to answer this question, there is something you can do is you can go and check out the reproducible.nixware.org page.
Where there is actually some monitoring that exists of Nick Suez's reproducibility.
And on this you can see that the monitoring is done on the ISO images, so the minimal ISO image and the GNOME ISO image.
And you can see that we have excellent reproducibility rate around 100% 99% for this.
And you can even go and and for the the packages that are not reproducible, check out their diffoscope.
The diffuscope is like a visual diff of where a binary artifact differs.
And you could try to understand thanks to the diffoscope where the difference comes from in the small amount of packages that are not reproducible.
The problem here is that this is only monitoring the small subset of Nick's package, the ISO image is the small subset.
And we really have a more general question of what is the proportion of the globality of the Nick's package, a repository that is reproducible.
So why is this question important?
It's because it could have the positive impacts on software supply chain security.
So basically when you are an X user, you could compile your packages on your computer, but this is quite a platform and celly, completely invisible.
So you don't always want to do that.
And sometimes you actually want to download the packages directly from the cache, it's what's most Nick's user do.
And when you do that, you kind of have a problem of trust.
And the problem of trust is that why would you trust the package, the pre-compiled package is that you download from the from a cache.
And it's it's visible that that hacker takes control of the of the ninspiratory cache or the builders and and start providing you instead of the right packages.
Free-compiled packages, it's sub-providing you backdoor package and you could do nothing because then if it's as control of this server, it could sign the packages and then you trust it and it's game game over.
But free-flow-producible bills, if you know that and you have the property that this package can be compiled on several machine and you obtain always the same thing, you could ask other trusted builders to compile it on their own.
And if they obtain all the same result, they can provide you with the hash of this result.
And then you can download the package from the cache and check if you have the same hash and be sure that you are downloading actually the right thing.
So this is why we care about super-visible bills because if you don't have super-visible bills, then you cannot do that because you can ask several builders to compile your package and then they obtain different stuff and then you can say nothing interesting about the result.
So, we are interesting, interested in the reproducible bills, we are interested in how much of these packages are reproducible, but we don't know.
So now, science, come to the rescue and what I'm going to present now is the results from a paper.
I'm going to that is going to be published on a conference named mining software repository this year.
And that's his goal, does functional package management enable reproducible bills as scale? Yes.
Well, this is what we did. This is a pipeline summarizing what we did.
Basically, we took some revision of Nick's package, 17 revision that we were taken from 2017 to 2023.
And for each of these revision, we did in parallel the same thing that's the Nick's continuous integration hydride does.
So we evaluate the release. Nick's value, which is the file that lists all the packages that must be built by the continuous integration.
Then we built all these packages like either of us.
And then, when a package is built successfully, I draw a product to the Nick's binary cache.
And us, what we did is we compare our local compilation result with the historical package that is in the Nick's cache.
So when you do that, there is seven things that can happen.
Either you obtain exactly bit by bit the same artifact, in which case we can say that is bit by double zero, which is what we want.
And otherwise, you could obtain something different here or even fail to reveal the package.
And in this case, we have an analysis by client where we do two things.
First of all, we compute, we generate the default scope, which is the data structure I showed you before, where you can understand better where the reproducibility is.
And with the default scope, we try to analyze it to understand where the package is not reproducible, why it is not reproducible.
And we also have something that I will not read cover today.
We also, by section next package, in order to understand when the anore reproducibility is fixed, which we switch to communicate and in which pull requests and try to understand with the context.
More about how the anore reproducibility are fixed in this package.
So, this is what we found out.
So, this is the proportion of packages that are reproducible in blue or in red, buildable but not reproducible from 2007 to 2023.
And we see that there is an increase of growth in the reproducibility of next package in this time period.
So, starting from about 70% in 2017, we are now reaching about 90% 91% to be precise in the end of our studies in 2023.
You can see something interesting here that I will cover just after.
If you look at this visualization, like more like an absolute one.
So, it is the same result, but instead of proportion, you have absolute number of packages.
And here you can see also the sheer voice of an example from about 20,000 packages to more than 70,000 packages in the time period.
So, in with these goals, we also have a growth of reproducibility.
With one metal exception here, so here you see that between these two points, there is a growth in the number of packages.
But there is a drop in the number of flow-producible packages.
And at this point, we try to understand the cause of this.
And so, we did basically the packages, looked at issues, pull requests, and everything.
And we found that at this point, around here, there was a pipe upgrade where they broke the bio-collegingeration.
It was not deterministic anymore.
And it was actually tracked by the reproducibility people in the package.
They went upstream, they discussed.
They first applied the patch, and then the ups and fix, and they applied the update to people.
And then the regression of a fixed.
This is like the one main regression that we found out in our experiment.
So, this is the first key result that we have.
Currently, 91% of packages are built by a bit for reproducibility.
What we did afterwards is we worked with the diffoscope.
So, basically, for each non-opposcible package, we generated a diffoscope, which is something that can be pretty long.
We put a stop time out at five minutes.
We dedicate five minutes of compute time for each non-opposcible package to generate a diffoscope.
And we got a big set of diffoscopes.
I think you're more than 100,000 diffoscopes.
And on this diffoscope, what we did is we tried first to manually inspect them to see if we can find
like things that you can identify regularly in diffoscope, like common cause of non-opposibilities.
And once we had this, we wrote a realistic that could automatically identify the discosuses.
So, these are like our four realistic.
The first one is dates.
Number one, recommendation in the Opposcible website.
You see, like, stop embedding your client dates in the build result.
And instead, I'm better, like, the starting in timestamp date.
Like, in January, 19 night, or remember the date.
But anyway, always, and if you have to embed the date, always,
I'm better, like, the same one, the deterministic one.
Instead of the date of today.
This is our main cause that we identified.
And it's the blue line here.
And the second one is the, is you name output.
What we call you name output is basically, when in the final artifacts, you embed some impure data
about your builder.
For example, the camera version is running things like that.
So, we detect it quite a lot of them. It's the red line here.
Then there is an environment viable.
So, when some part of the environment looks into the final artifact.
For example, the number of course, that mix uses to build this specific package.
This is the green line here.
And the build idea is some, some ecosystem, like, go,
will generate a unique build idea for a package, unique, but not deterministic,
that they will embed in the final results.
So, this is what the four scores of non-requisitiveity that we have found.
Interestingly, there is, what we can say is, even, like, the most common cause
is, even is present a lot in, in the current mix package,
which means that there is still some, some logging fruits that we can fix
to improve an exponential possibility.
And with those four juristic, we covered, like, about 20% of the cases of non-requisitive
which means that there is still 80% that we, at this point, don't understand well
and we think are more difficult to categorize and understand
and we might need, like, help of expert folks.
A conclusion, yeah, between the opportunity at this point is about 90%.
This justifies, putting some effort to use this as an asset to improve
self-horsal fighting security, which is the way I'm going to do in my next works.
Thank you very much.
I don't know how much time we have for questions, but I'm happy to.
Time for questions.
Who has a question?
Yeah.
Do you need a mic?
I will repeat the questions.
And then, the speaker will be picked up.
When you show the e-plot, showing that the number of packages is
both increasing and the number of, yeah, this one.
No, the second one.
This one.
Yeah.
Are those the same package?
I mean, the red ones are always the same, because the number seems so constant
besides the people.
So the question is, basically, in this red area,
are the non-requisitive package always the same?
And I don't think we precisely analyze that.
This red area is shrinking.
So at the very least, some of them gets fixed, that we know.
But we don't know exactly if there is like a small village of irreducible
packages that refuse to get the reproducible that are always the same here.
But that's something we could try to determine pretty easily.
So I'll follow back on this.
Okay.
There's another question back here.
What's your question?
Yeah.
Is there a red management?
You soak about the thing, a bad store or a back floor here.
You learn what to pay with P.K.
The deal.
But the main order, right in the sprites,
we're heading no, none revered with Reversive.
So the question is, what's the main other trend that can be identified
in non-requisitive, a bit?
So actually, there is not, non-requisitive bills are not threatening itself.
It's a missed opportunity of securing the software situation.
Yes.
What are the worst compilers for this win?
Yes.
What are the worst compilers for the reproducible bills?
So I think we covered this, especially in the precision in the paper.
But the ask-a-lico system is very bad for the reproducibility.
It's about 60% packages of reproducible.
And this is, we think that it's going to improve.
Now that in a, in a, a great day, they added a flag to, to have deterministic, by,
by generation.
So it would be interesting to know if this is going to have an impact.
As far as I know this flag is also not using an example so far.
So we'll see how this goes.
Yes.
Now it's certainly good.
If, and good, particularly huge, you will compare the, the either I
Sporey convenient to what you will be at a big time.
So, it's what is added is the team as as what, that is,
by either and the agency is tested is.
You assume that it needs test of reproducible.
But is it not necessary to write?
because the only need that in the app to this is the verification of it to pay it to you.
Yes, so I said, how confident are you?
That's the impact something is still the same then, it's really very positive, man.
Yes, so does you try to be on different machines with different questions and clear?
Yes, so look at any of you.
The question is, like, okay, the question is, basically, we build once.
What has been built historically and we compare and this is a weak proof that is actually a positive or so, how confident are we?
And my answer is, if you cannot really prove something is a positive or because you try,
it's much easier to prove something is not a positive or you need one example and prove something is a positive or is not really a positive, but
we do only once, years after on different hardware, even not running in the XOS and we find the same thing.
We think it's good enough proof that this is a positive one.
It makes better canals at back there.
So I think we don't have any more, like this, one thing that I also wanted to add to that,
or a few things, the next people can come and set up their laptop in the meantime.
So there's, you had this talk today, thank you very much.
Another round of applause for the thicker time.
I also want to...