Okay. So, hello everyone. I'm Gabba Boscoge. I'm going to talk about name resolution
in package managers from Lab Produce Beat Perspective. This is going to be rather practical
and like side-by-side comparison of three prominent package managers and it's not going
to be so theoretical. I'll leave this to the next presenter and yeah, just a few words about
me. So, as a judge, day job I work at Roche. In the Rochlinux team, we are providing the
IBM-based distribution for medical devices. I also work as a geeks developer for roughly
a decade now, which is a functional package manager. And yeah, as hobby I like astronomy and
I like sailing. So, let's get into it and see what we have. So, let's first have like a few
definitions cleared up. How I'm going to handle packages in this talk is that we are thinking
about them as a triplet, the triplet of package name, a package version and actual data
that is comprising this package. This is not always like 100% true. For example, you might
end up in the unfortunate situation that the same name and version is actually designating
multiple different packages. This is not so good. Yeah, then what is reproducibility? Reproducibility
is a property that when you are like building the same source then you get the same binary.
This is a pretty straightforward thing that we would like to achieve. It is not always working
like this. But why is this important? Why do we want to have this? That's because if you
think about the complexity of contemporary software, just by having a look at its binary
representation, it is very hard to come to any conclusion about what it actually does.
That's why we are preparing to work with them in source form. But then viewed like to
establish a correspondence that, okay, this binary is really corresponding to this source code.
But now with the current tooling that we have at our disposal, the only way to
actually do this is to rebuild it from source and then see that we are coming up with the same result.
Right now going backwards does not work. Yeah, it works for like small stuff, but for typical
software it does not fly. So yeah, when we are talking about reproducibility, we are talking
about different layers of reproducibility. We can have like a layer on package level that, okay,
this package is reproducible. If I build it from source then I get the same package. We can also
talk about this at different levels. For example, if we are talking about reproducibility as a system level,
then what we expect is that from the same definition of the updating system, I got the, let's say,
same set of files on the carbrive of the system. And for this to happen, we actually have to pin
the whole transitive dependency graph of the system image. Because if there is like just one version
is much anywhere, it is not going to be a reproducible system image anymore. Yeah,
why do we have a name resolution in packaging systems is that we don't really want to encode
this whole transitive dependency graph with all the versions for every package definition.
Most probably in the first place, we don't have enough information to do this. And then we
would duplicate also like an image amount of information. So we have to simplify somehow and
be able to express the dependencies in a way that we are referring to like not a complete
three-plat of the package. But let's say we are just speaking the name or the name and the
version of something like a smaller piece of information to identify this package.
We would like to essentially have something like this that we have a package called name
and this depends on this and this and this package. And also because we want to be reproducible,
we want to have this process to be deterministic. In most cases, this is not deterministic.
There are a few approaches that are taken, but usually it involves like creating a candidate set
of packages that we can resolve the dependency to and then somehow selecting one from them.
Usually let's say the latest version. But then this in latest version might depend for example on
when the package repository update was run the last time on the system and this kind of things.
And this is not really acceptable if you want to have a reproducible build of the system.
So somehow you have to circumvent this thing. And the other thing is that we don't have to
don't want to author all the packages that are like existing in the world because that would be
like super annoying. We want to share them somehow. So we want to have a place where we share
that okay, this software can be packaged like this and from this package definition you can somehow
build it or you have a binary that you can download that is going to correspond to that particular
piece of software and you can install it on your system. Yeah. So we are having like repositories
which is like called in a different way by every package manager I think that exists.
We already had this that for example in Rustware calling these the packages we are calling
creates and repositories we are calling registries. But these are conceptually the same thing.
Yeah. Okay. So right now what we are going to look at is how we are defining this process.
First we have like a bunch of configuration that is like the active configuration of the package
manager. One piece of this information is which of these repositories are used.
Yeah. So where I am sourcing the package definition from and this is like a common piece of
information. It is done very similar in each package manager. You have usually one or more
configuration files with information like this. This is an example for the Debian source definition.
You read an example for Cargo how you define the registry there. This is an example in
Geet how you define a channel there. Yeah. And then all of these repositories have some kind of
underlying data structure. So here in Rust Cargo I am pretty sure that there are a bunch of
people who is like much more knowledgeable than I am because Cargo I am using only as a user.
This is roughly what I know about it that you are pointing your you are out there and that's
more of the information you pass there. In Debian there I know a bit more about this.
I would say that I am forward user. So here you have this sign by
Stanza which is basically pointing to GPG key. And then in Debian repository you actually have a
release file which is signed by this key and it is containing an integrated database of the actual
state of the repository. Yeah. And in Geeks what we have is actually each of these channels are
Geet repositories. And then basically there we are using the security model of the underlying Geet
repository. When we are accessing it we have of course the option to have like TLS. And then in the
content of the Geet repository we have the Geet commit signing and all these additional security
properties that Geet gives us. And what is interesting about this is that when you are thinking
about this then the Cargo and Debian registry are like not really having their full history at
your disposal. Yeah. On the other hand what Geeks channels do because they are Geet repositories they
have the full history available. So that means that basically you can time travel backwards and
forward a longer channel definition. And then you can have let's say a package from like five years
ago pretty easily pulled in. Yeah. And also what this means there is it's pretty easy to actually
pin down the whole repo state. Because you just notify which channels are in use and you write
that like the Geet commits that they are at. And then you have a full fully reproducible repository
state. And in the other cases you have to actually resort to some kind of endlessly important
information to do this. Yeah. And then how does a package definition look like? Because this is also
important from the reproducible perspective. So let's have a look at how Debian does this. Debian
package is like defined by quite a few files. And the control file is basically giving you like the
name of the package. And it also tells you is dependencies. Yeah. And then you also have the change
log. The change log is essential telling you the version of the package. More or less. Yeah.
And then you have basically the rules as the building formation, the source format because Debian
is able to like build the packages in different ways based on the structure of the actual package
definition on the Debian side. And this is roughly building up a package. Yeah. So most of these code
are snippets. They are not full definitions. They are just the parts that are like important from
this perspective. Then as so. So. Yeah. So in Rust you do something like this. This is in a
single file. And then you have the name and the version. And then the dependencies. So this format
is like way more compact. And here in the dependencies you actually have a bit of
linions. You can say that it's like bigger than this, bigger than that, but slower. Yeah. So you
can have like a specification of the versions that are allowed for the resolute. And in this form
it is actually tying the versions, but you can have like more in for here. And then this is how
it looks in geeks. What you see here is that the package name is basically the plicated.
So the name of the package after this name stands up is what you are referring to the package as
on the command line. And the top one is actually defining a variable. And this is how you
are referring to the packaging API. And then in the inputs this name that you have there is corresponding
to these variables. And then the name resolute in geeks is basically the resolution of the variable
binding in the underlying programming language. That means that when you put this inputs list
geoc, then you already pin the version to the exact one, like a very similar definition to this form.
And this is always like fully pinned. There is literally no way to express anything besides
this binding construct. That means that this structure is completely reproducible from the
state of the repository. So a few observations, the BMP packaging requires
quite a bit of work. I think everyone who ever did this knows that this is not like
three of feet. It can be done pretty quickly. But yeah, it can be a complicated list.
Yeah. And the geeks package is basically this is what I told about is that the variable binding
resolute is already giving you a full graph of type dependencies.
Then there is usually a last step which is missing from geeks is that once you have the candidate,
how you actually find out which ones to manifest on the system. So normally in the BMP system
you are manifesting the latest version that is available in all the repositories for the given package.
Except if you are having the conflict or if you are defining pinning.
So that means that if you pin every single WN system you get reproducible build. Then you have
like one of these pin definitions for each and every package on your system.
This is perfectly doable. But it's pretty inconvenient.
Then in cargo we are creating the lock file. This is like pretty typical single bunch of
other package managers are doing this thing. That basically when you are building,
it is like resolving the dependencies, drops that into a file for you. And then that file is like
taking precedence for the future builds until you say that you want to update it.
This is giving you like reproducible set of sources to build from.
And then geeks we already resolve basically at the finish on time.
By this trick that we are actually putting the fully fledged package objects into the graph.
Yeah, then next thing is that when we have these repositories then there are a few
problems to take about. So sometimes the repository state is affecting your build.
You can work around this in all of the systems. It is like pretty much dependant on how convenient
to work around this if you want to reproducible build. And then the other thing is that the user
or the client side of the thing is not having control over this state.
Yeah, so if you get in control over this state then there is no problem. But usually this is
not the case. You cannot tell, let's say, the WMP people that were pleased don't publish
the tricks anymore. This just does not fly. Yeah. And then there are a bit of a problem
so we do look five base things. The biggest problem there is that you are not like require
to publish your look file. And then there is like a very hard to figure out. For example,
what your test suit was run against. Yeah, because it gets detached, you cannot like reproduce
the same thing because you are missing this information. So one way to improve here is to
increase committing the look file into the source control. Then there is a third thing.
This is affecting all the systems. Sometimes the like upstream just goes away.
And then you don't have a way to rebuild it anymore.
Yeah. So what is the benefit that is like provided by having central managed repository?
Is that the repository managers can enforce certain properties. So this is for example where
crazy.io can shine. Yeah, because they can say that okay, let's say we are not allowing to remove
stuff. Or we are not allowing to push a different binary corresponding to the name and version
and these things. And if this is like centralized, then this is giving you the possibility to do
this. And then the situation is way better than like on the wild west where everyone is
pushing everything into the repository. Yeah, but this still requires some coordination.
Yeah. So approaches that can be done when there is no central management is basically two
either you take over the repository to your side and define it state completely,
which is one approach that is like typically applied. And then the other one is
that you can add additional configuration information to always resolve to the same name. Let's say
pinning. Yeah, but this is usually like a lot of work. And then it does not really
mitigate against this package removal problem. Yeah, actually taking over the repository does
because there you just say it, you don't remove it and that's it. Yeah. And then in functional
package managers what we have instead is that this is like a completely tied
the repository state. It is always at the table to the actual build. So you can have a provision
file that is like telling you that it was built by having the package manager pointing to
these commits on these channels and that's it. And then we also do engage something very
interesting about the removal of the upstreams because there we are still relying on the source at least.
There are like a bunch of systems for example software heritage is allowing you to send the source
code to them and then they are archiving it. And before the upstream you are as are going down
software heritage still have a copy from which you can build. Yeah, and also basically in
this functional package managers the binary packages are just a cache. There is a promise,
let's say, that you can build the exact same thing from source if you want, but we are providing
in our build infrastructure, something that we build ourselves, feel free to track it, otherwise you
cannot our signature and then trust the binary is that we build for you. Yeah, so packaging effort.
So usually for language package managers it's relatively low. They are made to be frictionless with
that particular programming language. Yeah, but there are two things here. One of them is the diamond
problem when you are having like two packages depending on the conflicting versions of something that
is like single package. This is a 3K1 and then it can result in all kind of interesting situations.
You can avoid this for example by running the whole thing in a container, then they are not
touching each other and then we are happy. Yeah, and then there is the infrastructure problem
with language specific package managers, which means that basically the things that you built
are still depending on the actual versions of the tool chain whatsoever that is like not defined
in the package definition. And yeah, basically here you want to have an isolation.
And then the effort is usually like you create the metadata build instructions and then you
test the package and then you can release it. It's very similar to functional package managers.
And then there is a maintenance effort which goes into like how hard it is to update this.
Yeah, and once we have like all of this, then we are ending up with like some interesting data,
like how well this thing actually works. And what we get is that now we are having like a
95% of reproducibility for almost every thing, which is really nice. There is one missing point,
which is really bad as of now, is that most of the time we are going to build from really
Starbucks on the distribution level. And then the creation of the really stable from the sources
usually not reproducible. This is something that we want to address in the near future.
And then my key point about the whole thing is if you make this simple for everyone,
then everyone is willing to have reproducibility. So let's go here and try to make reproducible
be a CEC. Any questions?
So it doesn't seem to be the case, thank you very much.
Thank you very much.