We are going to tell you a bit about the backstory, about our current implementation,
and about future planned improvements for the binary packages.
Okay, so what happens if you emerge a package from the source?
I have to explain, maybe a little bit, our package manager is called Portage.
The main command is emerge, and obviously people are emerging packages when they install them.
And you all know, Gen2 originally, as a source-based distribution, that means what happens when
you emerge a package is you fetch unpack and patch the sources, you configure, compile,
install to a temporary directory, then there are some quality assurance checks.
The binary is stripped, the bug information, for example, is separated out, and as last
step the files emerged into your main system. And that's not so much different from other
package bills. Of course, we are skipping over a lot of details here. The dependencies are provided
by the main system, by the user system. Yeah, we did have binary package support in Gen2
already for a long time. These were the so-called X-pack packages,
basically a combination of a compressed tar archive and a custom trailer for metadata.
Nice thing was, it wasn't single file and tar can unpack it, but there were also a lot of
disadvantages. It's a custom binary format. It relies on undocumented tar behavior.
If you want to access the metadata, you have to use special tools. And there was no specification
versioning, not easily extensible, no clear support for any package signatures. So this had to be improved.
The other thing we should maybe know about Gen2 is, it's a source-based distribution.
Packages usually have been compiled on the user system, and that means there are no global
rebuilds ever. So there is no such thing as a mass rebuild. And we need to make sure that nevertheless,
the packages that are installed on the user system are all consistent. Examples for how problems
creep in are, there's a library upgrade, which increases the library's so-version.
There's a pull update, which means the package path for pull modules, changes, and all this
has to be addressed somehow. Then another problem is dependency automagic.
I mean, you all know what happens if you run a configure phase. I mean, this is like a picture
from image magic. You can see that the configure is checking for all sorts of libraries and
detects whether they are present or not present, and which versions, if you don't
have this under control, then your build result can depend on what packages and what versions
exactly are installed on the builder. And that's dangerous because if you then make a binary package
and install it on a host where maybe different things are installed, it doesn't match together anymore.
Example on the builder, we have a certain library, but not on the target. All we have, for example,
different ICU versions on the builder and the target with different so-versions. So build systems
can create so-called automatic dependencies on libraries and other dependencies that they find on
build time. Autoconf is actually a pretty good example because there you can override this automatic
detection really everywhere. And specify it directly. Yeah, to address all these problems,
there were a few old-style gen to help us. They are now by now all deprecated. So the most famous
one maybe is the rebuild. It just finds a binary that has broken dynamic linking,
identifies the affected packages and then rebuilds them. And similar one, it's per cleaner,
it finds per modules that I installed in the library parts of previous not installed any more
per versions and identifies the packages and then rebuilds them. The problem with these both is
they are essentially trying to fix what is already broken. Because if you have a pearl package
with modules in a directory where pearl is not searching anymore, well then it doesn't work anymore.
And it's surprising how much can break if, for example, some part of get text suddenly fails.
The second column is kind of the next approach. The preserve library is mechanism.
There it was integrated into the package manager. So basically on an installation of libraries,
it checks whether still there's any dynamic linking treat. And if there are still programs that link
to a dynamic library, the library is not removed, but it's re assigned to its reverse dependency
in the system. And at the same time this package is recorded somewhere and with the special
syntax you can then rebuild these packages and this way remove dependencies on obsolete libraries.
That was already an improvement because it doesn't usually leave you with a broken system.
But still it's kind of manual act and have to be replaced eventually.
And this here is what was introduced to fix both of these problems.
So called sub slot rebuild mechanism. Let's not go into detail what a slot or a sub slot
precisely is. It's a property of a package and we declare dependency in a special way.
You can see this here. This is strawberry, the music player. You see the dependency on ICU.
It's using this particular operator here. What that means is
any sub slot value of ICU fulfills the dependency. But if it changes, strawberry has to be rebuilt.
And inside the image of ICU, we have here the declaration of the sub slot,
which is increased when in this case the package version changes.
That is in our specification. It literally no explanation or no standardization what a
sub slot exactly means. It can be many different things. It can be a library as a version.
It can be just a library major version or it can be a package version like here for Pearl
for example. And here we have for example an update run where an ICU version was pulled in.
So in this case 78.2 and you can see that a lot of things have to be rebuilt and this is actually cut
off the list is much longer. It's kind of obvious. It's an essential library.
That's one thing. Then we get to quality control. I talked about the automatic
automatic dependencies. So the rule in general is no build must depend on the presence,
absence or version of packages beyond what you specify in the dependencies.
So basically we have to tell this system exactly do we want to have an image magic with
JPEG or without JPEG support. Having image magic without JPEG support is not particularly useful,
but still. And so basically you tell this the package manager by also called usefacts
and during the configure phase we then have all these function calls here. For example use enable
and then some flag name that expands into enable flag or disable flag. So basically it tells the
configure phase exactly whether a certain thing should be enabled or not. And this way this becomes
predictable. The same you can do with versions for example telling a program it should use
Qt5 or Qt6 and so on. Last important step was a new binary package format.
We went all the way and wrote up a detailed specification. Now we have basically a tar archive.
The auto tar archive is uncompressed. It has a certain list of members.
First, a marker, then metadata tar, it's signature image tar and it's signature and the
manifest which is a list of hashes with inline NupiG signature. This can be easily read.
It can also be easily identified by file and yeah lots of things we can do.
We introduced gen2 certification authority. So basically 111 keys, level 2 keys and then here
the release engineering key all very well defined. And on the client side we now have a small
utility as long as you don't configure it differently. I mean you can configure anything.
But as long as you don't configure it differently, once you first require binary packages
from a server, it checks whether there's already a configuration in it. It doesn't exist.
Then it generates a local trust anchor key and signs the release engineering keys
that came from the installation stage. The key signatures defined the valid package
signatures and many configuration options. Okay, now let's get to how the actual package building
process works. It's actually surprisingly boring. It's a system the inspawn which binds various
directories and runs emerge. There are some bit here, bit here. I have a subvolumes for
snapshotting. We run daily updates. We are both consuming and building our binary packages
and the various inspawns share package directories. We improved this a little bit by introducing
Steve the job server. The idea was basically we want to use our parallelism in an optimized way.
This is a token based job server that is a new development. It's handing out job tokens to the
various make files to the various make processes that run in parallel. This way we can basically
keep our system at a constant load no matter how many emerge processes are started or needed at
a certain moment. The inspawns have different configurations. We have KDE and GNOME based,
Open RC based server based. We build against different Python versions. We, for example,
sometimes pull in one lucky random package from Stan General Stable. We build a ton of GCSE
and Clang versions because we target developers, of course. Any different useful combination
generates a distinct binary package. Any different dependency specification generates a distinct binary
package. Any different dependency sub slot also generates distinct binary packages.
So in a way this is a little bit like Nick sauce without S3.
Right. Instructions sense. AMD 64 2 identical builders with the baseline ABI,
baseline user and the V3 user, AB64, the baseline AB64 user, and the C-flux are pretty
conservative O2, Prib, two-engineeric and a few error flags. The error flags are there because
in addition many of the system packages are built with LTO to improve package speed and optimization
in the end. Yes, we are aware that this is a pretty conservative setting. This is for convenience
and for stability. If you want to, you can still compile stuff here on your own and then you can
put in all the crazy flags that you want. Hardware, we have two servers at Hetzner,
two times at AMD Ryzen 9 with 16 Zen 4 cores and in total 120 GCSE RAM,
there are also used for the stage builds both native and Qemobased for RR64. We have
one server at Oregon State University, and a pair of ultra max, that's a Neoverse and one chip
with 96 cores, 256 g RAM, donated by ARM, and that's also used for the stage builds both for
32bit and 64bit ARM and for the development. Yes, the result across package categories is pretty
interesting because you clearly see here that with the target developers, we can see lots of
clung and LADM versions, and the rest is basically this year's all also development stuff with GCSE and other
languages and so on. Yeah, and then some other things like tish knives, keylips, this is Libro
Office mostly here are the browsers, we end up with KDE somewhere here and here and the frameworks
are not even listed anymore. But as you can see, it's a pretty large offering. Obviously, we can only
do that for AMD 64 and AR64. We can't do that for every single MIPS variant. So if we are going to
the less popular architectures, we had to come up with a compromise, and that compromise is
we limit ourselves to the installation stage content, which is still pretty useful because it's the
compiler, it's basically everything you need to bootstrap a system. And especially GCSE is often
the largest package that you have. It's usually the base ESR, we export the cache of stage builder,
and you end up still with a lot of things because for example MIPS is 20 independent director
is for big and young little and young for three different APIs and so on and so on.
This is how it typically looks like. Here's an exotic example, MIPS, big and young N64,
and still, I mean this is essentially GCSE and so on and so again, focus on compiling.
And the default configuration we still build from source, but we have all the prerequisites set up.
And if you hand over this parameter, get pin package, it changes and you can see here
the output of an emerge run, the green stuff is built from source and the pink stuff is pulled
from the binary server. And as you can see, it's combined freely. You can also do some
more tuning. You can basically tell it whether to respect use flags or also try binary packages
that have slightly different settings. You can tell it whether you should use binary packages
very dependencies change a little bit. In general, there are very safety faults in the package
last remark may be on this chapter. If you don't grade G-lipsy, then it is a safe way to
to the structure. So you can never do that. And also, no package that has been built against
a newer G-lipsy version will ever work against an older G-lipsy version. That's why the built
process automatically inserts a hard larger than version dependency on the C library into the binary
package. So if you end up with an, if you ever try something like that you end up with an error
message like this. So I'm not ending over. Hi, everyone. I'm Sam and I'm going to talk about
the future parts. So this is something I'm quite proud of actually because we started off with
how emerge handles these things is that you enable binary packages and if one is available
and eligible based on your configuration, we always use it. The problem is we don't really know if
your C flags and so on. They're not part of how binary packages keyed. So then they don't
affect dependency resolution. They're still encoding the binary package, but they don't control
what they're selected. So what often happens is you get people who are quite happy to have binary
packages for most things like Qtweb engine or Firefox. They're usually not patching these things.
These are very large, very sophisticated code bases. Most people do not have an interesting
patching these and if they were to, they'd have to rebase all the time anyway. So what they often
want is they want to use binary packages for those, but they have like their own packages of
choice, like maybe get or something, where they have patches to customize, preferences, whatever,
and they always want to use that for the source. At the moment that's not actually possible,
which is strange, but it's a nature of how the metadata words were used like. So I did what any
good false means anything it does and this is quite a frequent request. Someone came in and they said
hey, can portage do that, can I disable selectively just by package name and I had to say no again
to a very disappointed user. But of course I tried my life and I said do you want to give it a go?
And they said yes and I'm very, very pleased with this and they've actually implemented the whole thing.
So they've, yeah, so they've first they implemented it as like a nice little UX thing,
where these are all local packages and then you have a little G when they're being fetched.
Okay fine, but then more importantly, suppose QT web engine, you want to use it as a,
you want to use that as a binary, but like suppose you've patched some of its dependencies.
You can pass a special argument now or you will be soon on the configuration files as well,
where you can just say I'd really really like QT web engine as a binary, but please leave the rest alone
and that does work. Yeah, this is quite well-seaved to draw you to this.
The other thing that's coming soon is Debug InfoD.
A lot of these things, they're things which, in binary distributions,
you're quite used to because they had to have them from the start, whereas for us, you know,
the default has always been source builds. It will continue to be forever, but we encourage people
to use binary if they wish and if they stood there use cases. So we had to deal with some
house serving these Steveug Info on the side, right? Because if we put them in the binary
packages themselves, it would bloat them a lot of the time people aren't interested. Like,
you'll need to be bugging something goes wrong, right? So first it was just about enabling
their lighties, which is easy to tell GGC, it's always passive-grounds of the linker, and the
it was a bit of wiring, essentially. You had to, I had to implement an important
image that it would produce packed debug tables, which is where, essentially, when splitting the
the final debug information out, we produce a little table on the side in a new hierarchy,
which Debug InfoD can serve. I was very straightforward, Arson produced the proof of concept,
and then I sort of polished it and integrated the importage. So that part's done, and hopefully soon
we'll be able to provide Debug InfoD, which for me was actually a kind of a blocker for the next point.
So LTO. A bunch of distributions now, like, these ship LTO, but default, and LTO is great,
it's in time optimization. You give the compiler far more freedom to optimize, and this is useful
both for size and performance. One of the things that we found was, well, even though people had
been shipping this for a while, the problem is, while in Gen2 we default to the upstream configuration
usually we can match the distributions, too. The problem is, with use flags, you have these end
different variants, right? These usually map into pre-procise conditions. For example, you have
some strapped food, and it has some memory that depends on macro that's in a header. If I had
there isn't consistent included, you'll have two different sizes for the food strapped. So we
pass a bunch of error flags, which Andreas mentioned earlier in order to catch these. I've
spent quite a while on this. We've fixed many, many, many packages. We've bought many of them
streamed, but we've played a patch as well. So my hope is soon, we'll be able to enable LTO,
in the big host as well, and get smaller dynamics. You may notice, actually, that the diagnostic here
isn't great, because it just says, food and food, they're the same drink twice, right? But you
can't see all the actual differences. Like, I found a bunch of GCC boats, and I think I may have
got fixing something as well, so that you can actually show what the mismatch is.
One other thing, and again, a lot of these issues have been, they are problems that always
existed, but when it comes to source build, they didn't matter so much, but then we're buying
your eases they do. LTO, when you use normal dynamics libraries, there's no bit code in that's fine.
When it comes to static libraries, it's essentially a dump of IR, and we work with some of the
videos as people are stripping out all that junk, and so we can distribute them freely now as well.
It was an awful lot of code, and most of it was texts, but yeah, quite a wrap in there.