Okay, so no introductions, okay, this is distributing Rustian RPMs relative for fun, relatively
speaking and profit, and Fabio, my nickname basically everywhere is Dekla Torps, so if you
want to find me online that's what you need to search for.
Short introduction for myself, I've been for the raw packages since 2016, submitted my first
Rust package to Fedora in 2020, and became the primary maintain of the tools we use for Rust
packaging in about 2022. I've been a packaging committee member since 2018, and engineering
steering committee member since 2019. I think I'm going to look at my screen because that
one isn't flickering. Okay, so what's this talking about? Basically kind of answering the
question that's been floating around in this space for a while now is basically our traditional
package managers becoming obsolete, and I think I'm going to give a short answer to that at the end of
the talk. So for a quick background for people who don't know Rust, which shouldn't be in that
many panel, the compiled language based on LLVM, typically everything is statically linked
because there's no stable ABI, which may be coming in the future, but not yet. There's heavy
usage of features like compile time code generation, meta programming with macros or build scripts
and stuff like conditional compilation. You also have an official package manager and build
system and official package registry, which is cargo and crates.io, which means one of the benefits
of using Rust is that package management and dependency management is really easy, but that also
makes it really easy to add lots of dependencies to your projects. And on the codora side,
we have basically a hardware requirement to build everything from source. We have a hardware
requirement for compliance with the license terms, of course. There is a strong preference not to
bundle or vendor dependencies. We have a preference for running test speeds if that's possible.
And of course there are like incentives to provide good system integration, which for example for
command line applications that's like providing shell completions and manual pages and stuff like that.
The approach we use for libraries is that we create packages for Rust crates,
that only create contained a crate sources and no compiled artifacts. Again, because no stable ABI.
And even if it had a stable ABI, shipping the compiled artifacts wouldn't really be feasible
because you have a lot of combinations for all the feature flags you can enable for those crates.
And you basically get a combinatorial explosion for like a package that has 10 feature flags
you need to ship to the 10, which is a 124 different compiled artifacts, which just blows up
and not really possible. And basically the philosophy we have for packaging Rust applications
is to make installing it from the package manager provide something that's as close to installing it
from the cargo registry, basically as close to possible as to that or if it can provide something
better. So what are the things that we can do better than that? Let's take it. Oh my god, that's really
distracting. I hope it's not the cable. Okay. Okay, how does installation application installation
work when you use cargo? I only support installing applications by combining them from source.
So it requires users to have a local Rust tool chain installed, which is like cargo,
Rusty and LLVM, which is quite a lot of stuff. If your application has any external dependencies
from like CLI per race, you also need to install those. And at the end of that 10 cargo install,
also only supports installing executables into your own users path. So it only installs
executables, it only installs them for the local user. It cannot handle any other files except
those executables. So basically in the Rust ecosystem, some kind of workaround for those limitations
have been developed and have developed over time. Some of them are usually considered bad
practice like limitations of cargo build that incentivize bundling CLI per race or
statically linking those CLI per race and some creates even ship prebuild object files,
which is like, don't please don't. Because doing that means you don't have your users installed
those libraries on a system wide level. Like if you can bundle like a statically,
you can bundle SQLite sources, then it don't need this SQLite development
had as available on the host system. And like because cargo install only installs executables,
that kind of made projects adopt the self-contained executable approach. Basically bundling
all the data files and stuff you they need into the executable itself because there's no
not really a good support for loading them from disk because you can't install them to disk
with the default installation mechanism. But the other hand installing applications with RPM looks
quite different. Everything is integrated with the system package manager. The packages are
when you get them from the system like Fedora. They're built on a trusted environment. They're
cryptographically signed. When you install them, you don't need a rust tool chain. Any
development had as available on your local system and the packages can actually ship support files
for good system integration. As an example for command line application, it's usually
preferable for users to have shell completions manual pages, config files, data files,
whatever you need for the application to work correctly. Actually you also get like a smaller
footprint for the built application both at in-memory and on disk because they're like dynamically
C and C++ libraries. For example, SQLite is de-duplicated both on disk and in-memory
and it also disentangles the update cycles between like those libraries and the application.
So for example, if there's a security vulnerability in SQLite, it won't need to update
a bundle SQLite in every rust application that includes it. But you can just wait for the system
copy to be updated and you don't need to do anything for the rust application.
Okay, so you installed the application. Now you want to update it. How does that work with cargo?
Short answer is, it doesn't. It always requires user interaction. So basically running cargo
install again for all applications you have on your system. There's no mechanism to notify users of
available updates, no mechanism to put security updates and sometimes rerunning cargo install
isn't even enough. Because by default, it does nothing if the application version hasn't changed.
So if you want to pull in like a security fix from one of your applications, transitive library dependencies,
you need to force it to read download all the library dependencies and compile the application
even though the application version itself hasn't changed. And that's assuming that the security fix
is published in the library version that's compatible with the dependency specifications in that dependency
tree. So if that's a separate breaking version, then even passing the force flag to cargo install
won't help you. On the other hand, package updates in RPM. Again, integrated with system package
manager support notifying users of available updates and security updates. And in some cases,
there's even automatic installation of critical updates. For example, on servers, which just
systems like the NF automatic or on image based systems, which also have started to apply
system updates automatically. And that also decouples application and library depends
dependency update cycles mostly. Bonus points for using installing rust applications from
RPM packages, we actually provide much better test coverage for those applications and for
libraries. And the package builds run that test treats in an environment that actually matches the
environment that the users are going to install those applications in. So test treats is run on Linux.
It's run on the same fit or a version that the package is going to be installed on. It's going to run
the test against the same library versions and against on the same CPU architecture that you're
going to install the package on. And by doing that, we can also test coverage for environments
that are usually not available like in CI environments like PowerPC or system C. And in
overall, we also have a kind of package CI, which continuously rebuild packages against new versions
of their dependencies. And that way, we can actually catch regressions quite quickly, usually.
Okay, so you might ask yourself how does this work with packages that use mental dependencies?
Because the default approach is to not do that, but how does it work if you do?
I mean, the pro basically for rendering dependencies is that you can quickly get to a building
and working package. And because you can mostly avoid dealing with dependencies. And on the
negative side, it's often quite difficult to actually get from a working package to a package that
would be acceptable for inclusion in the package repositories. Because you cannot actually avoid
dealing with those dependencies. You can just put them in a box and say, I don't want to deal with this.
It would be nice if you could do that, but yeah. So some of the problems that that happen
in this case is that, yeah, you can't take it as a short cut for avoiding your responsibilities.
Like, you need to check or all the project license is declared correctly and completely,
you need to check through all the projects, contain the mandatory license text for licenses
that require license text to be included. Are all the licenses acceptable? Not all licenses are
acceptable for inclusion in the fedora. Are all projects legally acceptable? Like, is it covered by
patents? Is it some kind of media codec that can be added to the fedora repositories? That's all
things that you still need to check. And, like, due to limitations of how cargo windows dependencies,
it's also not actually possible to run tests for your library dependencies. So you can only run
the tests on of your actual top level application, but that doesn't cover tests in library dependencies.
So, how do we do things the hard way? As far as I know, mostly fedora and Debian, and I think
Maria also does it this way to some degree, which is basically create one RPM package per Rust
crate, and they, as mentioned previously, only contain the source code and some metadata.
But, by doing that, you actually share responsibilities between application maintenance.
Because by now, there's quite a large overlap between the dependency trees of most applications.
I mean, the Rust ecosystem has mostly stabilized on some popular libraries and most projects use those.
So, for that stuff, you can just rely on sharing those responsibilities with other people.
You avoid duplicated or it effort both for legal issues and technical issues, and you can actually
quite easily run the test suite for those libraries, and often find bugs, especially related to
less popular CPU architectures, like big-endian stuff, which is always funny.
We also tend to find a little VM bugs quite frequently. So, yeah, those are always fun.
We also don't get into dependency hell by doing this, because it's actually quite easy to
provide parallel installable library versions, cargo namespaces, dependencies both by name and version.
So, there's no conflicts involved.
One of the benefits of doing it this way is that actually looking at the libraries, fixing issues we find,
submitting those changes to upstream projects.
Over time, we hope that this involves improved the ecosystem for all users of those crates,
not only for other upstream packages, so eventually those fixes will roll out to everyone, basically.
Okay, how do we make the doing the quote unquote right stuff as easy as possible?
Subtemist little work has been put into improving the Rust packaging tooling that we have in
Fedora, like Rust to RPM, and all the tools around that.
By now, I think we've mostly eliminated all the busy work from the packaging workflows,
and I don't think there's any language ecosystem that has tooling that is polished in Fedora,
which is kind of the basting the myth that Rust packaging is hard. I hope, but if you've not looked
at Rust packaging recently, then I recommend you to take a look again because things are really nice now.
There are more improvements coming in the future. Some features that we have already developed
can't really be enabled by default yet because some of the targets we support ship RPM versions that are
too old, like RL 9 ships RPM 4.14 and RL 10 ships RPM 4.19, which are a bit old at this point,
but we're still supporting those by building for Apple, so extra packages for Enterprise Linux.
Two examples of recent improvements that we've made are like dynamically generated sub packages
that dramatically reduces the boilerplate that you have in RPM spec files for Rust packages.
It mostly makes it unnecessary to rerun Rust to RPM for crate updates or metadata changes,
but it requires RPM 4.19, so that's not available on RL 10.
Then there's support for fully declarative build system in for Rust crates for common cases
that feature was added to RPM in RPM 4.20, so that's only available on Fedora,
not even in Apple 10.
Can we answer this question now? I mean, at least in my opinion,
based on the missing features that are in new kinds of package managers, I don't think they can replace
traditional package managers quite yet, especially because of missing support for like
pushing security updates, which is kind of a critical feature in my mind.
Okay, so I think we actually have quite a bit of time for questions.
Yeah?
Yeah, I was wondering, how many of us are built in this like five of us where you don't even have crates,
really, the same sense, could I speak a bit? There are packages that are like a bit special,
the Firefox is one of them. I think the publish sources, the sources, they publish for Firefox,
actually contain all the things that you need to build Firefox, so they use a different process.
Okay, so what else did you just skip it up for in here?
Those packages are maintained in a different way, so they don't use the same tools.
Yeah.
So you mentioned that the Matean's workload of the dependencies shared by these users,
I mean, but you think that is actually happened, or the Matean has these actually,
taken more by the first person that meets the MSE, and they will need it anymore,
and then you need to find another one.
So the question was that that's sharing the responsibility for package maintenance,
actually work, or is it just sort of onto one person? Usually, that works, yeah.
So especially if you have like, hey, my application requires a library update for this library,
and for that, we need to like, push changes to five other packages that depend on this,
and usually that's the way things are shared between those applications maintainers by.
Needs coming from different sites, if that makes sense.
Yeah.
Can you provide a more details on the state of the center's line and center's network
as well as what is provided by the two last packages and whatnot?
The question was what the tooling support is missing for Apple 10 and Apple 9?
The only features that are not supported on those targets are the dynamically generated
top packages and the declarative build system. The packages themselves are identical between
federal and Apple.
The question was how do we determine if there are security issues, or how do we determine what
packages are affected? Yeah. There's a database of security issues in Rust Crates.
There's a website called RustSec and that's basically the standard way to track vulnerabilities across
the Rust ecosystem, and we are keeping track of those. There's also a CVE box filed against
Rust libraries sometimes, though, happily not that frequently.
Yeah. We have the mapping from the crate name to package name is easy because that's just
the one-to-one mapping. And even for packages that used to the end of dependencies, they have
mandatory metadata so you can map. You can query the package repositories for which package
it bundles this crate. And so you can compile a list of affected packages and then apply
the fixes, rebuild them with the fixed versions. How do you handle dependencies that are very
happy to play with your semantic question? Okay, the question is how we handle dependencies that
publish frequent major versions? Yeah. Yeah. Usually don't like them.
Yeah, it causes more work than if they don't publish breaking versions, of course.
I mean, as I said, it's possible to just ship multiple versions of the same crate. I mean,
I think the one where we ship the most different versions is the nix crate. If you know that one,
which has like 0.23, 0.24, or 0.25, up to 0.31 by now. We're trying to reduce the number of
different versions that we ship just because all versions kind of tend to go stale. And a
cumulative box, especially like with, I mean, for example, all versions of the lix crate are
now kind of incompatible with glipc 2.43. So I've been trying to get rid of some of those old versions.
But yeah, we can. The two approaches you can take is just try to patch everything to the new
version and submit those changes to the upstream projects. So those eventually add up end up in
new releases. Or you can package both versions in parallel, at least for a limited amount of time.
Yeah. Have you ever encountered support for your ideas for a resistance from the community?
Yeah. The question was, if there's been resistance to our efforts within the rust community,
or support, both. I think. I mean, some of the, some of the frequent responses we get is
why are you doing it this complicated way? And I mean, I think I provided some amount of
reasoning why we do it the way we do it, just because it. Not doing it this way is kind of irresponsible.
Um, on the other, Meteorical.
I think it's. It's like transmit the road to some?
Anyway, what about Aren't you thinking the number of cricket because of the
Sometimes, there are features that just add staff them, but they don't like it.
I get enough in ways if you want to feature them, but sometimes you can't really be sure
that you have just adding stuff, we'll use that to reduce the packages and add more features
than in what was actually used upstream, or just answer the last question, because
this is up time is up now.
I think the question was, how do we reduce the number of packages that we ship,
because some combinations of feature flags are not compatible with each other?
Yes, I mean feature flags are supposed to be additive, but that's not always the case.
Like, for example, you have a package that supports two different backends,
and you choose one of them, so enabling both is really kind of defeats the point.
Mapping both features from cargo metadata to RPM package metadata is still necessary,
because different applications might use different feature flags.
But again, enabling both of them doesn't really make sense, but they're both need to still be present.
I think that the time is over.
You can ask me afterwards.