Thanks a lot for coming. Welcome to my talk about a potential path for the future.
So I have too many slides, so let's get quickly over with. I'm Dan, I work for Sousa,
I've done package maintenance, I've been doing it for a while. If you want to send me
hate mail later on, you'll find it in the slides how to contact me, but please don't.
At least not with hate mail. So what are we going to talk about? Mostly start out where we are.
What kind of is the problem that I see in our in the way how we build distributions?
And then we're going to take a quick look at a few notable examples that popped up in the last
years and that successfully employed, let's say cloudy container technologies and attracted quite the user base.
And then I am going to present a proposal what we could do and essentially you can just see that as an inspiration
for what we could do or maybe won't. We'll see. So where are we?
And essentially I'm please don't take this as an offense, but if I look around the room,
unfortunately, it kind of confirms this a little bit. The open source.
I'm getting older as well. We're all getting older and there's not that many young people trickling in.
So tide lift made the survey that the open source community is getting grey and grey on.
If you want to find this blog post, you'll have to go to archive.org because they got bought by so-and-a-type at some point and now it's down.
But this thing looks kind of okay. People are getting older, but if you look at the data, it's like from 2021 to 2024,
I wouldn't count that as so such tremendous evidence.
So we went digging a little bit and if you're looking for developer surveys, that's the elephant in the room and that's the stack overflow developer survey.
And so what you see here is I hope you can read it.
It's essentially a graph from 2016 to 2025, how the developer age distribution evolved.
And then you can also see an interesting pattern evolving by itself, which I must admit I do find it surprising,
because you see that the majority used to be between 25 and 34.
And that one shrunk in 10 years by about the same amount that's the 35 to 44 core group.
So that part probably just aged. Fortunately, the part the developers between 18 and 24, they didn't shrink by as much.
So there's younger people coming in.
But what does this tell us about us?
Literally nothing because what's the overlap between the distro builders and general developers?
So let's look at a few other surveys. For instance, Linux Foundation ran a diversity and inclusion survey in 2021.
I overlay the stack overflow survey data from 2021.
And you can see well already the Linux Foundation, which is not just distribution adjacent people already there.
We see a big overlap of older folks.
Debian ran something like that. In 2016, sadly not since then, at least nothing with actual demographic data.
That one is with data from 2016. Again, you see quite a shift towards older people, older developers.
Now, from my side of the world, open-susa that definitely matches my perception of the open-susa community, which tends to be relatively old.
This is a maintainer survey from 2021.
I think it had only something like 220 respondents, so you've got to take it with a grain of salt.
But you see a certain trend here.
Sadly, I couldn't find really any distribution that did a demographic survey at one point at an later point, where you would see some kind of shift.
But I hope you get my point.
And then let's take a look at the heap new stuff, the CNCF.
They ran a survey called the state-of-cloud native development in 2025.
If you overlay the stack overflow survey from 10 to 25, you see it matches pretty much exactly except that best.
More young people, a little bit fewer old people.
But this shows you something, and that's essentially what do developers want.
So let's look at the stack overflow developer survey.
This is the most admired, most desired.
That's the admired versus desired column.
You see, admired desired darker.
It's number one.
Then at some point you get AWS Kubernetes, NPM.
Well, the web is eating the world.
And another thing that's also pretty, pretty jarring is code and documentation.
Collaboration tools get up is by far number one.
I have questions about point number four.
I don't know why there's Gira.
I don't know.
Every time I have to use Gira, I think about debugging Pearl code because it's more fun and more enjoyable.
So, but yeah, but you see Gira.
I doubt that's going to change anything.
Yeah, so anyway, now we have to ask ourselves why aren't we up there?
Why aren't we desired?
And I don't know how to formulate this nicely, but our tooling and our platforms.
They aren't just cool anymore.
We aren't cool hot shit.
And I mean, well, I think most of us are developers.
We like the cool new stuff.
And sadly, we've been around for 20, 30 years.
Stuff is established.
And it's just not super exciting anymore.
Also, to a certain extent, our values aren't that super exciting and that much respected anymore.
If you're very much into software, into software freedom,
people at least many developers tend to be more pragmatic.
And just slap my T on it.
Don't caff it's GPL or MIT or don't care.
Something that might become a problem and that one I can at least tell you from the open suicide.
The other lamps are not trained on our tools.
They don't understand packaging formats very well.
They guess wrong stuff.
And that's going to also be a problem.
So if I ask a cloud for a spec file, it will give me one for centers, not for open suicide leap.
And then I have to do the work.
And also building a distribution is very hard.
If you just take a look at what all goals in there,
you have to have to curate sources.
You have to build packages, build repose.
You have to do a lot of QA. You have to publish it somewhere.
You have to hold a whole lot of infrastructure.
And a bad part of that is also it's kind of soft.
Because I think we all do a pretty good job of it.
So why should you get in there as a new developer?
Because it kind of works and I don't care.
And for the average death, the whole pipeline in there looks pretty much like this.
At some point there is upstream.
And then I don't have a wizard head with me, but I take it.
And then the deliverables fall out of it.
So I don't want to spend more time complaining.
Let's take a look at a few notable examples that became very popular in the recent years.
I'm sorry I have to talk about Omar Chi.
That's so if you haven't heard about it,
that's a very opinionated arch Linux been with Hyperland.
It's made by DHH.
It got quite a lot of traction,
surprising a lot from the typical Mac user that develops their Kubernetes app and runs it in AWS.
But one thing that's interesting, it all runs on GitHub.
They meet on Discord, they collaborate all over there.
And if you look on their GitHub page,
and you look at the language distributions what they use,
it's mostly shell.
So it's mostly shell scripts.
I'm not saying we should replace everything with shell scripts.
And Omar Chi is just, it's just lay at on top of Arch Linux.
So it's not replacing for this distribution.
There's also another great example that's the universal blue product.
The universal blue project.
So they have a very nice quote that they put on the web page.
That describes the whole project.
The universal blue is a tool that allows Cloud infrastructure people to use their skills to help the Linux desktop.
Which, in my opinion, is a genius idea,
because you saw all the new developers trickling into the Clouds into the Cloud ecosystem.
And there's a lot of interest going there.
And they have very few touching points to get into a distributing
and universal blue tries to make that happen.
And how did they do that?
Essentially using boot C.
So boot C for those who haven't heard,
that's a project kind of adjacent to Podman and comes from the Red Hat side.
And it's essentially a way how you can deploy a container image that's been built
on your bare metal OS.
I mean, yes, it is kind of the wrong format for the job.
I do agree with that, but the damn thing is extremely successful.
And for completeness, they, everything is on GitHub.
The CIS on GitHub actions.
They talk on Discord, Discord, but as I said,
they've been very successful of recruiting new contributors,
because every new developer knows how to write a Docker file.
Every single one of them knows that.
And that's just a huge enabler.
Yes, what falls out of it is terrible to deploy on their metal.
I think no one from the boot C team would disagree,
but you just get, you just enable so many poor people to contribute.
So take away what do developers like?
They like GitHub.
I don't think we should switch to GitHub.
I think that's a terrible idea.
They like this score.
Most distros are already there because matrix bridges are cool.
They like opinionated distributions.
Yeah, I think many, there's many opinionated spins of open source of Fedora.
I think there's also variations of BB and Ubuntu,
but I'm not that familiar with that side of the world,
but I think we have some in that regard.
But they like to use their tooling,
and that's where we are lacking.
And so I would like to raise something with you,
and that's can we embrace container-based tooling more?
Maybe not because it's the best tool for the job,
but it's because it's the tool that will help us attract the next generation.
Because otherwise it's going to be the same guys in 20 years,
and eventually we're going to start retiring,
and I don't want that to happen.
So let's take a look at the base of the distribution.
So far we only look at spins that are built on top of all our packages,
but what's really the building block, that's the package itself.
So I come from the RPM side of the world,
and we built RPM packages from spec files.
I threw in one of the packages that I happened to co-maintain,
and essentially it's countdown, so it's not so long,
but essentially a package you have some metadata that tells you bits about your package.
You can add licenses, sources,
define built dependencies, more metadata,
and then you go into the actual building.
So this is RPM everything that starts with a percentage sign as a macro,
that does things kind of either variable or a conditional,
or it could be a shell script or it could be a section.
So let's ignore the exact details for now,
but this one would essentially first you unpack the package,
and you actually build it,
and you should test it if you can, you should, but sometimes you don't.
And you install the whole thing into some destination,
and at some point you have the final package,
you define what goes into the package,
and then my most unfavorite part is the change log,
which fedora fortunately now automated to just take from git.
And honestly, this looks all pretty arcane,
but let's take a leap of faith.
Is it really much different to this?
Where you define your build route,
you copy over your sources,
you install your build dependencies,
you build the damn thing,
you install it somewhere,
and you create another stage,
where you from which you assemble the RPM,
you add some metadata into that,
and then you copy over your binaries.
I think it's kind of similar,
and essentially at the end of this,
you would have a container image with a base layer,
that is your distro,
and the layer above is your package,
and then you just have to take this layer,
and convert it into an RPM.
And because I know how you really love
just theoretical stuff,
I tried it,
I made a tool,
I gave it a terrible name called Rosie,
it's very hacky,
but it kind of works,
and it essentially does exactly that.
So it builds RPMs from these stalker files.
And you just have to shift concepts a little bit.
You convert sections become stages in a stalker file.
Your file section essentially becomes copying files
from your boot stage into the final stage.
Macros.
I'll come to them later.
You have work around,
but I think we could also rethink things here.
The preamble where you redefine all the metadata
that becomes unfortunately also a config file,
because some things you just can't put into labels.
Fuse RPMs scriptlets for those of you
don't know them,
consider yourself fortunate.
We have to put them in another file.
But I think what's also a very nice thing is
currently every distro has some kind of isolation
to build their RPMs,
and for that we can just use container tooling,
because it can already do isolation.
Okay,
bigger issues.
RPMs has automatic dependency,
dependency generation,
so if you build something,
and it needs a shared library or something else,
RPMs will figure out for yourself,
and you don't want to lose that,
because that would be a horrible, horrible regression losing it.
But it's possible,
because RPMs have some tools for that.
Reproducible builds.
About that one,
I hope setting source state epoch will be enough,
but I haven't tested it yet.
Debug info is kind of,
I think that should be possible,
but that one I didn't get to,
so unfortunately,
unfortunately wasn't sick enough long,
over Christmas,
so I didn't get to work on this long enough.
I haven't looked into change logs,
because I hope we can just get rid of them,
because I hate them,
and I don't think they provide a lot of value.
Macros and filest,
well, let's see if we get to them actually.
So how does this really work?
I try to make a pretty graph that kind of explains this.
So at initially you throw in your Docker file,
you launch the first build stage,
which I just called Build requires,
where you copy in the sources,
you install dependencies,
and that one runs with network access,
and then you would go into the actual build stage,
which is really just another stage in a Docker file,
and that one runs without network isolation,
there you build your package,
you create a new container image,
and then you run the install stage,
where you just copy files from the build into the final image,
and that one you base again on your distro build route.
Then as a last step,
Rosy really does the heavy lifting,
and that's assembling from this stage,
it assembles the RPM.
It runs, so fortunately,
RPM has a binary called RPM Deps,
which runs all these dependency generators.
It's no longer being used for that,
but it still works,
and you can parse its output,
generate the dependencies from that,
and then there's also a YAMO config file,
I know some people hate YAMO, but cloud people like YAMO,
so that is used to really assemble the full RPM.
Yeah, and that's, it kind of works,
not yet completely, but it kind of works.
A few things about macros,
since I know that I'm running relatively short on time,
let's just breeze through this a little bit,
so the RPM macro language is,
it's M4 and CPP inspired,
and to make matters worse,
it also has a lower interpreter,
and it supports shell expansion and security-wise,
on a developer machine, it's actually very, very horrible.
But it's,
there's main usage,
like you can use it just for variables,
you can use shell variables for that.
The shell has conditionals too.
It supports something like functions.
That could be replaced by shell scripts,
the sections, as I said, these become stages.
The real big problem is new fancy RPM features,
like declarative builds and spec parts.
That one,
so honestly, I don't think we will get rid of RPM or that we should,
but maybe we can,
maybe we can just use parts of this to make,
to make the developer workflow nicer.
And the second part is file lists,
and that's,
so I must confess,
I don't know how, for instance,
RH or DB and do this,
but in RPM it's conventional,
at the end of the package,
you declare what you put in there,
and if you copy something,
and if you put something into the build route,
that's not in this list,
it barfs at you.
If you do it from a Docker file,
you do explicit copies,
so it's kind of,
so you have to be explicit about it as well.
The problem is really these additional,
these additional macros in front of that,
like config and config,
no replace for that.
I think to a certain extent,
we could go by convention,
so that if you put something at sea,
it's most likely a config file.
And if that doesn't work,
then probably,
then maybe use labels or something,
but this part,
I haven't yet solved.
So, and since this,
since I'm running short on time,
and this is actually more of a
45 minute talk,
I'm going to do one to skip a few,
and go to the call to action,
and I don't want you to take the pitchforks
and tell me that I'm full of shit,
and that this is a terrible idea.
I want you to take this as an inspiration
and as an example,
how we can rethink our tooling.
What we can change,
to get in new people,
and to ensure that we don't die out,
because I would really not like that to happen.
And if you want to take a look,
there's a link to RoC,
there's a link to the slides,
and I think we might have time for a question or two,
so thanks a lot for your attention.
Yeah.
Fantastic talking very much.
And how do you think we can make this decision
effectively?
Because we,
we think so much for a tool, right?
You took us a lot of them for a lot of years
to build the current,
and it's only a slide.
It's very complex, very sophisticated,
and we're able to take this for something that is simpler,
and I'm always the right tool for you,
but it's widely available in slide,
and one of the businessmen.
So how do we make this decision?
So for the stream,
the question is essentially how do we transition
from the current tooling to new tooling.
I think with this,
it's actually it could kind of coexist.
So that's of course a maintenance burden,
but so RoC doesn't work without RPM.
So it's not really a full replacement,
and I don't think that we can make a full replacement
in one go, that's not realistic.
So this would be just a step.
So I think we have to do things step by step,
because if you take a giant leap,
and re, and throw everything under the bus,
then I think 90% of the people in this room I just can say,
I'm out.
See ya.
This is crap.