Thanks, Fabian.
All right, for we get started, real quick show hands, who here is already used Epleton.
All right.
Well, you know, what about Epleton 9, 8, 7, 6, yeah, 5, 4, okay.
A lot of y'all been using it for a long time.
That's great.
One more, two more show hands.
Who is the late migrating to a new major OS version,
waiting for Epleton packages to get created?
Oh, there'll be more people.
What about who is the late upgrading to a new minor OS version,
waiting for some Epleton packages to get fixed
because they got broken in the minor version of date?
All right.
Troy does this problem.
So that's the stage for what we're going to talk about today.
The road to Epleton, my name is Carl George,
and I end the Epleton team lead at Red Hat.
So first off, we'll level set a bit and talk about what Epleton is.
In case anyone's new or not a familiar,
based on the show hands a lot of y'all do already know,
but Epleton stands for extra packages for enterprise Linux.
It's an initiative within the Thor project to create additional packages for Centos
and Red Hat enterprise Linux with a goal of enhancing the base distribution
without disturbing or replacing any of the stock packages.
So Epleton packages are derived from Fedora.
This diagram here is a rough approximation of how it works,
but it's not drawn to scale.
There's way more than 90 packages in Fedora.
A subset of those will get branch off and they go into Centos and then Rell.
Everything else that doesn't, isn't part of that subset,
is eligible to go into the Epleton repository over on the side here.
If a package later gets added to Centos and Rell,
we'll remove it from Epleton to make sure we continue to not conflict with any of those packages.
Let's zoom in a bit on that Fedora to Centos to Rell process.
I talked about that subset.
So this is the legacy relationship how things used to work.
Fedora Rell hides the role in release version of Fedora.
Fedora versions will branch off from that about every six months.
And then in the past, Red Hat would take one of those versions.
And behind the corporate firewall do private development to create the next Rell major version.
And then they would just one day say,
Tadah, here's Rell 7.0, and it's public now, and it's open source.
At that time, the Centos project would take that source code and rebuild their distribution.
And it tried to match Rell as close as possible.
That makes a useful distribution, but there's a lot of problems with it.
The goal at the time, the state of goal was bug for bug compatibility,
and that if a bug was reported, if that bug was also in Rell,
Centos wouldn't fix it.
They would just leave it.
And that was honestly a frustrating experience both for users and maintainers.
And that also meant that contributing to Rell,
there wasn't a way to do it directly because it was developed in private.
You had to contribute to Fedora, and then maybe in three to five years,
it would go in the next version of Rell.
The next major version.
So the contribution path was terrible.
Fixing bug reporting process was not good.
A lot of problems.
We've moved on and fixed it.
Good news.
This is the modern relationship we have with the three distributions.
Still, I've fedora high, moving along, and the Fedora versions branch off.
But that branching to create the path to get to the next real major version,
happens in public in the Centos project.
That right there's the bootstrap phase where everything's getting settled
and decisions are made on libraries.
And then, from that point, once we, at some point, we'll get, we'll say,
this is settled down, and this is now more or less stable,
and we will have the Centos announcement.
We just had that for Tim back in December just a month or two ago.
And at that point, it serves as the major version branch of Rell.
The minor version still branch off from that, but they are separate,
and then the major version is just there in the Centos project.
This fixes those problems I talked about.
Now, when Centos gets a bug, we can just go ahead and fix it in CentOS,
and then it becomes part of the next real minor version through to six months later.
And fixing those bugs also gives us the path to contribution.
You don't just have to contribute to Fedora and wait for years.
Now, you can contribute to CentOS and see results in a matter of months instead of years.
That's a huge improvement for people that want to get involved with the project.
In this way, with it shaping the next minor, becoming the next minor versions,
CentOS defines Riddhead and Fresnelenix.
It still has to follow those compatibility rules, though.
If CentOS were to change something or accept a contribution that changed something too drastically,
that could cause problems and break compatibility between, say, Rell 10.1 and 10.2.
And the business folks on the Riddhead side of the house, they wouldn't like that.
There's contractual obligations about what can change between the minor versions.
And those are defined.
If you ever look it up, it's called the application compatibility guide that has the specific things listed about how
compatibility works in Rell and what major version compatible means in minor version compatible.
So, because it has to follow those rules, it's still very similar to Rell,
even though it doesn't try to match it exactly anymore.
Historically, we've had two major problems with Apple.
I kind of alluded to it a bit in the show of hands at the beginning.
And the first one is the minor version problem we're going to talk about.
Rell is very stable that compatibility guide I talked about, but it isn't frozen.
Sometimes new minor versions will have a few packages that change a library interface.
And when that happens, it's deemed compatible enough to follow the rules for Rell,
but it could still require that third-party packages like Apple need to be rebuilt to be compatible with it.
In the legacy model, their top Apple only built against Rell,
and then it was used on SintoS and just treated the same part of that kind of bug for bug compatibility.
They wanted it to be as close as possible, so those third-party packages work the same.
And that mostly worked, except when a Rell minor version released came out,
SintoS project had about a month gap where they were getting things together, rebuilding all the packages,
until corresponding SintoS minor version would be released.
It's not really shown in the diagram there, but SintoS 7 basically was 7.0 here,
and then just became 7.1 here after a little bit of a gap and so on.
During that gap, if an Apple package had become uninstallable,
because of a change in a library in Rell, the maintainer had a hard choice.
If they rebuilt it immediately to fix the problem for Rell users,
they would then break it for SintoS users that were still on the previous minor version.
If they ignored it so that they didn't break SintoS users,
the Rell users were just left broken for up to a month or so.
So there wasn't a good answer one way or the other,
depending on the individual maintainer's priority,
which users wanted to be more friendly to.
But because it was only about a month time frame,
the problem was mostly ignored.
There wasn't really a clean way to solve it with the idea that we had to build Apple against Rell.
In the modern relationship, we kind of flipped that problem on its head and also extended the duration.
Now those library changes that are going into say Rell 10.1,
we'll see them right around here in SintoS months ahead of time,
instead of just a month gap, we've got a 3 to 6 month gap, the other direction.
So that longer gap made that particular problem with minor versions harder to ignore.
The first attempt at solving this problem was known as Apple Next.
We launched it in June of 2021,
and it was a way to rebuild against SintoS only when you needed to.
When you had one of those library differences.
It wasn't a complete duplication, it wasn't a whole separate Apple.
It was just a small subset of those packages with a slightly higher version or higher release
so that they would serve as an update for SintoS users.
That it was you have Rell users still using Apple,
just like they always had that didn't have to know about any changes.
SintoS users though would need to have Apple and Apple Next enabled on their systems
so they would get the right corresponding set of compatible packages.
This solved the basic problem.
It led maintainers have a path to solve to fix these packages,
and it allowed them to target both Rell and SintoS at different times when needed.
We implemented it without disturbing any of the existing Apple workflows.
If maintainers didn't know about it, they could just keep doing what they were doing,
and be none the wiser.
There were a lot of problems with this that we found that we came to learn.
It was often mistaken for Standalone repo.
I mentioned it wasn't a complete duplication,
but a lot of maintainers misunderstood it to be just that.
Even though we had a document and we tried to tell people,
but it wasn't intuitive for them.
There was a decision process for when you needed to use Apple Next.
You had to really research and pay close attention to the difference between the libraries and between SintoS and Rell.
And there was no inheritance.
If the maintainer fixed the package in Apple Next,
they had to fix it again six months later in Apple,
and then they couldn't take the bills they had done in Apple Next and reused them.
And also, being optional meant that SintoS users had a worse experience.
They may, I saw many times where somebody would report a bug and said,
this package doesn't work on SintoS anymore.
I know that Apple Next is for this and can fix it,
and the maintainer just ignored it and left it because we made it optional.
The second big problem that we've had historically with Apple is major versions.
For a variety of technical reasons,
if I can talk about more in the Q&A if people want to or afterwards,
is that each Apple version starts fresh with a fresh set of packages.
There's positives and negatives to that, but that is how it works.
And so when a new major version of Apple starts,
there's nothing in there at first.
maintainers need to go in and opt in and add their packages to it.
So when you first start looking at it,
Apple 9 might not have all the same packages that were already in Apple 8.
We don't automatically bring them over.
And the biggest part of that reason for that is the duration that that would be.
All the maintainers are volunteers,
and we don't want any of them to feel like we're forcing them into,
hey, you were maintaining Apple 8 for 10 years.
Here's 10 more years on top of that with Apple 9.
We automatically opted you into it.
We leave it on the maintainers to choose that.
But because of this, it calls some lag with packages being added,
and that also the following effect was that people that were upgrading their OS version
would wait.
They said, you know, I'm not going to switch from, say, you know,
send to a 6 to send to a 7 until all of the Apple 6 packages are in Apple 7 that I need.
And it wasn't just on the send to a side.
We also had that inside Red Hat.
We had that feedback from customers, where it's harder to support customers
on the oldest version of your product.
You want to get them on the more modern versions that have more fixes.
So when they push back and say, yeah, I'm not upgrading from, say,
real 7 to real 8 until all of these packages are available on Apple.
I know their community maintain packages.
They're not, we can't file support cases for them,
but I need them to exist.
It was actively blocking our customers from upgrading.
So we saw that as a problem as well.
This feedback from customers.
We didn't always know in this community side that it was a problem.
But eventually the customer feedback led to the creation of the Apple team in
SideRehat, which I now lead, started it.
I was the first member of the team.
The first thing we wanted to look at on our agenda was Apple 9.
And we wanted to take a stab at trying to improve on that major version problem.
And just, obviously, what we came up with originally was that we would start
Apple next by itself for 9, build it against Sinto S9.
And then quickly after the rail 9 launch, we would rebuild all the Apple 9 next packages
and to populate the Apple 9 repo.
Rebuild the against rail 9.
We thought that would make sense when we designed the plan,
but as we started trying to implement it, we realized this is a lot of extra work.
It doesn't make sense.
And so what we switched to was, we'd start Apple 9 just normal.
But we would build it against Sinto S9 at first.
We were telling people on the Sinto S9 that, you know, this is a preview of the next rail
minor version.
So this is our chance to prove it.
You build it against Sinto S9 today.
And in six months it's going to work on rail 9.0 because we said it's a preview.
So it should work.
And that's what we did.
We started building and gave maintainer six extra months before the rail 9 launch
to get their packages ready.
And then at the rail 9 launch, once it was available, we switched the repo over two
build against rail 9 proper.
And the good news is that plan worked great.
We did it as far as I know.
We didn't get a single bug report related to compatibility problems
between Sinto S and rail.
What we said came to fruition.
It worked as a preview and worked as a base to get things ready for the rail 9 launch.
And I mentioned the amount of time.
It was about five and a half, six months.
So package maintainers had a lot of time to get their packages added.
And the good news is that let us launch.
When rail 9 came out, we had 5700 Apple packages available and ready for users
in customers used just on day one.
And that was a real good news.
Lots of lots of applause.
Everyone loved that inside the business at least.
Made it look like I was doing my job on the Apple team at least.
That's all context setting up where we're at.
The main point of the talk, Apple 10.
The big benefit of having an actual dedicated team to work on Apple is that not similar to Apple 9.
We thought about it ahead of time about how we could start it early.
And once we had that working, we're like, OK, well, let's think even further ahead,
thanks strategically in the next step as Apple 10.
What could we do to make that better and different?
So it wasn't much longer after the Apple 9 launch.
We started planning Apple 10.
And some of the things we were realizing as we were planning this was that
Apple next, I mentioned some of the downsides earlier.
It was really a subpar solution.
It was bolted on to the existing Apple workflows.
It wasn't really a holistic solution.
And the more meta point around it was that with Apple and Apple next,
we were really just giving maintainers around about way to target two separate
minor versions at the same time.
And what we want to do is to really solve the problem,
just lean into that approach and fully embrace minor versions in Apple.
Historically, Apple had always been major version only and just kind of kept bouncing
to the next minor versions of real as it went along.
So we wanted to integrate with it better in a better way.
So as we were looking at this, the more technical details of it,
we wanted the branching process to be more intuitive for maintainers.
I don't even have a diagram for how Apple next works because I don't even know how
I would represent that.
But we looked at how Fador relates to itself.
We're all hiding the root release versions, how branch is off.
And then also the new centOS model where real minor versions branch from the major version.
So we tried to emulate that and make something that feels intuitive to maintainers.
So we'll have an Apple 10 branch and then the Apple 10.0 branch comes off from that.
This is right about the point where we're going to do that.
And the Apple 10.0 branch comes off from that.
This is right about the point where we're at right here right now.
We have Apple 10.
But it's technically Apple 10.0, but we don't have the Apple 10.0 branch yet.
We'll create that shortly and then the Apple 10, the main branch,
will become the preview of Apple 10.1.
Very similar to house into us in real relates to each other now.
In a similar way, I'll point out that Fador is in the same boat where if Fador 41 is the current release and raw hide,
basically considers itself Fador 42 today.
So it's like a preview of the next major version in the Fador space where it's a preview of the next minor version in the centOS real space.
Really big point that I want to emphasize here is that the Apple next problem about the double rebuilds,
where you have to rebuild for a library change in Apple next first and then Apple that goes away in this model.
You start to rebuild for some libraries as they change, but that just happens once in the main branch.
And then those changes will be prepared and ready for the next minor version.
You don't have to do it twice.
So we removed a lot of duplicate work there, or we will as this goes into the minor versions.
We're just now starting at this all brand new territory for us. Very exciting.
Aside from those branching mechanics, a higher level concept emerged during this planning.
Apple is a community repo.
So it makes sense for us to prioritize centOS, which stands for the community enterprise operating system.
An Apple maintainer can somewhat to a degree, ignore rail, and just focus in community spaces.
They can do Fador packages. They can do Apple packages.
They can work with centOS, and they can just ignore that mostly ignore that rail exists as a product.
They may get a bug report for something on rail that doesn't apply to centOS at some point.
And we don't want them to ignore that entirely.
But they can change their priorities and not necessarily have to worry about Red Hat products as much.
We are going to have it set up where maintainers can still build.
They can request older minor version branches.
We'll have the automatic branch in similar to what Fador does.
Any package with an Apple 10 branch will get a 10.0 branch automatically.
And if you create the package after that point, you can go back and request the old ones,
which will build against rail, but that won't be the default experience in workflow.
We have a very similar thing on the Fador side.
If you had a brand new package to Fador today, it won't go into Fador 41.
You have to go backwards and ask for, hey, I also want a 41 branch and build it there too.
If you just do it now, it'll go into 42 as it gets created.
And coincidentally, that is also a very similar workflow for rail maintainers.
I should really have this slide still a long time out this part.
For rail maintainers, they're default experiences.
They build it in centOS, and then it goes in the next rail minor version.
If there's customer pressure to put something into a currently released minor version,
maintainers have to go backwards and do a little extra work to put it in that one.
All of these changes, though, aren't just beneficial for centOS, they also help rail quite a bit.
Apple 9 used centOS 9 to start building packages for rail 9.0 early.
And Apple 10 is kind of an extension of that philosophy, doing that for every minor version, not just the .0.
And it avoids that double rebuild problem we talked about.
So in the past, rail or currently, a new rail minor version comes out.
And third party packages might not install on day one.
I mentioned delaying upgrading minor versions until packages are ready.
We can help get rid of that problem.
Maybe not entirely, but for the most part, have those packages ready to go on day one.
So nobody can say, well, I'm not upgrading yet to get all these security fixes because these packages broke.
We want those to work right away.
If we can.
So enough planning.
It's time for the rubber to meet the road.
Last August, we held an Epipsin Hackfest at Flotk.
This kicked off a soft launch period.
Earlier than we ever had before for an Apple release,
where maintainers could start getting to work.
I mentioned with Apple 9 that giving them more time proof to be really popular,
and helped us have a lot of packages ready quickly.
And so the logical thing was we had this great plan all these minor versions,
but let's let's just start when we, you know, let's pick an earlier time to start.
Starting too early in that branching process,
there may be some libraries still in flux that are changing.
One thing I noticed was that we had Pearl got rebased from 538 to 540 in that bootstrap phase.
Luckily, we didn't have that many Pearl packages in Epipsin yet.
I think we only had to rebuild two of them.
But that's the kind of thing that makes it where like,
yeah, this isn't why we don't, this is why we don't start like a year earlier.
We need a time to just write.
But after a few months of this, Epipsin officially launched back in December.
Thanks to the hard work of our community of Epipackagers,
we were able to launch Epipsin with over 10,000 packages available.
And remember, we had Epiply 9 up to 5,700 by the time of the Real and Mind launch,
and we're not even at the Real 10 launch yet.
But the way you had it where we were in the past.
And since I made these slides, it's already gone over 11,000 packages.
So I'm really curious to do the update version of this talking to the future.
When we have that package count of compared to 5,700 at Real Mind launch,
at Real 10 launch, we've got X number, whatever it is.
So that's our story so far.
It's far from complete though.
There's all expected timelines, no guarantees,
but next month we're going to automatically create those Epiply 10.0 branches I've talked about.
For every package with an Epiply 10 branch.
We'll temporarily build that branch against the snapshot of CentOS 10.
To make sure we don't get any of the changes that are coming in last minute for,
or the changes that start happening for 10.1.
And then whenever Real 10.0 actually exist, we'll switch to building into that.
And then the Epiply 10 branch will move on to be start populating Epiply 10.1.
And that's real similar to what we did for Epiply 9 with that snapshot mechanics.
A little bit of inside baseball down on the weed stuff.
We'll just keep repeating that for every minor version.
We'll rinse and repeat that.
And that's what we have coming up, and that's what we've been.
And that was the road to Epiply 10.
And maybe I have a time for a question or two.