Okay, good morning everyone.
Hope you're all feeling clear headed.
This is my first talk publicly, so be kind.
Thank you.
APPLAUSE
So we're here for a flat car and Gen 2 sitting in a tree.
It's about a collaboration of distributions.
And my button doesn't work, typical.
So yeah, I'm James McRow, better known around here as Chewy.
I'm a flat car maintainer by day, working at Microsoft,
just under a year now, and I have to say being paid to do this is awesome.
So I've not been in the job that long,
but I wasn't starving from scratch because I'm also a Gen 2 developer by night.
Officially since 2015, but I've been contributing since 2004,
and you might think, well, why did I wait so long?
And I know I've just started to talk, but I just wanted to make a really important point here,
that I was really worried that I wouldn't be able to contribute enough,
and I'd get kicked out.
And I hear this all the time, even just this weekend,
and I want to tell you, even if you're just maintaining one package,
and committing once a month, that's great.
That's fantastic.
I'd rather have hundreds of people doing that than like 10 devs
breaking their backs because, you know, burnouts real.
So yeah, I've come down from Edinburgh in Scotland.
If you think I don't sound Scottish, you'd be right,
but I have lived there most of my life.
And I'm known to do silly things like run Gen 2 on the Pomodoro Amiga from 1993.
You might have seen that yesterday, I hope to have it running again later today,
if you want to come by.
So, I didn't want to assume that you knew both distros,
or even one distro, so let's do a quick intro.
On the left, we got Gen 2, created around 2000.
There's some developers who do this as their day job,
as a community distro, and legal aspects are handled by the Gen 2 foundation.
It's a source-based distro.
There aren't many of those around still, but even on that.
And it's not derived from any other distros, it's original.
It's general purpose and extremely flexible,
so you can use it on massive servers or tiny little arm boards
that I have on my arm board at home in the corner of the living room.
And it's sort of famous flash infamous for its compile flags,
being able to optimize the hardware,
but I think the real power lies in its use flags,
which allow you to customize it saying,
I want this feature or that feature or not this library or whatever,
and yeah, it's very powerful.
And sports many architectures.
It was the last distro to sport Intel Lightning.
So on the right, we have flat car container Linux,
which was forked from the original Core OS container Linux in 2018.
It's also community maintained.
Our legal aspects are handled by the cloud native computing foundation.
Now, me being a Microsoft engineer, you might think, hey, isn't flat car,
maybe a Microsoft product.
Not at all, it's a community project,
maintained out in the open, non-git hub.
You can come and talk to us on matrix where we discuss everything out in the open.
And you're very welcome, so please do.
And we do have other contributors who do this as day-to-day job outside Microsoft as well.
It's in the mutable distro.
It's a hot topic these days.
What does that actually mean?
There's no package manager.
This size read only and upgrades are atomic.
It makes it quite good security wise.
It's derived from Gen2, so it's not obviously not got the package manager on it,
but it's built from Gen2.
But its history comes from Google's Chromium OS and the old container Linux,
but we've cut from that now and it's just directly derived from Gen2.
And yet, minimal just has what you need for running container workloads.
So you can have it with Docker or Kubernetes, pardon man, container D, you get to choose.
But it still feels like a familiar Linux distro.
We haven't cut it down so much that there's still SSH in the still batch and what have you.
And it supports MD64, ARM64, and unofficially risked five.
You might have seen the talk about that yesterday from Adrian Fladoo.
On VMs and bare metal, so we still consider those just as important.
And runs on, I think, 20, at least 20 cloud environments now, not just as you.
So moving on, what do we mean by sitting in a tree?
Well, maybe this kind of tree.
This is the family tree of Gen2 forks and derivatives.
We're in there somewhere if you squint really closely.
Or maybe this kind of tree.
So, this works.
I can deal with that.
Yes, so this is a black car's main code repository.
And might look familiar if you're a Gen2 user.
It includes a copy of the subset of Gen3 packages.
Not all of them because it was faster to just have the subset.
And having it as a git sub module would be annoying.
This way we can keep it aligned with flat car releases.
It just makes a whole thing easier.
Now, obviously, with this many shared packages, there's a good level of shared interest.
But we also use other Gen2 tools directly like Catalyst,
which we used to build our SDK.
Not flat car itself, but the SDK.
We use cross-step for cross-compiling and package check for QA.
So how's this relationship evolved over times?
So since Chromium OS, I'm the first maintainer to wear both hats.
I'm hoping I'm not going to be the last time I need to sweet talk.
I want to do people in my team, maybe.
But considering that container Linux started in 2013,
it's a shame it took so long to bridge that gap.
And stemming from that in the seven years of container Linux is existence.
There are only seven commits to Gen2's main package repo.
That's not great.
For about the same time period in flat car, we had 130 commits.
At least, it's hard to count.
I commit as my Gen2 address.
Sometimes it's hard to tell whether a commit is work or just in my free time.
But there's a overlap there.
If you're thinking, oh, isn't this all just you?
No. I can't take all the credit, even without me.
There was 47 commits from the rest of the team.
So they were already leaning quite heavily towards upstream.
We fork a number of packages in flat car simply because the file system lay out slightly different
with USR being read only.
But we've recently found new ways to deal with that without having to fork the whole package.
So that number is now reducing and bringing us closer to upstream.
And in general, we just see a lining closer to Gen2 as a good thing.
Whereas container Linux arguably drifted away from Gen2 over time.
We're going in the opposite direction.
Then we just believe that more commonality is good for both of us.
So, there's Gen2 help flat car.
Not by asking what flat car needs.
We didn't have one Gen2 to ask that.
It didn't come to anything yet, but we'll see.
But that's okay because that isn't what flat car needs.
So, flat car chose Gen2 because it's already a great fit.
It's very flexible, as I said.
And it's fantastic for cross compiling, which is very important for us as you'll see in a minute.
And you might think, well, we didn't choose Gen2.
It was contained in Linux and Kremlin OS before us.
But I think if we were making this decision today, I think we've still choose Gen2.
And at no point, if we said, oh, maybe we should try something else.
No, we're very much still on board with Gen2.
Gen2 helps by continuing to be well maintained.
Keeps the packages updated with a strong adherence to QA.
And I think the QA is really important when Gen2 is that flexible because you can't just assume
it works on my machine.
There's so many literally so many variables in play.
They apply security fixes very promptly, and that's a key focus for flat car.
And they handle our contributions, confidently.
You know, don't leave them dangling for ages.
And there's no issues with these being paid contributions.
No suggestions of conflict of interest.
They just recognize that we want to make Gen2 better.
So, how does flat car help Gen2?
Well, cross-compiling is one way.
So we cross-compile everything, even for AMD64, because we believe it gives a cleaner build.
By no accident, we're pulling in build components.
And it just so happens that I've been Gen2's cross-compile guys since about 2006.
And being paid to spend more time on those issues is great.
So, flat cars help by fixing cross-compile issues in many packages from, say, SSSD to open ice,
and e-classes from go to rust.
And the great thing about e-classes is once you fix the issues there,
it's pretty much fixes them for all those packages.
And we're enhancing the cross-dev tool chain utility for cross-compiling.
And, yeah, you'll be able to pick a different portage profile rather than the default embedded one,
which isn't very helpful.
And we're moving the need for extra flags when building against alternative statistics,
which means that you can build against a different directory,
other than the one that the tool chain's hosted in.
So, say you mount your root file system from your Raspberry Pi or something over NFS,
and you can just directly cross-compile against that,
and much quicker than building on the Pi, for instance.
And we're making it possible to cross-compile pre-fix systems.
If you're not familiar with Gen2 pre-fix, this allows you to have a sort of
Gen2 system in an arbitrary directory, not slash USR.
And flat car wants to combine this with system D6 to bolt on additional packages that you might need on to flat car.
So, we've also helped with the package manager.
We're Gen2's flexibility.
You can often end up in possible situations,
and portage tries to figure them out for you,
but sometimes use interventions required, for example,
with circular dependencies.
You need to disable the feature, but first time you build it,
and then enable it later, and that's a pain for us,
because it affects our bootstrapping process,
and we have to shepherd it through with scripting,
and we're thinking, well, it tells you what to do.
Why doesn't it just do that for you?
Turns out that's quite a hard problem.
It's a 12,000-line Python file involved.
I've said a bit for a few days,
and it worked in some simple cases,
but it needs a little bit more work.
What's this space?
We've added some new packages, so some of us use Gen2
to do our work, like me.
It didn't have to, but it helps with the upstreaming.
Many more Microsoft engineers use Linux,
believe it or not.
Maybe Gen2, but I have heard of one of them.
But even if they don't, I just wanted to set an example
to the rest of Microsoft to say, look,
if you're installing stuff on your distro
or packaging stuff in a private repo,
don't do that.
You know, make it available to everyone.
And that car's SDK is basically a Gen2 image as well,
so any tools we need there might also be useful for Gen2.
So we've therefore added a few packages,
in tune, which we use to access our corporate resources.
That's not just an internal Microsoft thing.
Your company might use in tune as well.
Similarly, as your VPN client, not just internal.
If you use Azure, you may well need that.
Azure VM Utils, as the name applies,
is used for running VMs in Azure,
helps with identifying the disks.
Gen2 was actually the first distro to have that package.
Git credential manager is an improvement
over using traditional SSH keys,
because it has like multi-factor authentication.
Doesn't just support Git Hub.
You can use it with BitBucket and GitLab as well.
And the Azure SDK for C++,
which we use for our signing tool that we have for secure boot,
but could be useful for all sorts of things.
Yeah, and we're keeping these up to date.
We didn't just throw them over the wall.
So yeah, and we're also helping with VM firmware.
So you may know of EDK2,
some open source implementation of the UFI standard.
It might not realize that you use this,
but it's commonly used for booting VMs of QMU.
And we do this all the time.
So if we need to test something out locally,
or in our CI,
we run extensive tests that spins up QMU all the time.
And it was quite unloved in Gen2 to be honest.
It was quite a few years behind in terms of the package version,
and there were loads of open bugs.
So we bumped the package,
not just once, but twice, and we'll keep that going.
It makes a pure boot work out of the box.
It used to involve a very convoluted bootstraping process
that didn't even work anymore, I think.
And we added support for other architectures,
so before it only supported AMD64,
now we've added ARM64 and Risk5,
and I've made the in people quite extensible in that regard.
So we're hoping to add more.
I think the next one will be the lung architecture.
And we noticed that other distros are providing
the images in different sizes for when those update these days.
You need four megabyte images to the traditional two.
And they were providing them in QK2 format,
as well, which is more space efficient.
So we did that too.
And note that upstream's build system doesn't do any of that for you.
So this all had to be done in the e-build.
It was quite a lot of work.
But in doing that, we resolved most of the open bugs.
So how might Blackfire help Gen2 in the future?
So as I said, we don't just post stuff over the wall.
We recognize we've got a responsibility to maintain these things going forward,
so we'll see more of the above.
And we were discussing just this weekend,
possibly assisting with key wording and stabilisation.
So we have architectures, teams in Gen2,
whose job it is just to check that everything works on different architectures
as they get updated.
Without extensive CI tests,
we could semi-automate a lot of that and just say,
hey, we've tested it on 164 and risk-vive and it's all fine.
So yeah, there's some enthusiasm there.
Blackfire has to pass the output of ported,
choosing some really ugly regular expressions.
And we really don't want to do that.
And there was actually a Google summer of code project
to add machine readable output like Jason,
so we're quite happy to pick that up.
So that's on the radar too.
And if Blackfire starts using these,
start using Gen3 prefixes I mentioned before,
then potentially there are a lot more packages
that aren't traditionally used with prefixes
will come under our radar.
So they may be issues lurking now
that will be ready in willing to place.
So some final thoughts.
So derivative distributions typically target a niche,
but this shouldn't be seen as stealing
users away from the parent distro.
The mutual benefit of working together is worth far more.
And I'd say more than that,
just every distro is your friend, right?
You know, we all work with the same upstreams,
so we can all benefit from working together.
And maintaining a distro can be your day job.
I've heard it said it fuzzed M that,
you know, anyone could get a job in open source,
and I thought, well, yeah,
but I'm going to get paid for Gen2, right?
Part two niche.
Here we are.
And although, of course,
primarily maintaining flatcard,
there are other people who do Gen2 as their day job.
And they could sometimes be friction between paid developers
and unpaid developers and due to differing requirements,
time scales, expectations, and we've seen here,
and elsewhere that with the right mindset,
the relationship can prove very fruitful.
Thank you for listening.
Thank you.
Oh.
Don't wait, we got something.
No.
Okay.
You have to go back.
I was watching your cues, I'm not sped
for that a bit quickly.
Oh yeah, I looked like I forgot how to do that.
Yeah.
More stuff.
Okay.
Thank you.