Hello everyone, thank you, thanks for coming, we appreciate to have such a full room today,
it's very nice, tells us that there's interest in similar kind of problems probably
than the ones we are trying to solve.
I hope your thoughts and experiences go in pretty well, and let's try to enjoy it.
If you have questions, we're going to try to leave some time at the end.
So yeah, let's dive in.
Before I can talk about Hateron, I need to talk a little bit about Cares, because you
need to understand the context of why these lessons and what is it exactly that we'll
learn in that process.
So what is Cares?
Cares is, we can call it a framework or a toolkit that takes most known, we could say,
limited distributions and transform them from their traditional OCI image into image-based
immutable system or what we call nowadays immutable system or special property in systems.
It's very nice, because we try to be open to as many distributions as we can cover, that
means you can start with Ubuntu, with DVN, with open suci, and most of these, as long
as there are some small requirements like running certain init system and such.
And it is declarative, that means that you start with Dockerfiles, and it is, and container
base, like I was mentioning, and you use Jamal for the configuration is very similar to
clouding, it's like a subset of cloud init, which for us is a feature, because a lot
of our workloads are for people running cloud native workloads, so that means they don't
have to learn a new tool.
And like I said, we run Kubernetes or traditional Linux workloads, and the main goal of
the project is to simplify data operations, so the idea is that you get, for example, atomic
AB upgrades, and if for some reason, one of the upgrades didn't work, you still have your
passive image that your system running at the edge doesn't all the sudden stop working,
but you have more resilience on those systems plus other features.
And we are 100% open source, there's no hidden feature behind a paywall or anything like
that.
We are at the moment, CNCF sandbox project, looking forward to keep growing in the ecosystem.
Who makes cars, we're basically five maintainers at the moment.
We have 10 official contributors, plus all the other people that have come and share a
little bit of their contributions as well.
We have the moment to organizations backing us up, is Mirantis is backing up one
maintainer, and Spectre Cloud is backing up four maintainers, and we have had three
stakeholders until now, these people who came for a certain period of time, made sure
that a certain feature that was useful for them was working, and then they didn't need
to continue contributing there.
Yeah, my name is Maura Morales, and I'm going to give now the mic to my colleague here.
Hi, I'm Dimitris.
So Tarros initially, we didn't think we would have to build a distro, because Tarros
made an idea is that this distribution agnostic, so it takes the distribution of our choice
and converts that to an immutable OS, and that's all the Tarros features on top, the
two operations, it makes it immutable, you have maybe upgrades and all, and some features
which cannot be put on top of all the distributions, and that's what we're going to discuss
next, but the main differentiator compared to the other similar projects is that it's not
a distro, so you can use your own distro, whatever you have, contracts with or you have
familiar with and such.
So like the previous slide already showed the distributions, we're trying to basen, but
there on top you have flavors without Kubernetes, we have two distributions of Kubernetes,
out of the box, k0s and k3s, and then you have different boards, different architectures
which we support and finally, you have BIOS based device or EFI, so that creates a big matrix,
you imagine all these combinations, you have to produce artifacts and we're talking images here
even cloud images, and there will be artifacts, we end up with almost 400 artifacts on these
to release, which is fine, that's what we're supposed to do, the one thing you can't
do, because it cannot be automated, it is to support all the features on all the flavors,
it this distribution makes its own decisions on what it wants to support, for example,
some of them are based on openersy, some others are on system D, and that means,
okay, you can actually bring all the features, but you have to implement them yourself,
which is a very hard task, so we support what we can on what we can, that's why not every distribution
gets the same features.
Yeah, so of course you can imagine that after a couple of years of doing every release around 400
artifacts, we start hitting the same walls every now and then, so there were some lessons
learned along the way, the first one is that surface area matters, and just to give you an example
of what we're trying to talk about here is that whenever there's a CVE and there's a patch from
upstream, all of our users have to build our new image, or we build the image for them sometimes,
but the point is that a process of image building has to happen, and then a process of upgrade.
This might not be a problem if you're running in a data center where you have unlimited bandwidth,
maybe it's an annoyance at most, but it's not a problem for you, but what happens when you're
running at the edge, which is one of the main use cases that we have at Caires, well you might
not have very good network and in those scenarios, you don't want to be upgrading that often,
but you still want to remain secure, right? So what we analyze is that one of the main factors
impacting us is the C library, so we looked at the different distributions, and we saw, for example,
that I'll find, just to give you an example, at the moment that we were making these slides,
had six CVE's found versus Gnucy library, which had 157. By the way, this is not
comparison into saying what's good and what's bad, not at all. What we're trying to say here is
the state of things. So that means that with muscle, we had to build less images less often.
The other thing is that in terms of size, muscle is one fifth of the library, and that means
that for the finalized image gets to be a lot smaller, and again, if you're at the edge,
you want this smallest image possible. So okay, we said fantastically, let's do everything with
Alpine only, right? But of course, that's not the truth. Yeah, so we based this on the assumption
we have two microphones, we don't rotate so much. So yeah, like we said, the components you have
that the features you can build on, so like we said before, some distributions are based on
system, these, some others on OpenRC, one of the main features, and actually, if you're familiar
with this blog post, let's not puttering is an implementation of that, I would say,
is what we call trusted boot, and at least the middle part, the measured boot part is based on
system D, so okay, you could implement it on OpenRC, I guess, if you want it, but it comes out of
the box and we're trying to use the components as they come, we don't have the capacity to
implement everything on everything, right? So one of the main features is based on that, that's
why, for example, we have this feature on system-debut-based flavors, but as you see,
Alpine is not one of them, so the idea we had, let's build everything on Alpine is not valid.
So when this situation, this telelimatization, where you have three things, and you get to choose
only two, so if you want it, it's more footprint, let's say, muscle for this scenario, and you
wanted the Cairo's experience, I would say, yeah, you can get the Cairo'sified Alpine,
you could get the measured boot feature with the Cairo's experience if you get one of the others,
but we didn't have anything for the combination of small footprint and measured boot, or for all of
them, if you could dare wish for that, but there is one project that actually did that at the
same time we did that, and that's post-market OS, and we thought about going back and say, okay,
why not Cairo's five post-market OS, but like we said, one project has different goals,
it's very hard to align, so if post-market OS decides to make some changes or compile the
kernel differently or whatever, yeah, I mean, we could basically, sorry, it's like yeah,
but if you make some choices, it may break it for us, and it happened before that some
patches they did for example in Ubuntu, Ubuntu was running fine on Raspberry Pi with Cairo's
top, but at some point because they don't use Ubuntu runs Raspberry, Raspberry, they didn't care
about the specific flag in the kernel compilation, which we cared about because we use Ubuntu,
and it kind of broke Cairo's with Ubuntu on Raspberry, and other things like different versions,
other versions of system D, and things like that, so you don't really control what you get,
you get what you get, and it's better to stay after first and not have to patch things like we said.
So yeah, I mean the next idea is obvious.
Yeah, so we went back to the drawing board and we said, okay, what if we build it ourselves,
right, like famous last words, how car can it be, it's just a Linux kernel,
sorry, I'd say it does two words together, plus system D, and minus patches, turns out it's
a lot harder than what it seems, but as we heard, there were some projects doing that, unfortunately
we found out too late, now we finally got to meet here in Fossil, and maybe we got to collaborate
in the future, but in the meantime, we decided, okay, let's build Haydron, what is Haydron?
It's based, the core, as I was mentioning, is Mosul system D, and Vanilla Linux kernel,
thanks to now the patches that have been contributed to system D with Mosul, that also works really well for us.
For booting, you get two options, you either get the booting with trusted boot with EFI,
or you get a traditional what we call standard boot with Dracut and Grub, and we try to stay as
line to any of the projects we use as much as possible, so if they make a change,
recently we were checking that the day after there is a patch upstream, we were already having it
in our pipelines, which is really cool. But now if you notice, we spoke about the main components,
and the main components doesn't include a package manager, so of course that is very small,
which is very useful, but we still need to have ways in which we can upgrade our systems and so on,
and that's the part that plays really well for us with Cairo, because we had already implemented
all of that, so the two of them work together so that we can do the proper life cycle management,
like we were doing with all the other distributions, so you again get all the AB atomic upgrades,
you get the rollbacks, if something falls back, you get Kubernetes, if you need it to,
we can do upgrades through Kubernetes and so on and so forth. So at the end of the day,
we realize that our distribution is going to come and solve all the problems, that's actually
not an option, but we realize that we need to still be using all of those distributions,
but there was a small segment there in the middle where we needed small footprint,
mesh of boot, and cloud life management that we didn't have before, and now with Hadron,
we can cover that part, but whenever our users are going to come and say like, no,
we have reasons to keep using, for example, Red Hat, or Susie, because we pay licenses, and we are
in a, I don't know, we're going in government or whatever, then they're going to stay with that,
we're going to continue providing the exact same support that we've been provided all of them until now
for all of those distributions, the same thing if someone, because their team already had a certain
know-how and they want to continue with the boom-to, they can keep doing that, but the difference is that
we had cases where some people were coming to us, we're buying machines of the shelf, you know,
nothing fancy, small nukes or whatever, and they come with old firmers that cannot boot
200, more than a 200-meg image, then now we can provide through them a solution through Hadron.
That's basically the reasons why we build it, the reason why we bring it here is because
we think through the community is the best way to improve this work, so we want to get feedback from
you guys, questions, if, like I was saying, we met with the people from Post-Mark market and we found out
things we have in common, we would like to know more what we have in common, if we don't have to build
the distribution ourselves, we will not do it, of course, but in this case, it still makes sense for us,
but yeah, it's up to the community as well to reach out, like we're trying to reach other projects
and figure out in which ways we can work together, and that's it, if there are any questions,
we're happy to answer them, thank you.
The question is, how are we distributed in Hadron? Yes, we right now build artifacts on GitHub
and they are through their packages, you can download the OCI images.
We also produce, whenever, so Hadron on its own might not be very useful in that sense,
unless you want to build from scratch on your packages and all of that, the easiest is that
the Cairo's releases will have already the full, yeah, the Cairo'sified version of Hadron
and you can download it, like traditionally we were distributing Cairo's in GitHub as well, yeah.
Yes? How much more? How much more is the Cairo's image that I would like to say?
How much smaller is a Hadron image? That's a good and tricky question, because these are not
sometimes people try to compare them just by speeding up a wound to container, for example,
or any other container, and you have to think that in this case, when you have the Hadron
containers, they already have everything necessary to be clarified, so it's not apples to apples,
but more like an apple's word. But yeah, basically the smallest we have gone at the moment,
I think it's 11 megs for an image, and once it's put on the system, of course, I think we're
talking about a 145 megs, because you have to think it has to come with an active passive
and recovery image altogether, so it only makes what I'm trying to say, sorry, that is really long,
is that it only makes sense to compare it once it's in the system and compare it against other
Cairo's systems, which before with Ubuntu used to be easily one gig or something like that,
for us it's very significant.
Yes?
What's the preferred way to do that issue of software, like if you didn't do it just last year?
Very good question. What's the preferred way to extend the Hadron?
We're trying to base ourselves, like we like to say in the shoulders of giants,
a lot of this work is based, for example, on system D, so we promote that people use
system D, system extensions, that could be one way, why? Because we already provide the trusted
mechanism, that means that you could also sign your extensions and make sure that everything in your
stack is really only the software that you want to run there. But at the end of the day,
is that Docker file, so you can do it any way you want, you can download binaries and copy them
somewhere, or you can build them yourself within your Docker file. So it really depends on your needs
and what's more practical for you.
Thank you very much.