All right, for the next talk, we are going to tell us everything about generative
immutable and the applicable safety or input in decarimages, let's go to the end.
Yeah, thank you everyone, and nice for feeling this room so quickly.
So what I'm going to talk about is a bit about us, so this civil infrastructure pattern
project, what is this links foundation project that we created nine years ago, it's
goal has to enhance Linux for long-living industrial critical infrastructure use cases,
you may run it on some of your devices, but you don't know it, or some of your use
cases like a train.
We have a lot of things that we do in the project, maybe you heard about the CIP kernel
before, something which lifts that long as the other ones we heard about in the previous
talk.
We do a real-time for that, or we support the real-time project on that testing, additional
testing, we work on security certification for that domain we heard about before and device
updating, principle for us is upstream first, own project seconds, members are coming
from both suppliers of that domain as well as users.
One of the users, so to say, is my employer Siemens, I'm working for Siemens Technology,
foundational technology use days, doing in-house Linux consultancy and development, in
the CIP projects, I'm leading the kernel team, but I'm not doing the work, other people
do that.
I'm maintaining the ESA CIP core project in that project as well, we will hear about more
in the following slides.
And yeah, there are some other projects I'm maintaining and contribute to an open source.
Yeah, that's my daily life, so to say, so we are using Debian, you may wonder why,
for that kind of domain, well, there are a lot of reasons, back and forth, but anyway,
we consider it a mature, high quality mainstream in-line of distribution with a lot of
features, you may need it runs on old hardware, and when you want, it scales down from
small to big, which is also important these days, not too tiny, yes, you know, but small
enough for all use cases.
Yeah, it comes with security updates, surprise surprise, even these days, up to 10 years,
that's also quite important for the domain, and yeah, it's a strong open source community
that we like to build upon.
That's why CIP, when the only CIP, but also when it was before, selected Debian as one of
our feet and one of the pillars that we build upon.
So what are the missing pieces in our puzzle, so to say?
So we're not installing servers, we're not installing Linux systems on train control systems
or on IoT gateways, we actually flashing complete images to that device normally, in the
factory or in some kind of deployment process, and then these devices travel away out of
reach, so there's no one typing in some passphrase for fallback, these things have to
be really robust to be updateable in the field.
That is different from the classical server system I would say.
So we use very conservative approaches here, so you know, you don't see any kind of rocket
signs in the future in the next slides, it's really really basic stuff, and some of them
is probably too conservative by now, but anyway, we want to have this AB pattern for updating.
We want to have the artifacts for that, obviously as well, so not packages, but images
or Delta images, and all that has to be, well, these days, securely booting, obviously.
There must be no gap in this boot process between kernel and file systems, where to
before, what's different from classical installation on an XSX box or something like this,
we sign usually ourselves, we don't trust the existing key infrastructure there.
Yeah, and we also have to customize a few things still, and we try to reduce that, well,
we're using a binary distribution, obviously, but there's something always like the
kernel, for example, for various reasons, but also other small pieces, not hundreds, but
other tens of packages that we like to customize for that.
So how to do that, how to express that, they have the other talks about this before, just
to give you a bit of the background, we use a build system called ESA, which is a, well, I
say it's a unique combination of describing both package building for deviant as well as
image creation, in one piece, so to say, in one repository, still, just like
Octob build systems, you can reuse from that, a bit that we actually do, we use the bitpake
task engine for that, we use the big image, for example, we use some other pieces from the
way ecosystem for building up this thing, but in the end, we're generating deviant packages,
the deviant way, and integrating them mostly the deviant way. We use the layering idea of bitpake,
so that you have a core, then you have this ESA, because we talk about maya later, and then
put your product on top. Those layers can actually weigh more, if you look about large systems,
but that is possible as well, so we have better reuse of these customization bits, even if
they should be few, they pile up over the time, so this is what you can stack this way.
And we have a configuration management tool again, just for reference cuts that we use to pull
in all these repositories together. Now, this is a normal way, classic way to do AB updating on
better devices, on critical devices, so the goal for us is really to not touch the running bits and pieces
for the version A, while we are preparing version B and trying out version B. That means
no shared file system for the critical bits, really separate paths, and that is the way we'll
talk about it before in the past, how we built up the system, we have boot system, which is
immutable, we have root file system on there, and in the end, you have some consistency,
some state that you keep with the device, but it is usually less, and yeah, it is, this is the
other day, the only, currently the only single point where things move together, but generally
you have separate passes there. You have some update agent running on the device to manage these
kind of updates and switch over, but still if you look at that, probably you can build it up,
shouldn't be that hard, but still some questions pop up, if you work on that, like, okay, what
do we do with the boot loader? Well, you would, you if you what do we do there? How do we build the
UKI's, and how do we build the UKI's, which we do for a better, for example, if you have
the overlay or explains the replaced the DTV's, the device trees. By the way, overlays for your
root file system, for that consistency of that, how do you describe the, how you build it up,
do you do a rollback for the consistency as well, so that you can update not only the root file
system, but also the consistency of this case, and how do you build the chain of trust between
all the elements? Now, small pieces, small to the side, also like, for this encryption,
we had a before, how to hook it up with a TPM or whatever hardware security module you have there,
and obviously, which agent to use for the update management. So, all of these things,
basically, have to be answered, you can answer them individually, and this is what many products
in our environment, or it did before, some did it. Good enough, others did it, well, improvable,
and the end, you end up with a lot of knowledge spread out in projects, and that is obviously not
optimal, because one mistake in that thing, wherever, and your system is in danger, or at least
it's becoming not that reliable as it should be. Let's one of the ideas behind pulling this all together,
in at least one way to have some kind of best practice sharing this way, and distributed to the users.
So, what is now is us here, because thing that we are here, pulling together, it brings
some decisions on the table, you can follow them, then you are happy, you may not follow them,
okay, then try something else. These pieces are, for example, we decided to go for your if
base booting, even on the better devices, which these days becoming more commodity, if you think
about system ready, for example, but anyway, this is the decision we made. That means we have
also here, at this stage, not using system D boot, but using UFE boot guard, which is also
UFE-based boot order, more for embedded, without any kind of menu or something like this,
as a switcher between the AB path, and also currently still as the generator for the UKI
images, which has historical reasons, but now it's moving forward also on system D side,
we have, for example, the DB integration there, that was one of the reasons to do the ourselves.
We have key logic in, for the boot paths description, in an interim of the hooks,
these days, so that is about how to select the AB boot paths and how to build up the chain
between the elements there, integrity for the root of S, configuring overlays, as I said,
partitioning groups and all these pieces, and nowadays we are also working on these AB snapshots
for the consistency. Another decision we just made, well, there's ROG, as we know,
but we've grown for SW update, a long, many years ago, as device agent,
we have the ROG-Dorbin overhandler for that in that context as well, so that we can switch
between the AB paths here, and obviously support or integrate, so to say, what SW has
supported, like encryption and signing of the artifacts. We also explore the other image
update for that, so if you want to go deep on that, there's a link there, and yeah, the
plumbing that you need to do a read-only route at best in this case for the ambient on top of that.
So how does this plug together? So keep in mind, we are doing off-device configuration, building,
and signing, so that makes a lot of things easier in the sense that we do not have to deal with
the problem of a normal distribution, how do you sign something on the device, and whatever,
or how do you resolve it, you don't have to sign a device, we do it anyway of device,
that's the advantage here. The layer comes with a bit of help us to do the signing against HSM,
so how do I model, or whatever you have there for your build procedure on that. And then it describes
this bootchain. So again, the details are in other talks, just to give you an overview on that,
as I said, if you boot got selects the AV path, we put an image studio ID and the
clarity hash into the inner drum with S, so similar to previous talk, and sign the whole thing,
and that basically change or put to the next level, so you have then, so to say, the identity
of the next element, the route S, we encrypt, also on first boot, the partition, and decrypted it on
following against the TPM, or an FTPM, with here on the arm side, for example,
other system D, or all the systems, also use the cleavest script sets,
be used to invert your hash again, to select the corresponding route of S,
swanship S, or error, as these days, has an option for the persistency snapshots,
we just hook it up again with the UUID and say, okay, for that route of S, and for that UKI,
take that corresponding snapshot from, yeah, in this case, butter S, at the current stage,
but it can also do other things, and select that thing, and then you have to say the whole chain
built up and then securely identified, and then finally mount the overlays on top, coming from
bar, we have to describe those few bits, or more, hopefully not the whole system, which should
become, yeah, overwriterable, and then persistence across the bar updates.
So again, these are here, because it provides these kind of building blocks, and it also provides
some images for that, but those are for reference and for testing, not for protection.
This is normally what you do, and this is what our users do, they built on top,
the actual production description, how the device should look like, what it should do,
what it should use from that, and what not, but this is the normal idea behind it, so it's not
likely to get a final image from that call layer already, that's what you do on top for your
actual design. So, no demo for me, but if you want to try it out, play around a little bit of
that, go for the repository, clone it, the build system requires a privileged Docker or a
apartment environment, but then otherwise just type in the command and that menu pops up and select
something from that, this support x86, I'm 32 bit 64 bit and even wrist five these days, so
looking forward to Pixie, to fully support it, not all features for wrist five yet, but yeah, it's
coming. And you can start these things in QAmo for example, x86, you can just touch it on an x86,
domain machine, it also works, and the other devices are more special in this regard.
So, yeah, that's for trying out, okay, sorry because build it folks, your logo is meshed
anyway, we need also reproducible images, bit identical, why is it essential? Well, supply
insecurity obviously is one reason, but it also makes the data a bit smaller, so if you do not
get random output just by changing one bit or not reliable output, so this is another important
step, but we work together with reducible for quite a while in the context of deviant,
but not only, so many packages they are already reproducible, it's helpful for us,
we try to close the main gaps there and now focus for us specifically on the aspect of image building
reproducibly, so like for there being, for example, it's about the ISO image which should be
reproducible, but we do that also with the product images, sort of say, so many not all of the
ISOs here, we call images produced right now are reproducible, between the lot on the file system
production, we also have to patch a little bit on open, and better for that regard for example,
we also patched some tooling which is needed for building, yeah, the ESP is in things like this,
so does file system, there's something to open there, but anyway, we have a solution at least downstream,
we work with stifoscope that you can actually scan complete images, disk images, do the visualization
of the elders there, and we have a weekly pipeline running which checks that our images that we have
currently under on the radar are kept reproducible and nothing is regretting there, yeah, in general
we try to work with a deviant upstream more and more, so we or you continue to talk about all bits
because we don't want to keep them all the time in our layer and we don't want to keep them as
at hawk package building, so if you boot got for example, it's an upstream package plan out of
the dependencies as well, we took over the maintenance for that, we worked this SW update upstream
where it becomes a distribution package and not just a york do package in the sense,
so it was very much focused in the past on being built on configurable, for distribution,
by distribution, it obviously more flexible, runtime configurable, it's now doable,
so yeah, it's an official deviant package, we use it, in some cases we have more version
of our deployment for historical reasons but the future is always going for the upstream package.
We also worked the step-in snapshot.org, so the infrastructure to have a reproducible package set,
it can pick from, that is working way better now, thanks to Felix and the room,
we are still trying to improve a little bit on, in the drummer's production there's an open issue
we're discussing to have to be more complicated and there's more to come on that.
So what are the future plans, so we're never done, next step is coming, as I said,
we are currently working on the AB snapshot for the persistency and some good approaches there,
we have something in-house already employed, which is conservative but it's working,
we're going to look in something more new, it's the butter effect approach and there are some
open questions and I had already discussed in these days with some folks here that could be done
different, you should be done differently but we definitely have to work on is the recovery procedure
and the reset procedure, also factory reset, encryption has to be looked upon again if this is actually
a good stacking up here right now with the grip underneath, there's some problematic cases where
you don't want to have butter fs and the mid-teen, for example, obviously you don't want to have
a snapshot of your locks, that is currently, is not expressible but it can be made, it's not that
hard but there are other use cases and it's not should be the only solution for us,
so as I said, we have internally already the deployment with the snapshot for the block players
so to say, there's always three around, we know that, we will have probably more alternative
in the future to describe this pattern with different implementations depending on your demand.
Delta update improvements for an Ableman actually for UKI is technically not the hard thing but
that's what I was doing, we are close to that, yeah, one of the weak points I have to
bit as maintain as our documentation, so it's a little bit not that easy to get started,
it's the demo mission are there but if you go from there to your own product, that way may
be, should be made more convenient and more clearer so we will work on documentating the interfaces
that we have there and providing more light up HelloBulls currently are probably in.
Already work internally but also something that's not that easy if you look at it as a measure put
so that we can even go up to remote at this stage and at some stage there are some use cases
fortunately currently we are getting very far the verified boot but yeah it's coming.
Also discussed already was from folks here, how to do, maybe how to make things more reusable
that we are currently building for the inner drama best in the turn out that we probably should
look into drag-out soon for the for-devian, so yeah people have to look into if you can make some of
these pieces possibly reusable even outside of the special domain we have, they will see it and again
this is a collection of pattern and ideas so we will also look into further evolving yeah as the
distributions for the class of use cases evolve if you can pick up something from that and reuse it
ideally we don't have to have all the gluing done ourselves but rather doing well we using more
what is already there out there so with that again focus on on this activity is really on the
robot unattended software update in the field locked secured that's all possible with that in theory
in practice you have to do a lot of plumbing and if you do it wrong you have problems possibly
we really want to provide these kind of best practices that we are using, share it this way
and evolve it it could be blueprint you can just in getting inspired from that you can use this also
maybe in a context of a yup to build a built-out system but well if you're using it on a
deviant system then you have a lot of pieces that you can reuse we wanted according to test them
certify them and maintain them where needed look at the repositories and if you have any follow-ups
on that there's a meaning this as well so we happy to take input and feedback on that as well
with that thank you for your attention
is the device software or device programmer also open source and they still need to trust in proprietary
so the question is if the firmware is open source it clearly depends
so obviously we try to have it open source so some again for a cement some of our products actually
use co-about others don't we use u-boot then it's open source but that could also be proprietary bits
it really depends it's not part of c-i-p so we keep the firmware so to say out of scope for us
it comes with a device at least this is the vision and that's why it's not in clear answer to that
yes please
I want you to know is there a bit of a busy fetching the feedback from the computer
repository or is it fetching the sources from the package so the question is
if we are fetching for the build procedure the binaries from deviant upstream or if you're just
fetching the sources yes yeah so both and but the normal case is as I said we want to use the binary
parts of it so we are fetching the binaries and we are relying on reproducibility ideally
we are patched fetching from snapshot if we want to have a specific version you pinned it for
example if you have to customize something or if you have to let's say use the package which is
going to be in the next release of deviant already now then we go for salsa for example and pull
the depinization and the sources from there or we even go for upstream differently and pull our own
deviantization on top and build it at hawk during the image build process as well that is the
as I said the unique feature of the user build procedure that you don't have to do it separately you
can but you don't have to you can also describe if it's a single repository the whole procedure
from customizing to integrating an image in more questions what is slides which kind of versions
get the CIP like the SLPS so the question is about our kernel okay so which version do we pick
so we started with four or four some nine years ago eight years ago and so far the regular
schedule is every two years we pick an LTS kernel and declared to be an SLTS kernel that just happened
again for six or twelve and that's currently the schedule the decision is done by the members of
the project because we in the end we have to carry the effort and the commitment for that so it's
happening within the CIP workgroup and the CIP technical steering committee in the end
for the question is if we want to use system the what extension so it's extension yeah so
if we want to use more of system the extension for that the concrete plan doesn't exist yet
but as I said we want to evolve that over the time so we currently went for the
pragmatic simple approach of plugging a lot of things out together ourselves and again it grew
and now the point is actually to revise and think about again if it grew to far if you can start to
reuse so if you look for example we started this cleavace because at that time system the
CIP Dendol wasn't available for the version that we were running there been 11 nowadays obviously
we used the newer deployment system decrypt and roll and further things will probably
the time yeah come automatically if the step for example towards breakout is happening then
we will also look obviously more into what can be reused from there we don't want to reinvent
wheels if it's not really necessary so we rather want to add then the bits and pieces to that but
yeah that happens over the time as you know you may also miss the point where you should
better take something which is all you existing by now okay okay that sounds very interesting we should
discuss that
one star one star one star one set so the question is about how do we what do we do with the
watchdog or how do we do the monitoring of the AP yeah so that's what I think I stepped over
because of the time yes one of the reason we have our own bootloader here is that again how to
sell watchdog drivers to if he boot got sorry to it was system reboot and we actually use the
watchdogs from your boot where available so when we're booting from your boot you boot is
responsible for setting up a watchdog to monitor the complete boot procedure up to the point
where your system is up and running again where we don't have that on X86 for example we use our own
watch watchdog starting system so to say our watchdog driver in an if you boot got which
starts the watchdog and hands it and that's it then hand over to Linux or Linux can continue that
and finally yeah stop it so to say or or take over the running of that watchdog this is part of
the concept yes so that we can actually detect if something gets stuck kernel booting in the
drum with a starting over whatever looping later on before the final system is up and running that
would be detected by this watchdog pattern let's go pan biops so we're spanky all
that