Oh, wow, it's a funeral, that's fantastic, can I start?
Thank you very much for joining.
I got really scared because there was talk about satellites, and there was talk about
robots, and I'm here to talk about software updates.
I was scared that nobody will show up, thank you very much for coming.
My name is Llanovit.
I live in a beautiful city, this is the city where I live in, probably Bulgaria.
I work remotely, and I contribute to open source projects, especially to embedded Linux.
I have experience with several systems for update technologies, and in this talk, I'll
do comparison of some of them, not all of them, some of them.
So this is the agenda, we're going to talk about a better Linux updates in general, we're
going to focus on three solutions that are quite popular, really good solutions, and finally
we'll wrap it up with some conclusions.
Here, let's talk about common embedded Linux updates, the first strategy, which is the focus
of this talk, is about AB updates, we're going to discuss this in more details, but basically
we have two identical petitions, A petitions, B petitions, and we're switching between
them on updates.
The second update strategy is Delta updates, the best description in my opinion is like
GitforFast systems, it's more complicated, it has some advantages, some disadvantages, there
are container updates, and what is most commonly going on is that we have a mix of all these
options in real world scenarios.
So let's start with AB updates, a very quick overview, so we have the two identical petitions,
the A and B petitions, we also need a data partition where we have the persistent information
that remains on the device after the update.
We need a client application that is running only a better device, so this client application
will perform the actual update, and in order to switch between the petitions, we obviously
need the boot loader to do the switch, which also means that we need to restart the system
after applying the update.
For every update system, it's very important to have a fallback, it thinks go not as
planned, you still need to be able to build the device.
So the data updates are really cool because they do comparison of the fall system and deploy
as an update just the different.
Of course, this has a lot of advantages, there are some disadvantages as well, because sometimes
you don't have a B partition in this scenario, and you might run up in trouble if the
data doesn't go as planned.
That's the side by side comparison, so A and B updates obviously, it takes a lot of storage
because you have two identical petitions, it's basically twice the storage for the root
FS, the update size is also in general large, however, you have very good backups.
For the delta data, it's pretty much the opposite, it's a small storage because it's a one
root FS, the update is smaller, which means that you are consuming less bandwidth, and
you still have a row back to previous states, however, the disadvantage is that you cannot
do proper food backup.
There are a lot of open source solutions out there, food doing software updates.
Some of them are A, B update, some of them are delta updates, and many of them are actually
providing both options.
So these three are in both just because in the next slide, I'm going to talk a little
bit more about them, and I have more experience with these three.
Through the years, I have also contributed to actualizer, which used to be a system based
on Libore 3, it was made by a German started company, which after that made it successful
exit, but nowadays the service is shut down.
So in order to do this talk last summer, I did a couple of integrations of Mender, Rauk,
and SW update on Raspberry Pi 5, and I also did integration of Rauk and Mender on this board.
This is an open source hardware board with IMX 8M plus system in the module.
It's designed by ONMX, it's a company in Bulgaria, where I live in Florida, so we're neighbors
and good friends.
So this is an open source hardware device.
So the experience that I'm sharing here is based on the practical work that I've done
for this board.
First, let's start with Mender, how many of you are using Mender?
Okay, quite a lot of people.
It's available as free and open source solution, or paid commercial enterprise plan.
The free solution is for AB updates.
They also, however, have a cloud service, which you can use.
You can also use it in standalone mode, it's up to you.
It's written in several programming languages.
The Mender client will recently rewritten from goal to C++, with the goal to be able
to run it on more constrained devices with real-time operating system.
However, this look, as you understand, it's about embedded Linux, so I'm focusing on embedded
Linux.
So Mender supports the yok to project, but also Debbie and based distributions.
I have to say that on a daily basis, I'm working with the yok to project and open and
better, so the majority of the examples here are about the yok to project and open and
better, but you can use these systems with different build systems or distributions like
Debbie and as well.
There's supports quite a lot of devices, these are the devices that are supported with the
yok to project and open and better.
Raspberry Pi, Roxy, Bigger Bone, everything that's popular is already supported, and in terms
of the yok to project, Mender supports ScarbGuff, which is a long-term release, and it's
going to be supported until 2028.
So if you are starting to project right now, this is a very good choice to use the
yok to project with the long-term support release, and if you have some of these devices,
you can quickly try out Mender on them.
This is how Mender works, but in general, this is valid for other systems that use AB updates
in general.
You apply the update and after that, you have to reboot the system, however, for Mender,
what is specific that you have to inspect that the system boots find and to do a commit,
to make sure that the update is finalized and everything is okay.
So Mender works in two client modes.
You can use it in a standalone client mode, or the mode where you can use the Cloud Infrastructure
with Mender for management of leads of devices.
So this is an example of how to do the integration with the yok to project and open, but
it's here just for reference, there are some specific variables that you have to set.
We don't have enough time, so I'll just skip and fast forward, but if you're interesting
in this type of things, please talk to me after the talk.
So Mender has the data partition, this is the partition where the persistent information
should be stored, so across AB updates, you're updating the rootFS, on the next update,
you have a new rootFS, so whatever your applications need must be stored in this data partition.
And also the Mender client uses the data partition to store its own configurations.
The advantage of Mender is that it has several adults, so it's not just for software
over the error updates, but you have several adults for additional features that you can
do with your embedded device.
Mender also supports delta updates, however, this is part of the commercial plan for Mender.
It's a close proprietary solution, and we're at Phosem, so I'm not going to talk in detail
about this, but yeah, it has it.
The next thing that I would like to discuss is Rauk, how many of you are using Rauk?
Wow, so many people, all right, so Rauk, it's an open source, a lightweight update
client that runs on the embedded device, it has a lot of advanced features, it supports various
multiple updates and areas, the advanced features that are really cool are HTTP streaming
and adaptive updates.
And security is a very important part of Rauk, it has been there since the beginning,
so Rauk is really good if you are looking for flexibility and a truly open source, fully
open source solution.
The licenses of the different bits and projects that are available for Rauk is the source
quota for Rauk, is available under the Rauk organization in GitHub.
So these are the stats, how to do a proper Rauk integration, so if you pick up a board,
that's a very custom and you haven't run Rauk on it, you have to do these things, you
have to select an appropriate booth loader, most of the time it's your booth, but not
necessarily, you have to enable SquashFS, use X4 as a root file system, and create specific
partitions, well this is something specific for the Rauk to project an open and better,
and we have examples for this as well, and there's a Rauk system conf where you put configurations
how the slots are configured, Rauk is very flexible, so you have different options how to
deal with the data partition.
You already know from my previous slides why the data partition is important, the good
thing about Rauk is that you can cover various different scenarios, whatever fits your needs.
I most of the time I use this one because it's simple, but not necessarily, sometimes you
may need something more complicated.
The advanced features, the HTTP streaming is really good because if you have a server that
supports HTTP streaming, you can download and install the Rauk update simultaneously, and
this is very convenient, save stimes, and also you don't need to download a huge file,
store it somewhere, and keep in mind that if it's constrained and better device, there might
not be enough space on the device to download the data, I've been in this open situation,
and the HTTP streaming is really good because it solves this problem and the adaptive
updates is something that I've mentioned because it's kind of delta updates in a different
implementation, so as you can see, the solutions are going to combine strategies, it's
because people need it, and step by step we're getting there, we have different solutions,
but they're all going through this path of combining strategies for updates.
Metal Rauk community is a young to open a better layer that I started five years ago.
They deal with to have a very simple demonstration, how to use Rauk on Raspberry Pi, soon after
that, other people started contributing to it, and we converted this from Metal Rauk
Raspberry Pi to Metal Rauk community, and Riko from Pegatronics approached me and said,
would you like to move this layer as part of the Rauk organization, I said, yeah sure,
and now we have several machines that are supported in Metal Rauk community, I'm the main
trainer, contributions are always welcome, so if you want to add another word, you're
welcome to do it, there's a lot of work to be done on Metal Rauk community, I'm doing this
primary in my spare time, and we can do a lot of improvements, so if someone wants join,
if someone is looking for an open source project to contribute, Metal Rauk community is there,
and we need your help. The next solution is SW update, who is using KSW update, quite a lot of
people, so it's another flexible open source update framework with small footprint, and it does
automatic updates, it's again compatible with the Yoke to Project, but also with build
route and the packages, as far as I know with the Depp packages, it's experimental, I've used
SW update again with the Yoke to Project and Open and Better, so my experience with it is true this.
Here are the licenses, and I would like to do, as we are moving on, I would like to do some
side-by-side comparison of the features that Mender Rauk and SW update have, so all of them are doing
AB updates, all of them are good at doing AB updates, so when you are picking up a solution,
each one of these solutions is good to do AB updates, each one of them is good to do rollback,
rollback is really important, because you need your system to be always working,
and the rollback provides you an opportunity to switch to a non-previous state if the
data you've just installed is not properly working. Mender has some additional adults that I mentioned,
these are configured on the monitor and the travel shooting adults, these are things that are
not directly related to the software updates, it's just the features that you get with Mender
as a bonus for managing your devices, they're not present in the other systems,
how are the other systems have some additional unique advantages? For example, SW update has a local
weapon interface, which means that it's a web server that runs on the embedded device itself,
and if you are in the same local area network, you can open in a web browser this web server
and upload your update through the weapon interface of SW update, which is quite nice in specific
use cases, most of the time I'm using all these solutions for software over the year updates,
but not necessarily, sometimes you can do USB updates, which means that someone is going
with a USB stick, plugs the USB stick, you have to create your deaf rules and automatically
to install the updates, sometimes you can do this local web interface.
So, this is another side-by-side comparison, which is how the implementation, the licenses,
the programming languages, so for Mender, this is really curious, I know that go is a big thing,
but they had to switch from go to C++, so technologies are staying, although we see a lot of
new technologies, still some solid technologies are used, and it makes sense to use them just because
they have more constraint footprint, and we have, yeah, the other project integrations here,
all of them work with ScarbGuff, for Rout, we are also maintaining the master branches, so some of the
boards are working with the cutting-edge with the master, the contributions, Mender, and Rout
are accepting contributions through the standard GitHub pull requests, with SW update, it's using
main-inclist, and the measurement server is an advantage of Mender because you get a
turnkey solution, while the other systems are focused on the open source and the Linux implementation.
If you're looking for a third-party server to use with SW update, or Rout, there are some options,
like a clip scope bit, this is an open source solution, QB, also you can use AWS, IoT, and integrate
updates through it. One project that's worth mentioning is Lip, Uboot, and Anvil, so basically this
this is a project that allows you to read and modify Uboot variables from the Linux user space,
and this is super important, if you're doing software over year updates or software updates,
with the three systems that we mentioned, because somehow you need to tell the bootloader
that it's time to switch, and this is done by setting a variable there.
Another thing is combining these solutions with containers. Yes, that's possible. I know that there are
a lot of people in love with containers, that's something that's widely used nowadays, even on
and bad devices, and you can integrate, for example, Docker with any of these solutions,
and I recommend you to make sure that the container stays on the data partition as in the
remains after the data of the data first of the device. So at the end of the talk, I would like
to discuss the to wrap it up with some conclusions. One question, is there anyone here who's
using a proprietary in-house solution for software over the Arab days?
All right, one, two, oh, quite a lot, okay. No judging.
All right, I think it turned. So it makes sense, especially if you have started this many years ago,
and you have to maintain it. So no judging here, I know that there are no universal solutions.
However, if you're starting a new project today, I highly recommend you to pick up an open source
solution for software updates and just focus on the core features of your product, because things have
changed over the years, the past decade, they're really good software updates solutions nowadays,
very solid. I think the biggest challenge is to pick the one that suits you best,
sometimes this could be problematic. So make sure that you're picking a solution that fits your needs.
The duo AB update mechanism is a good choice. It takes more storage, but it's more convenient,
because you always have a B-partition that's there, and you can switch to it.
Obviously, depends on the boot loader and you need to reboot the system.
So MenderRowk and SWV update, all of these solutions are really good for AB updates based on my experience.
There are other good solutions out there. I just have more experience with these solutions.
That's why I'm talking more about them. MenderRowk is a little bit hard of game, because
it provides a turnkey solution that you can just use with the whole fleet, with the whole
management UI in the cloud for fleets of devices. And also, of course, Delta and Adaptive
updates are available for some of these solutions, like addition to the AB updates.
And once again, choosing the best solution, open source solution depends on your needs,
but I highly recommend you to go with an open source solution. Thank you very much for your time.
Any questions?
Thanks for that. That was great. Cut a couple of quick questions. Have you worked with Foundries.io at all?
Because we use them, we find them to be quite good for Delta updates, containers, or let's have a chat.
How big are you for? Foundries.io.
Yes. Actually, I think it was here the slides.
Okay, so they have actualizer light. This is the name of the open source solution actualizer light,
but Foundries are the major development there. Yeah. Cool. And, you know, we mostly were with
about the Linux, with the Octo open and about it, but we do some sort of art class-based stuff.
I would try to quite useful to be able to leverage something like, you know, an existing open source
framework that can also be deployed down onto, you know, some of these efforts. I had a little bit of
a lot with Mender, just updating ESP's. And I wonder if you know of any of these other
solutions that can actually sort of be applied to more of an art class environment like Zephyro
for your art class or something else? Um, yes. Well, I'm not, um, more folks on the
Linux. So I don't have practical experience with real-time operating systems and Zephyro
in terms of software updates. I know that Mender is now supporting Zephyro. I think Mender,
actually, yesterday we were discussing this with a friend, Mender, and he recommended the Mender,
I haven't used Mender for however. I recommend the Mender for things Zephyro updates.
And the whole reason why Mender read written the client from go to C++ was to support these
real-time operating systems like Zephyro.
Thanks. So if you have a more complex system to PIDATE with more,
to be components like FPGA or my controller that you still also want to PIDATE,
I know that you can bundle those binaries within as the PIDATE bundle, right?
Can you do that also with rock? Or would you suggest one of the other for this kind of more
complex updates? Well, yeah, it's a good question. And yeah, it's a very valid scenario.
Different solutions that we covered. They have different naming, for example, some of them
called the file that you distribute to device artifacts are called it around bundle and so on.
But in general, with some adjustments, you can achieve this with any of these systems,
because you can deploy the update. I'm speaking broadly here. And after that execute the script
and many of these solutions have these hooks that you can execute a bunch of scripts to
finalize this relation and deploy the different components of this update to different
even different hardware devices. I hope that the sensors question in general.
So you mentioned Mender has the server side counterpart that rock and SWP don't have.
That's an advantage, of course. But question is, is it kind of mandatory to have it or is
it supported to use Mender without? And if it does, does it make sense in some use cases to
use Mender without the Mender server counterpart? Yes, it is possible to use it in a standalone mode
where you can run, for example, an HTTP server locally and do the update from the server,
actually during development, most of the time I'm doing this, because it makes sense from the
development perspective. For your other question, since Mender is a commercial enterprise,
probably they would say no, I'm not sure if it's a good idea, actually. But yes, in general,
you can use the Mender artifacts with a third party solution for managing the devices.
Talk to me later. And the guy would have has really cool stickers.
So if you like three stickers, quiz the man.
I have a question here. You mentioned that rock has some support for streaming and know that there
is some support for delta update, but it relies on CAsync and it is not clear to me if CAsync,
or casting a country where Mender is pronounced, is still maintained somehow.
Yeah, but they switched to adaptive updates, which I think uses a different implementation.
And for the HTTP streaming, yeah, I use it on daily basis when I'm dealing with rock,
because it's very convenient. The only trick here is that you have to run a server that's
supports HTTP streaming. Okay, I think we run out of time, sorry, but I'll be around so if you have
any additional questions, please reach out.