Hi everyone, my name is Andrei Andriyushan, I'm a DevOps engineer at Analog Devices and I'm excited
to talk about our process for refining the release strategy of a custom Linux distra.
I will share how we've optimized the release process for a Kuiper, which is Analog Devices
Linux custom Linux distribution.
And I will start by presenting what was before, sorry, I will start by telling you what
is a custom Linux distra, what is Kuiper, basically Kuiper serves as the primary distribution
for product evaluation boards, offering reference designs, device drivers and a variety
of development capabilities.
So includes pre-build boot files for over 140 FBGA systems, sorry, base projects and
30 plus Raspberry Pi designs, and provide access to 130 plus device drivers for seamless
integration.
Also supports application like libio, scopy, I also love scope, which can run directly
on embedded boards, or via wired wireless network, USB or serial port.
Note access is available from Windows, Linux, and Mac host enabling the data gathering and
post-processing using Python, MATLAB, C-sharp and more.
Also is compatible with silings, Intel, Raspberry Pi, and was recently tested on NXP.
Platforms, basically Kuiper simplifies the accessibility for embedded system applications.
Now by the end of this talk, I will want you to get insights into what was the old
really slow of Kuiper Linux in order to have a better understanding of the changes that
were made and how they impacted the performance, which are some of the major improvements
that we needed to integrate in why, and also the current state of our custom Linux distribution.
The traditional process was slow, rigid, and printed delays due to lack of modularity,
and the high-intended dependence between components.
All the software components were built during the image creation, which meant a longer
build time, and another reason for the long release cycle was that the software components
were released through Kuiper image, and with no modularity, a bug found in one of those components
would delay all of them in the end.
Starting from this, we identified the necessary improvements that needed to be done in order
to bring Kuiper to the next level.
Now we came up with a few ideas that we then implemented, then we then implemented, sorry,
and the key modification are creating a stabilization branch for each release component,
breaking down the monolith structure to a more modular one, also is better for our software
components, because we can separate them during the release, and each component would
have a separate release.
Also we would create debion packaging for those components, like libraries, user application,
those that I previously mentioned, like Scope, Libion, and other stuff.
Also, a modular configuration for custom images based on the other change invention.
So we expanded the component level testing with a modular custom Docker image, because
now if our image is more concise, we can create Docker images and test those components
directly on them, and after that we will be sure there will be integrated faster and
cheaper, and also improvements were made on the CI site by optimization, like running
on GitHub, introducing PR templates and code owners for a better collaboration.
Now let's start with the improvements.
I will first talk about the stabilization branch that we introduced.
This was added in order to isolate the release development.
This approach allows us to focus more on preparing the release, while the main branch
continues to accommodate new features and updates.
The result, faster, usual resolution, more focused testing, and a stable release pipeline.
Based on the diagram below, you can see that the master branch serves as the main branch
development.
This branch, the features and the fixes are initially introduced in master, and also
in the next table, which is just a stabilization branch, that is forked and prepared for
the release branch.
After the next table branch is created by branching for master, this branch is dedicated
just for stabilization, ensuring that the code is closely tested and prepared for release.
Fixing updates are often pushed, I mean, are pushed both on the main master branch and
also next table.
Once the next table branch is considered ready for release, release branch or release tag
is created from it.
But during testing, during testing, we find bugs, and that's why we need mostly all the
time to create release patches.
That's why new commits with fixes are uploaded on the release branch.
After release is finished, as you can see, the next stable branch is overwritten with the
latest changes from master, which includes also the other features added been while,
in my release stabilization, and is prepared for the next release cycle.
Next, we move the way from a monolithic structure for monolithic release process and separate
separate the entire release into smaller independent components.
In the past, everything was bundled together and a bug found in one of the components in one
area could delay the entire release.
Now components like software tools, libraries, and boot files are managed separately independently.
By breaking down their release this way, we've eliminated unnecessary interdependence,
between components.
Each component can be developed, tested, and released on its own schedule, leading to
faster iteration and reducing risk.
Each software component has a deviant package created for it.
It will be uploaded on our Linux repository, and this way, we've simplified the distribution
and made it easier for users to install and update their components and utilities like
libraries.
We're all back to a previous version, if that's one doesn't work with the current state
of the cipher.
But one of the most significant steps that we've taken to increase flexibility is the
introduction of a modular and customization of a modular file.
This files allows users to build custom images by selecting the version and the components
that they want the image to include.
For instance, if you're working on a project that doesn't need scopey, it needs just
libio, you can specify the configuration file to install just libio, not the other components.
The nice thing is that you can also specify the version of the component that you want to
be installed.
This approach not only makes the release process more modular, but also gives users the freedom
to create their images how they like, ensuring that no unnecessary components are included
in the image.
Not just regular users, benefit from this, also internal teams because this internal
image is customizable and they can integrate their product more faster because we added
an extra script and this way new components or additional dependencies can be easily added.
There's a stage in the image creation where you can install everything you want at extra
libraries or dependencies or run your own best script.
Let's say that install something is external not only analog devices components.
Now on testing site to improve reliability and compatibility, we've implemented a robust testing
process that works at two levels.
First, each component is tested individually to ensure its stand-alone functionality.
This allows us to quickly catch and address issues specific to that component.
Second, the components are tested as part of the pool-chiper image.
But with the twist, because we've introduced Docker lighter images that are built specific
for that software component.
This way, this Docker images replicate the chiper environment during the development process.
By testing within this controlled environment, we ensure that the software components are compatible
with chiper during development cycle.
This approach allows us to catch integration issues faster and
rather than later stages, which saves significant time and effort.
Now, if you're realized that you've seen familiar with chiper,
maybe you've participated on my colleagues presentation from today
about advanced test harness infrastructure for validating
arm and FPGA base systems, which is here.
He also presented how chiper is integrated in this testing process and how it helps.
Now, to optimize our CI pipeline and make the development process more accessible and efficient,
we transition from Jenkins to GitHub actions.
This move brought several benefits.
First, bid logs are now available to the users.
Increasing this by transparency.
Additionally, fresh bills are available to customers and users
after a push is made on the chiper repo, and they can choose and download those images after
like, I don't know, two hours a commit was pushed on the full image one.
But also, we've expanded the build configuration, because previously, just a 32-bit architecture was supported,
and now, we have 32 and 42 bid architectures.
And for this two-type, we have also a basic image, which includes just,
which it doesn't include decks of the environment, it's just the basics,
but we also have the full image that includes all the software components and stuff.
For this, to further improve collaboration, we added also pull request templates.
Because it's an open source project, you can go search through the code and see a bug,
you can actually create a PR in order to fix that bug.
And to be sure that we know from star what we have there,
you need to check a pull request template, and also to be sure the right person's look at that PR,
we also added code owners to streamline the reviews and approvals.
This changes like the build process faster, more flexible, and more user friendly,
while also encouraging better collaboration among developers.
Now, let's take a closer look at the current state of Kuiper
of our and our release process.
By using a configuration file,
users can now custom and create their image mapped to their specific requirements.
They can select only the components they need, and ensure, and this ensures lightweight and efficient builds.
We've shifted from building each component independently to, we've shifted, yeah,
and releasing as a deviant package, not through Kuiper itself.
This packages are then uploaded to our Linux packaging system,
which allows for faster and flexible builds.
This changes also simplifies updates for users, and it makes easier to install and roll back
to the version of the component that you need.
By dividing the release cycle of individual components,
we've insured that bugs that are found in ones of their components,
one delay and affect the whole release process for the entire image.
This approach simplifies the overall process,
minimizes bottlenecks, and increases development efficiency.
So, this is the end of my presentation, and if you have any questions,
you can write me on this email address, or you can ask me here, or link to any questions,
as come alongers, yeah, that's it.
Yeah, sure.
So my question is now, what you've described is that we moved from a whole distribution release,
we're rolling release cycle for this script,
because we're using individual components individually,
and for how is it decided which components in which question go into the whole release them?
Yeah, well previously, we released Kuiper, all of the components were installed.
So, if a bug, as I said, is found in one component, then we couldn't release all of them,
because we needed to fix that, but now, as we release the components separately,
in the end, we can release the Kuiper image itself, and if one component is not compatible,
it will be released after that, but we can release with later version of that component.
So, yeah, it's more modularized now, it's not a whole, as it was before.
No. No, you can have Kuiper as a whole, but also individually components.
Yeah, we have sometimes them.
What kind of framework do you use in the previous version of the new version,
whether all components, components, or did you use anything in the resource store?
Well, most of the tools are open source, because we integrated analog devices,
software tools, and basically, Kuiper is created to integrate easily the analog devices tools,
and we work open source. So, this is based on Debian, yes, and you still use the Debian option
records for the components that you are not interested in changing the new.
Well, we use a lighter version of Debian, and we install just what we need in order to make our products.
Yes, yes, yes.
Then, thanks a lot. Thank you.