Hello everyone, hello everyone, my name is Pavovietlorek, I work with Colabra and I joined
Karnasji development team in July 2023.
Today, I would like to share with you what we've been up to, what's still in the
works and how you, Karnas developers, Karnas maintainers, automation or quality assurance
engineers and everyone who's involved or just interested in similar efforts could benefit from it.
I will start with just a short introduction to what Karnasji actually is as a project.
Next, I will describe Karnas status of things in detail and after that, I will talk
about potential next steps that we could take.
I will also show you how you could try everything out yourself and finally, I will
show you a few closing thoughts.
So let's start by explaining what the Karnasji project really is and if we go way back
to its early days in 2014, it started as an independent initiative by ARM-SOC maintainers.
It later became an automated system for Karnas builds and bots and testing them on embedded
ARM platforms through collaboration with Linnaro developers.
Its dashboard is something that I believe many of you might already be familiar with.
Throughout the years, requirements for the system grew and in many cases to the point which
the classic or now legacy architecture could no longer support.
This resulted in time-doubt queries, high infrastructure costs and also high maintenance
costs.
It eventually led to the idea of rethinking the design for a new Karnasji system and
its instance hosted on Karnasji.org, which I will later refer to as a service.
Also, putting a new system in place is why the former one is now being referred to as
legacy.
But both system and service, however, are not the only components of the Karnasji project.
Together with growing requirements, also the whole ecosystem expanded from improving test
quality, which is an effort that only makes use of Karnasji systems, but its scope
is far different from just the test execution automation.
Through, for example, preparing GitLab CI pipeline templates, which is for Karnasji development
and testing, which is a project backed by Karnasji developers, but currently its separate
effort.
Also, there is another, completely, maybe not completely, but relatively separate effort
of collecting all different testing results from various systems, not only Karnasji,
specifically, and delivering them to any part interested in receiving or processing them
further, which is KCI DB.
But back to, or maybe, if one more thing, if you'd like to learn more about these efforts,
go ahead and have a look at the linked materials, the one in the middle GitLab CI pipeline
definitions is a pretty fresh one, it has been updated just last week, and now back to the
growing ecosystem.
This was actually the main reason behind the new system.
It had to be designed with the extensibility in mind in order to easily integrate with
all of those various services.
But what's important to remember from this very short introduction to Karnasji project
is that it has expanded and evolved into an umbrella for various efforts related to Linux
kernel testing, system and service are just the components of the whole stock, and the
service scaling and maintenance costs led to redesigning the whole system.
Now, once we have this out of the way, let's review the Karnas State's current status of things
and have a better perspective at various testing efforts under Karnasji project umbrella.
Don't worry, we'll dissect this whole testing landscape diagram in a second, and starting from
top left, we've got the inputs, so the Git trees, that's Karnasji system monitors.
Below that, we've got the component that is responsible for storing the test definitions
for also building all of those artifacts that will be later used and dispatching all different
tasks to other components. And about this patching tasks, we've got the labs component
which is responsible for actual execution on physical hardware of the tests that should be executed
by Karnasji. On the bottom right, we've got also other systems to relate it to Karnasji testing,
but not specifically part of Karnasji umbrella project, like in the zero day Red Hat's cookie or
saysbot, which also feed their results to KCIDB on in the middle right, which as I mentioned before,
collect all of those testing results. And finally, we've got also a new web dashboard for
presenting results to the end users. And let's start going into details from this web dashboard.
It probably might be the first contact for developers who just start to interact with new
Karnasji system, and it attempts at providing only relevant information instead of being a
data overload for new cameras. It's available at dashboards.karnasji.org.
It currently is under heavy development, so if there is something you're missing or something you
see improved, you'd like to see improved, please let us know. And we'll get back to it in a moment.
But where does the data for web dashboard come from? Or maybe before we go to that, let's
talk about the regression tracking. Because web dashboard is important, but what we can get from it,
and the potential information that should be easily accessible through it would be the data on
regression found in all of those systems. There were several attempts at improving the
states of things and several POCs, the proof of concepts developed recently. Currently, we're
moving the most promising ones to present their data back in the web dashboard, so that it's
more easily accessible. And now let's go back to where these components take the data from.
And I mentioned it before, KCIDB is a results collector from various sources, not only Karnasji
systems, and it's a single point of truth for results delivery and also reporting or finding
regressions. It's data can be accessed through web dashboard I showed you in previous slides,
but it also exposes its contents as a graphana dashboard. But where do these all of these test results
come from? For that, we have also a component for labs for actual test execution on physical hardware,
but we do not limit ourselves only to physical hardware virtual devices are also present in those labs.
Labs also cover the computer resources for build farms, and although currently most of the hardware
labs are based on lava, so linear automated validation architecture, the new system does not limit
ourselves just to lava labs. It's just easiest to get started and connect more new hardware to
Karnasji. But again, the new system removed this limitation which was one of the drawbacks of the
legacy one. And those labs can execute tests, but who dispatches them? Instead of the legacy
hard to scale monolithic application, the new system for this purpose provides just thin abstraction
layers for specific tasks. And these tasks are for example three monitoring to watch for changes
in relevant git trees, but also processing API events and scheduling predefined tasks
start in maestro component as well as reporting back the results collected from labs and
submit them to for example KCIDB or to the developer who created a change processed by maestro.
As you can see, there is also an email report component on the very end of the pipeline,
but that's something that caused some issues in the past mainly with not entirely reliable
notifications, so that's something that's still under development and we've got the email reports
currently in the Karnasji main list in order not to flood anyone with false positives.
And now that we've covered the automated execution, let's go to the on demand one.
For that, we've got the new KCI dev tool, which is standalone utility to interact with Karnasji.
It supports custom submissions, even for arbitrary commits, so not only monitoring the git
karnas trees from maintainers, and knowing that not everyone might be a huge fan of
web interfaces, it also allows you to retrieve the results in a machine-readable format for further processing.
It will also be providing the ability to do the automated by sections from found
regressions, and this feature is currently being finalized, but still under development.
If you'd like to learn more about it, go to the kci.dev.
The key points I'd like to highlight once again for this new system, and its integrations are
the extensibility that was the main idea behind the system redesign, which also resulted
in a much improved scaling of the whole system, and something that I would like stress once more.
The new system is no longer bound to lava hardware laboratories, and that opens up the
whole new way of interacting with physical devices.
As for the next steps that we could take this system from, if you'd like to have a closer look
at how things currently are, the easiest way to do that would be to just access the
managed instances, so either production service, or if you're more interesting in the most recent
changes our staging instance of the new system, you could also try the guided way of creating
your own local instance of the new currency system, and there is also a semi-automatic way of deploying
many, but not all of the components in this new system, stored in our currency i-dash deploy
repository under local installs directory. And if you do, please let us know, I mean the
accuracy i project, what you find that might still be missing from these components, but you'd
like to see in the development workflow. Maybe there's something that might have been
indicated, and there is already a tested component that would suit better the testing workflow,
or maybe there's some new hardware that you'd like to see being tested by the currency i,
and connected as a hardware testing laboratory to the whole pipeline.
If you have any comments on any of those topics, you can either send an email on the mailing list,
or let us know on IRC, channel or matrix channel, or simply hope to our Discord server.
And all of the slides have been already uploaded to the first-time event page,
so all the links are also available there. And now just to summarize it all,
I wanted to share with you today that the new system is steadily going through
a stabilizing phase, which also comes with some setting the legacy one.
If there is any feature that you'll depend on from that setup and has not been migrated yet to
the new system, please do let us know. But this whole new setup was focused on delivering reliable
test results and the only relevant reports. So the main idea was to prevent
maintainers burnout from increasing and hopefully helping solving it. But solving the CI needs
for Linux kernel community is not really just a technical challenge. In big part, it's also a
community challenge, and that is why it's crucial to continuously discuss what can be further
improved. And that is why I hope I will hear from you about your experience with the system.
And with that, thanks for your attention. If there are any questions, I will be happy to answer them.
The question was about the test database and how they can be described to be executed.
So depending on your test needs, the go-to answer would be try to write K unit,
look if there is an LTP that already supports your use case.
As for the tests that are currently executed, they are, like I said, K units, LTPs,
simple boot tests, and also as the most of the physical hardware labs use lava as their
underlying system, they are just lava test job definitions.
Many test job definitions are available in the Linux test dash definitions repository.
I can share the link later if you would be interested in having a closer look.
So the question is about scaling and how with expanded use cases, we grow all of the components.
Yes. All right. So yes, the scaling as always is not an easy task. That's why the new system
started really small with selected feature set and a small number of use cases being supported.
There are several ongoing efforts that I'm at improving. For example, KCI DB performance,
so that the queries that you'd run on this component would not simply time out, but return
relevant results to you about the growing all of the systems.
Yes, one have to be cautious about adding new use cases to be supported and adding more hardware,
more test suites, so far the KCI as a project has hit a few limits, but it's not something
that the new setup does not support. It's more of the throwing more compute resources at the
system problem, kind of problem.
All right. So the question was about the data coming from different instances like staging and production.
And data coming from staging instance is not something that goes to KCI DB, which is later
accessed by other post processing components. Staging data is throw away. If we want to make sure
that it's everything's fine, we compare it with previous results and if it matches, that means
new release on staging is good to go and the changes can be migrated to production, but overall
the data coming from staging is assumed that might be faulty. So before the new deployments,
we compare that the key indicators are fine. We've got several specific trees,
monitored on the staging instance that even have branches with non issues, so that we can also
test if the new deployment of the system still catches those regressions.