Hello everyone and thank you for joining this presentation. My name is Stefan. I'm currently
working on analog devices and I'll present today an automated way of testing custom
and using a multi-power hardware platform. The presentation is focused on system-level testing
mostly showing the importance of it in integration between hardware and software and to ensure
that those components are working together, there is required to robust testing infrastructure.
So some key attributes like a raw-dair automation integration, rigorous testing, are some
of the must-have for having obtaining that quality from the middle and of course a reliable
testing infrastructure not only optimize and decrease the testing grant time but also reduce
the probability to find bugs in production.
So our custom Linux distribution is called analog devices, hyper Linux. It's a free open source
Linux distribution and comes with a lot of pre-kept files like different applications,
Linux drivers, libraries and a lot of examples.
Hyper-supports, multiple hardware platforms, AMD, Intel FPGAs and also Raspberry Pi and recently
NXP. It also incorporates Linux package repository which includes, which is included by default
in the distribution and the image. So it makes all the update process much easier.
For more details about the hyper, the last time from there, my colleague Andrea from there
will have a talk on 6 p.m. regarding how we optimize the whole release and it will
be in the same thread here on continuous testing.
The flow for individual components is using the classical tools like HR pipeline Jenkins,
GitHub actions and so on. The CI will build a software components across multiple
operating systems and the resulting binaries that can be Linux packages with those installers
or even other kind of binaries or files can be saved directly in GitHub's repositories
in that package repository for the bin or internal servers.
This approach not only improves efficiency, but also ensures traceability, version control
and artifacts acceptability so they can be accessed from multiple ways from multiple
other machines. And of course, that somehow required to scale the system to more and more
repositories. The testing process in hardware closely mirrors the one in software.
The workflow begins by writing specific boot files in the hardware. Most of the hardware
are supporting SD card and I will present in few slides what solution will help for that.
The test are executed in parallel on multiple platforms reducing in this way the overall testing
time and increasing of course efficiency. Most of the popular testing frameworks are focused on
application level or focused just on a specific hardware platform. So that's the main reason
for which we have created our own testing framework called test harness, hardware test harness.
And this hardware test harness is designed to unify testing across a wide range of platforms
being consistent overall. So as you can see builds are triggered by pull requests, pushes,
and GitHub we still add the manual trigger and the crown in Jenkins there is a main server
and that main server will, that JFrog server will just keep the artifact. It will
attach a buffer there. And once the build passes the result binaries will be sent to the main Jenkins
server or the main Jenkins job and from there that they will be distributed and executed
to multiple agents each agent has a hardware platform usually different hardware platforms attached.
Most of the tests are written in Python and they have tags to tell them on which hardware
there are compatibility and where they can be run. So as I said the intermediate server
we are using artifact from JFrog from there. Main reason is to have it as a kind of buffer because
we are using multiple repositories. So to eliminate the risk condition of having pull requests or
pushes in the same time. Another one is to be able to have multiple instances of the test harness
also that artifact to the server will be like a buffer and every instance will just download
file from there for testing and also we use it to organize the files by timestamps, by versions
and to increase the scalability of the whole framework. Jenkins agents it's also to be physical
machines and it wears why. It's harder to connect it to them so they cannot be in cloud or VMs or
anything else. So they are handling all the connections like USB, Wart, SSH, sending scripts to
the hardware. Overriding the files from the SD card if there are SD cards used and so on.
And also the build server in our case needs to be a physical machine and there's not enough to
have a cloud one or VM. One of the reason being because there are implied a lot of computational
resources. And also for example special licenses for different tools like silings, Intel,
Quartus, Matlab and so on. So it's pretty hard to have a VM with all those tools pre installed
and licensed. So a good framework should be adaptable to different hardware types and testing scenarios
and modular enough to accommodate all the changes. By changes I'm referring to number of repositories
that can be added to be tested in this hardware setup. Number of devices and other test,
hardware board or number of automated tests if they will be modified or expanded.
Many time there is a necessary a test manager to queue test also to distribute to execute and to
collect results. But I'll go first through some implementation details and afterwards come back to
this test manager. Some of the tools that we used main one is Jenkins. There are some advantages
so it can be hosted individually with not abandon any cloud or any other sub tools or dependencies.
It also integrates easily with GitHub with Artifactory or other additional tools
and there is a very big online community that help on issues. Some other
features from Jenkins like Jenkins share library, dynamic script language, resource locking,
proof to be very useful in our use case. The next one is an in-house developed tool. It's called
Nebula. This is actually a collection of Python scripts that manage hardware connections as I said
where the internet j tag USB and so on. By using PDUs for distribution unit from there and also
USB is the card MOOCs, the other one from here. We managed to have the full control remotely
of all the hardware. Those two can be both from the internet are available.
And we are using them to ensure that the hardware does not remain hang in a state that it's not
usable. More than that both tools both set up from their PDU and SD card MOOCs are controlled
through Python so they got integrated into Nebula. So now we have a set of commands used to
do everything with hardware. Netbox is another tool probably some of you knows about it.
So this was initially designed for modeling and documenting network cracks. That's why those
pictures from network cracks back. But also it fits better in our use case because in the end we
managed to put our hardware in those network cracks. So that this was necessary once we scaled up and
we started to add more and more hardware to not just be laid around the desks and shelves and think
like that. This was great. More than that we are using the netbox to get the Nebula config files
for each piece of hardware from there we have all the information regarding connections.
Device under test what tags or attributes has that hardware and think like that.
And all the information from netbox is updated only whenever we need a new setup device under test
to be added to be modified or re-arranged between them. So it doesn't imply manual
modifications.
As I said, Jenkins shared libraries it's a very good way to centralize the groovy script in
one single place. It can be a repo. We can have tests on that repo.
In our case it contains the definition for common functions and pipeline steps that can
be shared across multiple Jenkins server or agents. We use it to update the agents to update
the tools to manage the hardware and so on. And this approach ensures an efficient process of updating
and maintaining the same functionality across all the machines from the test hardness.
By mixing the diagrams of continuous CI with continuous testing will obtain something like this.
Behind it there are about 100 pipelines, CI pipelines and over 10 build servers.
We are using Azure, GitHub actions, Jenkins, Docker and so on. And for most of the repositories
besides the build, we also have the hardware test results from the board farm back to the GitHub.
Some of their components like libraries or other things that can be tested, it didn't
individually directly in the hardware on different platforms, have the results sent back to the GitHub.
Other of them that depends or have multiple dependencies will save the
output of the build on internal servers. Even if that internal server is the package manager
or JFrogart factory just because they need to be combined with other components to be tested.
So Linux packages are created automatically for multiple distribution, multiple
Linux distribution. And we have that package where we have two environments testing and
production easily to switch from one to another one. Let's now have the results are looking.
So first of all this is the main Jenkins output, the standard one was not enough of course
with switch to something a bit better, which was blue ocean. Blue ocean is just a plugin in
Jenkins to see the results on different pipelines. This helped us to identify exactly the stages
and hardware where the problem was fine but still not enough. Then we switch to J unit with some
graphs and in the end we switch to log-stash for processing results, elastic search for storing
them in database, keep on up for generating all the graphs and so on. The tax value, none of those
was not good enough because developers were needed to go into Jenkins or any database of results
and look for their polyquest results to know if it's okay to merge or not.
So even if the results are shown in graphs or dashboard or so on. So the final step was somehow
to close the loop. This was one of the most important features from the test harness and the main
challenge here was to ensure that the private data is not shown into the public repositories.
I don't know any kind of Jenkins laying site, internal site PE and so on.
And down so here was GIST, posting GIST and besides the build status having also the hardware
test results and with this system in place we were finally able to enable the required CI to pass
in the GitHub so that we ensure the branches stability. As I said, the recovery hardware setups
with that PDO and SD card MOOCs because it was somehow a common issue when the boot files produced
by the CI were not good enough. In this case, hardware setups were hanging, they require manually
intervention. But now with this system in place the frame of detects if the board is not booting.
Even if that board is FPGA, it is very pire or other type. And we have a kind of golden file
system which is a reliable baseline of boot files that are over it and every time a boot is passing.
So we are every time over writing those files. However, there are still some rare scenarios
where the hardware is broken or fully disconnected and requires manual intervention. But we
use the manual intervention a lot with this. So not that I have been through all the main tools
that we used, let's say the overview diagram. On the left side you can see the TRIGONIC mechanism,
Jenkins files, Jenkins server, the Jenkins server, main Jenkins server manages the testing
request from multiple GitHub repositories. We have multiple test harness instances, as I said,
and all the results are collected together with elastic search, Kibana, and most of them sent
back to GitHub. Test harness supports now tests that are in different languages, a Python
C++ MATLAB, and so on. Harder boards are locked only when the tests are running.
Otherwise they remain accessible for remote people, they can connect to harder boards and do the
bug or development. And also the results are well structured and presented clearly as you saw
with gifts back to the GitHub. And now let's see how it looks in real world.
This was the prototype a few years ago, early beginning stages while the boards connected
between them laying around on a desk there. At that point we were working on adding support
for multiple platforms, ARM 64, everything that's related to the SPGA from Intel,
and so on. And also we tried at some point to use AirPies as Jenkins agents.
And this is how it looks now. We have two rocks filled with hardware.
They are almost fully controlled remotely. It supports the Kibana Linux, the distribution
that I talked in the beginning. And we are working on adding the octosupport.
Also on multiple hardware. And many a little bit better distribution of the hardware
across multiple physical locations. So to be able to have one single set up or just a few set ups
in one location connected to the main Jenkins servers or the results to be collected together and sent
back to the GitHub. In conclusion, we have managed to implement a complex testing framework
that can be triggered from multiple GitHub reports, but still keeping the Chrome and manual
triggering from Jenkins. Harder set ups remain accessible from remote connections,
allowing colleagues to do development or debugging on them. It supports multiple
platforms and can run tests in different languages. The resources got optimized by using Jenkins
agents inside Docker containers. So this means that on every physical machine we can connect
multiple hardware like three four set ups. There is a robust recovery mechanism in place and
manual integration got reduced a lot. And also the test results are well structured and bind
back to the GitHub, ensuring in this way that bugs are found in a fairly as possible. So that's
how we stream either software and hardware integration and the testing process. Thank you.
Sure. For pushing buttons in hardware. Yeah, there are multiple solutions. Yeah. So the
question was if we have a solution for pushing buttons, yes, there are multiple solutions. You can connect
with wires there and use Raspberry Pi is the case, but the video that I show in previous slides
power distribution unit is able to do the power, the hardware power reboot or power off power on.
If you need only for power, it's a K to use that power distribution unit. If you need really
to push the buttons and you cannot do it from software, you can use Raspberry Pi or MPK or
some other hardware and connect wires on the buttons. Yeah. Yeah. No. It's not required in our use case.
Yeah. I will repeat the question. So the question was if there is any solution to flash the
images on disks. Yeah. So we have that SD card MOOCs if you are referring to SD cards or anything else.
We have platforms that support SD card and that SD card MOOCs works like MOOCs between the real SD card
than the USB. So we can switch it to be seen by the host by laptop or machine as a card or something
attached to the host or we can switch back to using as a card attached to the hardware.
And from where we switch between two modes combined with power distribution unit, we know exactly
when to reboot the board, the hardware reboot and we can do that. For other boards that do not have SD card
and are working just by pressing down by programming them, we have also solutions by J-Tag so
that it needs to be a host, laptop or any kind of physical machine where the hardware is connected
and through J-Tag or other solutions we push files like micro blaze etaps, for example,
the octo, the same. And in that case, in the worst case where there is no SD card,
is that they have usually a manual switch and needs to be put on programming mode or booting mode.
Let's say that. And that manual switch is handled with wires, just wires sold there and then it is
a very wide pair of ok and control. So, just one small question about the SD wire module. Do you
plug it directly to the depth point? Yes. Or have you ever tried using a SD card extender? Because I have
a bunch of them, they work with the one I plug them directly, but I need the SD card extender to reach
into some of our modules and adjust the support. I repeat the question. So, the question is about
the SD card MOOCs from this picture from this slide, if we plug that into the directly into the
boards, yes, the answer is yes. So, as you can see here, it has the shape of SD card and actually
here it is the real SD card and USB from the host. Some problems that can be here are related to
the attention level. So, the SD card either from the board may have a different speed or tension
level compared with this small device. In that case, you just need to go on the hardware and do
some tricks they are to switch to other speed, other level. Yeah, the voltage and yeah.
In most of the cases, this can be done directly from the boot files device reasoned, yes.
Is this an option? How is it custom design? No, this can be found on the internet.
Yeah. Actually, there are multiple models, yes. No, it's not a neighborhood that some of you
may know, it's something developed internally, but it's a documentation. It's our documentation.
What is the name of the board? Of the board, it's SD card USB, USB SD card mooks.
Sorry, that time over. Thank you.