Okay, so, hi, today we will talk about the ultimatum cloud that we are filming
by this and we are about framework, I am Maché, Mr. Mitchell, I am from, okay, so, I am
Maché, from 3M depth and generic measure currently, what we are doing, we are doing
various stuff for the scope of this presentation, what might be important is that we are developing
boot firmer mainly for X86, so we can simplify by saying that we are developing the UFI BIOS,
but open source one, so it's the BIOS you know from your computer, but it might be more open
than the traditional one, and we need to validate that to make sure it works correctly
on the certain machines, so the agenda, we talk about the history of our framework, about
what kind of hardware is supported there, how to interface, how we are interfacing with
the hardware, and I will show you some examples scenarios of test we are running.
So the history, we have been using a similar approach since around 2018, when we have
been evaluating PC engines, core boot releases on a monthly basis, so we needed some kind
of regression test suite that we would run, at least each month before release to make
sure that everything is working as intended, using that approach is executed over 50,000
tests, and publicly released over 150 binaries of open source firmer for I believe six or seven
different platforms, and it was mostly an internal project, that the firmer was open
so the test was not, we have published it like two years ago, small part of it earlier,
what are the use cases of this dashrar open source firmer validation framework, of course mainly
validation of firmer in general, not only open source, you can imagine or we even had one
use case where we did tests on IMI firmer as well, but mainly we use it for testing dashrar
firmer releases that we released for different hardware platforms, for test driven backfixing
for regression testing, so after introducing new features we can check if we haven't
break, other features before the major release we can run all of the tests we have to
make sure it works as intended, it mainly works currently with dashrar with UFI payload,
but we do have other firmers and cases as well, it's mostly written in robot framework
as the main test environment, we do have some Python as well for level of keywalls and some
shell scripts mostly as some rappers or hyper, are you familiar with robot framework, have
you been using it, okay, so it's like a generic open source automation framework, it's built
on top of Python, so it is a Python underneath, you can use Python to implement libraries
for robot framework as well, it's from what I know, it's used widely with web application
testing and many more, but it's not so widely used in the lower level, maybe validation,
I know one more project, which is kind of level, which is open BMC where robot framework
is also used and they are basically my kind link, but it's links distribution for the
BMC on servers, I can say that robot framework has excellent documentation and great
community, it's a really fun project, so what platforms are we talking about here, we have
a wide variety of hardware platforms, we develop a firmer floor, so a lot of stuff as this
one, we have some network appliances, we have some desk stops, so the MSI one, we bought
a two-core boot, like two or three years ago, which is quite modern, so we have different
kind of hardware and we developing a BIOS for them, so we need some means of testing those
releases on those different hardware units, which poses different challenges, so we need to
have some small lab that we put those hardware boxes in and we need some interfaces to remotely
run some tests to let the developers work on them, we need at least some power control,
we need some testing interface, we need some GPA control sometimes, we will walk through those
in details right now, so power control is perhaps the most important one, most of the time we
use one of these two, we have like a custom board, with this relay, we can switch power on and off
and for some other boards, so it's not possible, we just have some Wi-Fi plug, equivalently,
which we have a lot of firmer, which works pretty nice, and we can use API to the power supply unit,
and at the end of the board, we've developed a couple years ago, with open source hardware
settified, you can find a lot of the schematics if you are interested in, and we develop some
server application on top of that, which allows also to use HTTP API to control the power,
GPAios, and so on on the device on the test. Next, if you already can control the power supply
unit, we also need to control the test flow itself, we need to send some commands to the device
and to receive some responses, so for that we mostly prefer to use serial ports whenever possible,
so we, in fact, you'll turn it, so we connect the serial port from the device to the other board,
our RTE, and then we set to net service, it's like a service, which basically exposes this
serial interface in local network, and we can use it from different machines via turnet,
this is the primary one, there are also others like SS site, but SS site is not really enough
for firmware validation, but we can use it in a lesson, in some cases, when we are doing some
OS level testing, we have also, in terms of input, we have also used the keyboard simulation,
in some cases we can only read from serial, but we need to write from USB keyboard, because that is
like one line only for you up, in some words, we need to control the PIOs, most important
ones are power, and reset the buttons, because obviously we need to be able to have that
level of control over the board mounted in lab, in some cases we need some more, like power
let status, so we can read if the board is in fact already booted or not, or at least
powered on, there are also many custom GPS on different boards needed as well, sometimes we need
to, I don't know, reset signals after we flash firmware, and we need to build different kinds of
circuits to support that, and all of those are connected to our RTE control board, and then we can
use REST API to control the device, maybe the most important is the firmware flashing, we need to
provide external interface for developers, whenever they are developing firmware, more often than
not they will break the machine, and they need to externally flash that motherboard, so depending
on the board we have either some flashing headers, or we need to come up with some different
creative ideas, such as clips to clip onto the SPI flash chip to flash the bias externally,
that's also connected to this RTE control board, and we can flash all of that remotely,
to support all of those hardware interfaces, we have the CLI tool, which allows developers to
easily run the typical scenarios, for assets management, we are using sniped currently, so
we can check out the device, so to indicate that I'm working on this one, so no other people will
be using it at the time, then I can reach some backup firmware right my own firmware, power on the
supply unit, power button press, increase the pressing and so on, and grab serial output from
the device on the test, so I can execute the hold of a lot of the flow on the machine remotely in
the lab. The same set of libraries we have for the CLI tool, we are using as a robot framework
libraries in test environment, previously it was a problem that we had like different set of code paths
and some scenarios worked well with test environment, some not in CLI tool, and the other way
around, so that's now integrated to use the same set of libraries, and now I would like to
describe what kind of tests we need to perform on our firmware, so typically it consists of
like, and time to set up, switch some option, put in some OS and check if this option works as it
should work right, so that's end-to-end functional testing on the actual hardware. So in this
example scenario we put the platform, so we need to send some commands to our control devices
to power on the board, then we start reading serial until we see the very first strings,
data as which key needs to be pressed to enter this setup, so we enter setup, then we get this
menu of our serial, we need to parse it and compute calculate how many key strokes we need to
for instance how many press down we need to execute to reach the position we need,
so we enter for the in this case the shower system features, then we are in the next menu,
here we just need to press enter to enter the shower security options, and here in this
example we want to enable an SMM BIOS write protection, so we again need to
read out this setup, this menu layout over serial, parse it and calculate that we need to
one out of them and enter to select this option, and then we need to also to press F9 to size,
then agree to save restart, and that's it in terms of setting this option state,
here's how this example look in robot framework code, so we have all of these actions,
I just described but in robot framework keyboard, so we have the keyboard implemented that
do that kind of actions like going to certain menu, then sub menu, then enable
some option, so I think it's taken through in this case, then we have keyboard to save changes
and reset, then to boot from a selected operating system, and finally we execute some command in
this case flash from command in the OS to see if we have the desired outcome,
but some that we can parse or file the certain case case, and here we have some random amount,
example how to run such test, we can run this in QMU as well, so we have some
wrapper script that can allow us to spin up the QMU with the server firmware, so on the left side
you can see the execution from robot, and on the right side you can see the execution from QMU,
so we can in lifetime we can expect how the robot framework test phase pass or file,
on the right side you can inspect how the machine is being powered on, part of the menus are being
changed and so on, some links to the community if you'd like to reach out and discuss
other contact information and that's it for the main part.