Good morning everyone. Thank you for the organisers to invite me here today. I'm going to be discussing
our approach to custom hardware in the loopcasts to assure our defence of Linux as a feasible
real time operating system. I'm a software engineer at Colting joined in June of 2024 and in
my spare time I'm a student at the UK's Open University studying combined science technology engineering
and maths. So what we're going to go through today, I'm going to go through what it is exactly
we're trying to test what we're trying to measure, how we measure it, the results we've we've
come across, the results we've come across and how we communicate those results to our colleagues
with the trustable software framework from eclipse. So let's go to the starting point. The
starting point of what we're trying to measure is we're considering Linux as as a component in
a safety-related system running on a multi-threaded multi-core processor which this means that
the in typical safety applications you want you want to have you want you want good real-time
guarantees and those are much easier to generate on a micro-controller running something like a
single-threaded time trigger architecture than they are on a multi-core operating system with
something as complex as Linux. So to meet we have to evidence that for Linux to be feasible
it has to meet those same real-time hard constraints as a micro-controller running something
much simpler and we're working towards certifying that under IEC 615OH as a safety function of Linux.
So this is very important property that the system needs to have and we need to measure and
assure that for every iteration every build of our operating system. So how does the Linux then
offer real-time capabilities? Well since kernel 3.14 it's provided a scheduling policy
called Shed Deadline. So scheduling policies are how the kernel balances time on the CPU
between different competing threats and Shed Deadline is the only scheduling policy well one of
two scheduling policies in the kernel that provides real-time guarantees. So this means that
the requirements for the runtime and period of processes use a specified in no seconds rather than
arbitrary noise values which are giving processes priority relative to one another. So just to
say that we want a particular thread to have these real-time constraints and to use the Shed Deadline
kernel policy for that we use a Cisco called Shed Setata in our thread and when we're
spawning the thread we use Shed Setata and we specify the period the deadline and the runtime for
that thread. So the period says how often should I be on the CPU? The deadline says within each period
by what point in that period should I have finished my execution and the runtime says what is my worst
case total execution time and if we exceed that total execution time the kernel will send us a
SIGX CPU signal and kick us off the CPU. So the kernel has got to balance the demands of multiple
threads all calling Shed Setata and all providing these real-time constraints that they have.
And to balance those it performs the admissions test. The admissions test says it runs every
time you call Shed Setata and it and the kernel calculates whether it can balance this thread
with all the other threads that are already on the system and if it can't balance and the real-time
requirements of this new thread then it will refuse to admit it will refuse to Shed Setata altogether.
So this is a diagram just to show what I meant by that so we have so this is CPU time
along the x-axis we have a period each period, Rusty Workup which I'll come to later this is our
process under investigation it gets this block of green run time it gets it every period and it
gets it before the end of the deadline in every period. It doesn't matter when it happens when it
gets it's run time within the deadline as long as it happens before the deadline that stuck
is factory. So how are we going about measuring deadline Shed Setata? Well we're measuring deadline
scheduling in the context of control OS. Control OS is a Linux distribution based on the latest
upstream kernel with a user land from the free desktop SDK projects and we're measuring scheduling
performance on two hardware platforms with two different architectures we use an Intel look for AMD 64
architecture and a rock 5B for AR64 architecture these are representative, these are representative
target hardware platforms obviously our client would would bring their own hardware into play
and within within our control OS images we have a Rust program called Rusty Workup
Rusty Workup is a program to exercise the deadline scheduler so it has two threads it has a
monitor thread which starts and then it spawns a worker thread by calling Shed Setata but
the critical thing is it allows users to specify the period deadline and runtime that we want
the process to have so it lets us customize what's going to Shed Setata to be a representative
critical process. That worker thread in the meantime while it's spinning it just does
some simple numerical calculations on the CPU with the important thing is measuring how it comes
on and off the CPU and it measures when it's coming off on and off the CPU and measures it's own runtime
and it sends those measured periods and run times to an onboard safety monitor which is another
Rust application that is designed to trigger an external mitigation if the scheduling would be
faulty outside of specified parameters that might indicate a fault with the system that might
need the system to be rebooted or but we also log all the periods and run times for hours and hours
as a system runs through the system de-journal so that these can be downloaded from the system
and analyzed later but the problem that we came across was that when the Rusty Workup is measuring
its own period and runtime it's using the same clock that the kernel use to constrain period and
runtime so there's a common cause of failure here if the kernel was lying it would be lying both
to itself and to Rusty Workup and we wouldn't know the difference so we thought we need a way to
get a signal out of the system somehow during scheduling so that we can have an external clock
measure those periods and run times as well so we can determine whether the kernel was
lying to us or not so that's what this looks like so if we see this Rusty Workup run times
this is the time on the CPU for our March critical process if we customize that critical process
such that when it starts running it gets sends a signal and when it stops running it sends another
signal and it keeps doing that all the time we can measure the runtime which is the time difference
between the started signal and the completed signal but we can also get a measure of the period
which will be the time difference between one started signal here and the next started signal here
so all the while that the critical process is measuring itself and logging to a journal we can have
a piece of hardware listening in and taking second measurements of what of the scheduling performance
so so just to do this we went about designing some custom hardware we used high count for that
and we based our hardware around the Raspberry Pi Pico so this is the PCB that we use you'll notice
it has two Raspberry Pi Pico microcontrollers the one on the right is the one that's actually
listening to those signals and performing arithmetic with its own clock to determine periods and run
time and the one on the left is responsible for flashing firmware to the one on the right
at the start of every C.I. drop so that we have good assurance of where our measurements are
coming from we have traceability of the firmware that was used to generate those measurements
so as I say the one on the right this is the verifier we have we have here two U.R.T input inputs
so the signal descent over U.R. the top one is that RS232 voltage level and bottom one is at TTL
logic level that's just because the rock 5B and the knock which are our two target hardware platforms
they have different voltage levels for their serial ports and and every 10 seconds it computes the
mean maximum and minimum period and run time based on the signal that is received on the U.R. and
it sends those out because the ammo over U.S.B. CDC and then the probe we use the open source
debug probe firmware that the Raspberry Pi Foundation produces which receives binaries from the
test run at M. Flashism at the start of each test over serial wire debug protocol because it turns
out that the Raspberry Pi debug probe that you can buy they're running open source firmware and
have basically identical hardware to a Raspberry Pi peak so we thought you'll be integrated
along one PCB and that A to maintain a better T the T maintain a better T so the firmware then
the peak on the right the one that's doing the measurements this needs firmware on it
we chose to write that in rust we use the embedded how and RP 2040 how crates and
we architected that firmware such as as soon as the signal is received on on the U.R. from
from the rusty worker on the device and the test it will trigger an interrupt it will immediately
take the monotonic timestamp from the pcos clock and it will then queue that into a buffer
and then in the main loop of the program when it's not been interrupted it will deco all those time
stance it will perform the computations on them and it will produce a Yamal output like this
with our measurements down to microsecond precision so we say the runtime in the period how many
samples we had and the minimum maximum and mean the reason we chose rust compared to the alternative
which is C is that rust requires far more of the both peripheral initialization code or at least it
does when you use the RP 2040 how crates and that means we have better understanding of exactly
what the code is doing and exactly the timings and precision of our measurement so for example
in our rust code we have to declare up front the frequency we want to use on the crystal oscillator
of the Raspberry Pi peak and when we do that in code and it comes out to 125 megahertz and then that
makes us realize if we're taking the reciprocal of that then that means if we're trying to measure
a critical process which has a period on the order of magnitude of hundreds of nanoseconds which is
about the reciprocal of 1.525 megahertz you know actually that's inappropriate because it's only a few
clock cycles of the microcontroller but the benefits of rust is that you get compile time guarantees
of type memory safety and actually thanks to the type system in rust and the way the
the crates have been constructed any invalid hardware configurations will be rejected as type
errors at build time which lets you catch problems with the firmware a lot easier a lot more
easily and lastly the embedded rust tool chain is far far easier to handle than the embedded
seat or chain it doesn't go and download stuff from the internet at build time you can use the
native rust compiler and when you're building that rust compiler there's only two lines in the
configuration you need to change to specify you want to use you want to target the thumb v6 and
architecture with the Raspberry Pi peak code for cross compilation so about building that firmware
one of the things we assert as part of the quality assurance of our projects is that all the
testing tools need to be built with the same strict guarantees about reproducibility and
provenance as the operating system itself so for that purpose we use Apache build stream
which I won't go into detail about but it's just to show that our builds
all our operating system build and our firmware builds are declarative so we just
specify which one of the tool chain we want to use we specify which link are we want to use
and we simply specify the source code so this is when source code is coming from
we specify all the crates we want to use and then we just specify an install command and this
goes straight into the docket image that our test runer uses and a benefit of this is
that one of my colleagues will be speaking about later today is because we're using a standard build
system you get a software build material for this for free so that will aid your traceability
story so looking at the recap of the test architecture then we have our two picos we have
the Rusty Worker sends a signal to a verify pico which does the measurements and through a
Python test library these end up in an open search data lake to be an allies later and then
from the second path coming along the bottom the Rusty Worker measures its scheduling itself
it would also report that to the test library through the system de-journal and that
ends up in the data lake so in our data lake for the same scheduling events we have we have
two measurements one made by the process itself and one made by our external hardware so what do
our results say so we've been running this now for about nine months we have hundreds of thousands
of measurements in our data lake and according to our external measurements it's a kernel isn't
lying we're sure that our self measured our self-reported measurements are accurate on the
look where we can say their accurate within plus or minus a hundred microseconds on the Rock 5B
we can say their accurate within plus or minus ten microseconds and that is a factor of
the u-art on the Rock 5B being a peripheral on the same chip as the SOC where is on our
look are u-art is a PCIe peripheral elsewhere on the motherboard so we've used Jupiter notebooks each
each week to allies previous weeks tests and check our performance and if you this is a graph
on the right just to show it's not it's come across all right on the projector but the distribution
in orange is the self measured distribution for the runtime the main runtime of a critical process
and the distribution in bloom is from our external hardware you can see they they line up very
very neatly and some other interesting results we've got from this is that I said we found that
the external measurements are more accurate on the Rock 5B than the look because we understand the
hardware differences between where the u-art lies in the system how close it is to the CPE
but we also saw that during the development of of control ls during that period my colleague
Mateo introduced the preemptarty kernel option which means that kernel threads are allowed to
be interrupted as they run so this is supposed to give better real time performance and we actually
found the car external measurements got less accurate when we enable preemptarty and we understand
the reason why for that is because cave threads are involved in writing to serial ports and if
they're they're getting interrupted then the time taken for a signal to get out of the system is no
longer deterministic so finally I'm going to come to how we communicate these results to
stakeholders and we use the eclipse trustable software framework to do this on control so
the idea of the adjustable software framework in brief is that it gives you some very high
level assertions about what's required for a quality software project and then it gives you
tools to make measurements you make of your own system against those assertions so an assertion
is get each assertion is given a score between zero and one it's a decimal number between zero
and one about how confident we are that score is true and we aggregate the scores for low
level evidence items to get an overall trustable score for the project so if we want to
for each CI run that's using this external hardware and putting data in open search we want to
make assertions for that exact commit of control because we're building a commit with
testing a commit and then we're generating a report for a commit we can use Python code which
will automatically check over search compute the results through the same computations we're doing
in our Jupyter notebooks and provide a confidence score between zero and one for example here saying
that the absolute error according to the data we have in our data like for this particular
share of control on look was we're asserting that that delta is less than 100 microseconds and you
can see it's got a score of one because the validators checked the data and produced that score
and we get a HTML report visible to state holders and engineers for each iteration so what does our
argumentation look like then so at the very top are two of those very high level objectives
I said the trustable software framework dictates for particular software projects on the left hand
side we have argumentation to say that all our tests or the tools we use for our tests are reproducible
and we're reproducible because we're using embedded rust tool chain in build stream so we have
intermediate argumentation which fundamentally we're saying we're looking at a CI job that tests
the reproducibility of the image and validators looking at whether that passed or failed
and it's producing the score that flows through intermediate argumentation along with other
items to provide a score for this high level assertion that the build and test environments are
constructed from their resources and then on the right hand side the bit that we're at the part of
the graph that's concerned with the part of the graph that's concerned with the quality of the software
we look at our open search data we have intermediate argumentation we feed that into an
STPA analysis and risk analysis of the project to provide the final goal that we've identified
a risk and that risk was the kernel might be lying to us and we've provided evidence to show
actually we're happy with that risk because we've got evidence that the kernel isn't
so in conclusion I'd say external measurements have provided great confidence in our analysis of
the next deadline schedule and it's meant we have a deeper understanding of the kernel and a
deeper understanding of the hardware we're using to make those measurements which means it's
our risk analysis will therefore be more informed and you can see that all the different pieces of
open source software and hardware that we've orchestrated together so that we can do complex hardware
in the loop tests to prove that that an open source kernel like Linux on a multi-core system is
suitable for a hard real-time applications with hard real-time constraints and industry
and lastly to say that the trustable software framework point of that is that it allows you to
communicate those results to the state holders and also to engineers as engineers and we're looking
at the bottom of the graph well looking at those statements that say the difference was less than
100 microseconds or whatever but state holders they're looking at the top of the graph in the
searching that yes we've done a proper analysis of the risks in our product and so thank you all
for your attention thank you for the organizers that's all I have to say today if there's any questions
please
John on the half of code that we'd like to thank the customer nation last year we had indicated
that we were contributing and opening the project up we did unfortunately there's some infrastructure
issues that we had to work through over the last 12 months that we can we can claim unfortunately
it happened after the slides that submitted but on the right the right hand side you don't know
if there's foundation that or you will find the trustable project fully there the full
break was an out-up the changes for the last year is now available in the board the commuters
coming in helping us advance the project
okay that's made any questions for me or no one question yes but so you were using
separate crops so you can confirm that they're not affecting each other that they're
separately being measured so on yes do you have like any initial stage where you're checking to see
whether or not there's significant problems between them naturally and better of you than taking
that into account in your statistics we're not going to imagine it would improve the results
of our progress and there's some of this is being done well so the question was about whether
we can whether we can measure the drift between those clocks the clock of the pico and the
clock of Linux before to improve the accuracy of the test I'd say the answer to that is that
we've we've not found it necessary such I think this this external measurement is only there
as a sanity check but performance these differences of on the order of magnitude of tens of
microseconds are acceptable enough that we can attribute it to the hardware to the UART
to the actual time well I mean we're getting quite close with a with a board rate of
1,500 we're getting pretty close to actually the time it takes to the bits to get along the
wire but it's something we can I believe we have got an issue open to be for example connecting
these to a more accurate and precise as ever scum to be plotting to be calibrating our external
measurements is such thank you very much thank you