Okay, so good morning, my name is Pavel Pisa, and I have there a talk about my experience
with real time and control from the Nux environment, so I will start with some introduction,
and then I will follow about the history of fully-preemptive patches, and the events
of prevails attempts to implement some real-time capabilities for the Linux operating systems,
and then I will follow to control our network to the communication which is used in most
of the cars and even industry and so on.
To introduce myself, I have first met with the Unix operating system in 1988, the development
workshop of the Academy of Sciences, where was a special system for the Mikrocontroller development.
It was based on 68 cases, and has one half of the megabyte of RAM, and it has something
like five megahart megabytes of hard drive, but I fall in the law, which is this system and
with the Unix and the concept, so then when I have started the development of labor-interference
and hardware for the project of my father, and then even for our own company, then I have
interest in the Unix system as well, and one day my schoolmates at the university came with
the set of the discets, and it was the first version of the Linux system, and we have tried
to use it even for some development and increment way switch to the Unix environment, and from
1996 we have done all the development of the embedded systems and so on in the Linux environment.
I have started the university in 1990, there I have with my colleague, Peter Porazell,
and we have started to do even hardware development and so on, and okay in 1993 I have met with
the Unix and because I have interest to control of instruments through the Unix environment,
I have started to look for the possibilities to use it for RS485 communication, which we use in our set-ups,
and I have found the control area network and tried to implement some drivers for it, and so on, and
usually even that our customer swans and the staff on Windows, we have first developed our drivers on the Linux.
So this is on the right, our first communication card, which was for our open micron protocol,
which is open source project from the days, and still it is available for Linux Windows,
systemless environment, unfortunately, that driver is in such a day that it has never propagated
to the Linux mainline, but we use it, and when we have built that communication interface,
we have added on the card, even the control area network, controller, and I have started to work
on the Kenbass support in end of 1990s, and then we have also our project at the university,
and we have implemented the Ken drivers and can open a subsystem for Linux, which is been
in use till 2020 and so on, by more companies, but in parallel, there appeared a socket
Ken solution, which won the community world, so I have helped even to that project.
Even that our project has not continued on the for the Linux, still our drivers are used in
another real-time system, for example, one week ago, we have mainline our Ken
and Ken every subsystem to the RTMS, so it still is alive, how many of you
have some idea what it means to work with real-time system, and what is important for real-time
system is not that throughput, but it is important to have some guarantees that you fit
into the time, which is given for a given computation, so for example, if you are controlling
the steering of the car, then if your computation is exactly precise, how to
turn through the crossing the best way, which minimum effort and minimum effort is and so on,
but the computation takes longer than is the time to react, and you hit the pedestrian or you hit
the building, then it is great that you have very precise algorithm based on many layers of
neural networks and so on, but it is of no sense, and it is saying for the communication, because
if you find that there is a pedestrian in the street and your communication latency to the
actuators is too long, then again you hit him or her and it is a problem, so for hard real-time
systems, you have this fixed time, and for hard real-time, you basically should guarantee that
it never happens, but it is too high guarantee because you never can declare that your system
cannot be misbehaved, but there are such attempts and you need to follow some rules according
to the criticality of the system, that the probability of the failure is once, 1,000 years, or 10,000
years, or something like that, if it is, for example, a blow-up of the nuclear power plant, then
basically you should work with those probabilities, on the other hand, if you take care about real-time
for some multimeter stuff and video and so on, then it is called a soft real-time, and in this case,
if you lose the frame from time to time, then basically it is making a quality of service
or little wars, maybe somebody will not want to pay the full price for this, but it is not
so big disaster, so we devise those two categories, those hard real-time system are used for the
control systems, for the avionic, automotive, industrial, robotic and so on, so it is one of
very interesting areas of real-time for multimedia and so on, and if we start with the attempts to work
with real-time on the Linux system, then there has been more attempts to run real-time system
in parallel to the Linux kernel, in some smaller real-time executives, which can provide
remaining time to the full-featured Linux system, because usually, have relatively simple stuff which
runs in some close-loop controls, which you need to run at that, with that guarantees, timing
guarantees, but then you want to have the comfort on full-featured system for the graphic and
for the communication and so on, so this is reason why, for example, even on Windows, there are
soft PLCs, which run in the house, inter-applier of the Windows system and use the rest of the time
to run the actual full-featured Windows, so there has been some similar attempts for the Linux,
and there was start of the community project and effort to gather together to start to
communicate and change the information, and any territory of this was Peter Wenzobler from
the Center of the Transfer MicroTechnics. He was developing a micro-PS electric testing
trick for the micro-multures to measure the torque, which those motors could provide, okay, this is
this setup, and he has been used to use the commercial stuff at the university, it's map-lapse
morning again, so on, but when he switched to the industry, he has not access to those extreme
expensive environments, so he wanted to use Linux, and he now, Nicholas McGuy, who was expert in this
area and worked it on that dual kernel real-time Linux system, and they decided it would be useful
to gather together and to exchange the knowledge, and it was the first real-time Linux workshop,
even which has been prepared this way, it was in 1999, and the other people who has participated
was Victoria Reiken, who was the author of that approach of dual kernel, where we have the real-time
smaller real-time executive for the real-time part, which is running in the remaining time,
non-real-time Linux system, and Linux system can upload modules with real-time task to the real-time
domain of the system, there has been problem with the patents, Victoria Reiken has blocked
the others to do the fully open source solution, there has been alternative RTI, but due to the
patents, they have to invite some bypass of the patent, so they have implemented areas as the
supplier for propagating the interrupts between multiple systems, but it was the dual kernel approach,
and there is interesting start of the attempt to make even the standard Linux
capable to run a real-time task, and I would revine back to 2006, here, where there was 8 real-time
Linux workshop in Ranju, University in China, and there was quite interesting debate between
people proposing a different approach, how to run real-time application on the Linux, and
this is the first time I have met with Thomas Blakesoner from Linux, and he proposed that
based on the experiment of the Kansas University court system, it is possible to slowly change
the main line in which the kernel would be real-time, so it was one approach, another approach
which has been proposed and defended at that time by Niklas Megar from OpenTech, was that real-time
Linux, there has even attempt to implement a GPL version of that commercial version of the
real-time inux, by the way, the real-time inux has been later bought by Windriver company,
so it is part of the software stack, so it was dual kernel approach, another option was that
areas which has special subsystem for the interrupts and RTI and Xenomai, which was proposed by
a group of universities from Switzerland and Italy and so on, Roberto Bacher has defended
the network shop and the last one, no, not last one, second to last one, was his mother report
from University of Polytechnica Revolentia, who proposed a extractum, which was high-provisors,
which can run in one domain non-real-time Linux and in another domain, some smaller real-time
time-posts, alternative, and the last one was Professor Hermann Hartick from the Technical University
of Resden, who has suggested to use L4 fiasco, micro-chernel, and again run in one domain, full-feature
Linux and in another domain, two run real-time tasks, on some small-posts, in some small-posts,
environment, because usually for the actual feedback control, you do not need so much resources,
usually, today you want to have something like artificial intelligence and so on, so it gets
complex, but for the standard close-to-controller of some service and so on, it is simple.
Okay, but we get back to that mainline Linux approach proposed as a fully preemptive kernel,
solution, and what is important to understand, the real-time is not as fast as possible,
so if we modify Linux kernel such way that it will be even a little slower with a little
slower throughput, but it can give or provide some guarantees that it fits into the
required time, for example, timing of some feedback and so on, something like when
he sends a one, it is good, and okay, that attempt started as that court system, and the main idea
to change the mainline kernel to the real-time operating system was that it has became already
capable to run in parallel the task, and even in kernel staff on multiple processors,
because Linux kernel has started to support symmetric multiprocessing in the version
to zero, it works such way that there was a big kernel lock, which allows only one processor
to switch to the kernel mode and other has to wait, then there was ability that when the
kernel task was in the waiting for the IO, it can switch to another task, which can go into the
kernel, and then slowly the locking granularity has tuned to locking only the critical sections,
but still the critical sections in the kernel has not been printable, they have been
has been protected by the spin locks, but that fully-printed kernel switch those spin lock
apart, so critical sections to be printable with the trick that they basically provide
to the model of the Linux kernel as if you have infinite number of the processors, so they
abuse the ability of symmetric multiprocessing of the kernel, that they virtualized processors
yet again, and basically provide for each task separate processor, okay, I am over simplifying
it, but it was the main idea behind this, and this allows to slowly and gradually change the
Linux kernel to have the fully-printed variant, and even that in something like 2004, there was
a version of the Linux kernel, which can work with this fully-printed approach,
the way to get it into the main line to about 20 years, because it has changed so many
subsystems in the Linux kernel, that it was necessary, basically to gradually change one sink after
another, and the Linux storehouse has not so big interest in the real time, because he has main
interest in the throughput, so each gradual change has to be even a vein for the use of the main
line in oops without real time, so first, purchase which get into the main line was some lock
checking, which finds that even the main line kernel has bugs, which can be exposed only, for example,
or three processor systems, which such system has never been built, but at that time, but that
locking test from Thomas Blakeson and others has proved that there are those problems in
us, start to understand that it is useful, even for main line, then real time, mutaxis,
which ability of priority inheritance has been designed by Thomas Blakeson, but
in us storehouse said, okay, it is so complex that I will never get it into the main line,
but when Steve Rostett has written documentation for this and described how it works,
then a Linux storehouse accepts this solution, and gradually we get to the situation that we get
patches into the state that it can get into the main line, last blocking problem was
the print key, so basically the first line of the code which Linux storehouse has written,
because when you want to log in from multiple threads, without blocking it is quite complex.
Okay, now I will go to very quickly through the way how to write real time tasks, so you need to
log the task in the memory, you need to select the proper scheduler, there is some way how to
start the task with correct scheduler, there is how to write the sampling frequency or some
loop which can run at the specified time, and you should have some handling if your task is
interrupted, this is a simple hardware for Raspberry Pi, which you can build with very
rich three or four components, which allows to run the full servo control of the DC motors,
this is simple loop which you can build to control the servo, and okay, if you want you can use
even some sync more complex, so you can build real time tasks from MATLAB simooning, we have
even our oven target for the Linux which I think is better than the MATLAB works, and you can
control for example para kinematic, and on the same system you can run the visualization and
simooning and so on, and the real time tasks which are working at something like 1 kilohertz,
or we have even proof on x86 something working even at static kilohertz with no missing sample,
this is FPGA approach to control a DC motor and because the MATLAB simooning is quite expensive,
there is the attempt to build something open source of the Python, this is attempt of the
Roberto Bachelor from Switzerland, he is picing correct system, so you can draw the control
diagram in the editor and build the code and so on, but if you build the real time stuff you need
latency testing and to be sure that your system behaves correctly, so you can use a secret test,
it is a area where open source automation development is testing a lot of systems in the
quality assurance farm for real time, and you can see that fully primitive kernel,
you can get on the x86 to something like 80 microseconds, on the embedded target it is little worse,
you see that they are running those long-term tests for each day in the year for many hours testing
the system that there is no, no misbehaved, no sample above where the latency is longer than
100 for example microsecond, and if they find that there is a problem then they can offer the
service to solve this, and there is the famous real time event where the finally last year in the
September the new store was accepted the packages for the fully primitive kernel for x86,
risk five and 64 bit arm to the mainline, so it gradually gets to the mainline and now you have
it as part of the mainline, it has been released into dot 12, into dot 13, we have given
the easy prevention option to because fully primitive kernel has drawback in the throughput for
non real time task, because they are switch it too quickly and too much switching has troubles with
the throughput, so this is the easy prevention, we can look at zinc architecture, arm architecture
we have latency profiles, we have the long-term profiles, it is run on OOS serial and we are
doing a can and can have the testing at our system where we check if the gateway or some system
in the can network gets a message how fast it can provide the output on the second network,
so we measure this latency to propagate the message through the system and this is our set up
which is running each day, those tests and okay the result are quite bad, because okay because
the even that the system has very good responsibility for task, cyclic test and everything is okay
when you want to run again, it is today based on the socket can and goes through the networking
clear and we are fighting with the latency here and we are attempting, okay excuse me,
if you're on the other side of that tape you're not visible on the camera,
oh okay so it is mistake but I have feedback if I'm on the other side, okay so those are the
latency profiles for the can communication and I'm now communicating in Sebastian and Mark
ain't ever done that to resolve some such stuff, so it is what we have done, so I will quickly finish
you can find some educational, most educational stuff for Raspberry Pi and some other systems,
how to test a real-time canell and controls DC motors and permanent magnets in chronos motors
and do different kinds of server controls and so on, you can find our articles and
and there is the famous, you know, our publication from Thomas Glicksner about the main lining
of 40 apprentice pitches and there are pointers to the, to the Linux foundation real-time project
we get open source automation development labs and so on, for our stuff which we do with our
students, many TZs for Linux and RTMS new ticks, we have the Picing Corarras alternative for the
Matlab Simo link which is already used by some universities over the world, so it is probably all
because I do not have too much time but if you want to take some reflets they are here so you can
look at our project in the details, okay?