Hi everybody. I'm Jonas Talmensen. Today I'm going to be talking about OS test. I've been
measuring topics on every single operating system that I can get my hands on. So, I'm
Jonas. Previously I worked at Google for 80 years. I did release engineering testing and
supply chain security for that programming language. These days I'm working full-time on
the sorts operating system and I've got done funding to work on OS test. I'm going to
be talking about today. So, sorts exist my operating system. I just saw the most
chaotic boot ever and I've implemented about 90% of topics in sorts according to the
data that I'm going to be sharing today. Sortics is fully self-hosting and installable operating
system. I'm aiming for server, site, as the primary use case. So, it's actually running its own
infrastructure on sortics.org. It's building itself every night and everything is just done natively,
including this presentation. So, did you know that as a new public's 2024 standard? So, we didn't
release a new standard for eight years and finally, they pushed out a new standard with
exciting new features like AS, Prindiff, NGN.H and just other quality-of-life things. Unfortunately,
their official change is just really, really incomplete. So, I did a blog post about that a few
years ago where I actually found out what's actually new in the standard. And the big problem
is that there's no official test suite for POSIX. There is one that is proprietary, but you
can't really just go and use it yourself. It's a bit out of date, but I'm hearing that it's actually
being updated to the latest version now, which is really nice. But we still need something that
can benefit all of the open source systems. So, the good news is that my friends at NLNet Foundation
they were willing to fund this work. So, that's what I'm going to be presenting today. So, my plan
is to have 100% coverage of POSIX going up to button. So, the first idea is to have a lot of generated
tests that were used for actual standard. And then make sure that's a test for every single function,
every single declaration. And then I move on to actually invoking every function in a handwritten
manner with hello world inputs. And then, finally, I go on to write a lot of low-level
detailed test reads for every single area. Such that at every time we have ever increasing
detail, but there's always 100% coverage. If you have ever read the POSIX standout, you'll realize
there's actually almost code. It says things like the time header shell declare the TM structure
of which shall include at least the following members. There's a lot of sentences like that
and they all over POSIX. So, I passed POSIX with regular expressions. With HTML and a lot of regular
expressions and a lot of special cases and a lot of insanity and it's the worst code ever. And guess
what? It works. So, I was able to output machine readable API files for POSIX. One line per
definition, which says essentially this declaration must exist. It must have these allowed types.
It could be a function, maybe it's a macro. There's a lot of different requirements. Some of
these are option groups and POSIX. There's different namespace rules. It's all in there.
And it's machine readable. Which means I can generate tests. So, I can generate a 4,000 test
for instance here you can see to check whether our clock gets time exists. I assign it to a function
pointer. We'll be right type. And the assignment is only allowed by GCC if it actually has the right
parameters. So, not just any function is allowed. Only one with correct compatible types.
And likewise, I can do the same with struct members and so on. And in do so, I find a lot of
missing features. And then I've went on to do the same thing with the namespace because I can do
things in the opposite direction where pre-processed the system headers. And then I just figure out,
oh, here's a list of everything that's defined in those headers. And then I can come check with
the factual list of what POSIX allows under the different feature macros. And in doing so,
I find a lot of namespace pollution. And there's everywhere, even on GCC, especially due to the
new POSIX standard, which shuffled a few things around. But just because the functions exist,
that doesn't mean they work. So, a lot of functions in practice, it turns out that just return
e-nosis, e-inval, or that just plain buggy in practice. And it turns out that's 1188 functions
in POSIX. So, this is what happens when you invoke all of them. That's my YouTube video. But for instance,
if I just give a Easter Island on food, I'd get free out. And I do the same with every single function.
Some of them require a lot of complicated setup, others are super simple. And it actually turns out
that a lot of functions are stopped in practice. But now I have a data on that.
Then finally, the final phase is to write a lot of detailed testreads. So, so far, I've written a bunch of
stuff on IO, malach processes, pseudo-terminal signals, and UDP, and so on. And when you start poking
at these interfaces in depth, you find bugs instantly. It turns out that if there's a sense
in POSIX, some system forgot to read that sentence. And they're going to have a bug. Especially,
if you look at the stuff that most people actually don't use in practice. So, the next step was to
make a laboratory with every single POSIX OS. All I need is the ability to SSH, into the machine,
run tar, to copy stuff over, make to run it, SSH to execute scripts, and CCF calls to run
compilations. And I have all of these operating systems. If you have any operating system that's
proprietary and is not listed here, please do get in touch because I've wanted to test everything.
So, I ran the test everywhere, and now I have a data problem. Because I found more bugs when I
can possibly report anywhere. 75,000 data points, and yeah. So, even my website scripts
crashed. So, I had to rewrite everything. So, this is my website as it looks now. It's a lot of
data, but it's meant to be actionable. If you have an operating system, you will have a column,
and you can click all the red boxes to explore the different results. So, I'm a front page
show aggregate values and percentages, and you can click things by not 100% and then go in and
see what is missing. So, this is how different operating systems compare and POSIX go in from
left to right in terms of how well they actually do. And you can see that minics is actually the
worst POSIX system. No surprise, Linux is actually the best. But there's a lot of really interesting
stuff in here. Like, OmniOS is doing extremely well. AIX is actually the best proprietary
system, which is very faithfully strict POSIX 2008. So, yeah. I don't think anybody else has done this
before. So, I did the same with the new version of the standard to differ the differences. The red
bar here is how well they're doing on the new standard. And you can see minics, which unfortunately
is abandoned, they also function very well as time capsule to show that some of the historical
systems that just happen to have some of the new stuff because the fact of stuff got standard
the highest. Meanwhile, my strategic system actually implements 90% of the new POSIX
stand-up in their decisions. So, if you want to try those out, my system is actually the best one to
do it on. So, here's some other bugs that I found. For instance, on Dragonfly BST,
there's a few functions that have run parameters in the types. This will probably not matter
if you invoke the functions in practice. But if you make a function pointer to these system calls,
that's going to blow up when passing API mismatch. As in terms of runtime bugs, I find a lot
of that stuff too. So, for instance, if you make an alternate signal stack and execute a new
program, well, POSIX says that when you execute a new program, the alternate signal stack is
reset, but it turns out that Dragonfly BST, Open BST, actually preserved a bit because they
forgot to read the sentence in POSIX versus reset this state. And that means after execute,
well, that process might be an unstable state if a signal happens. Here's not a really cool
graph. So, I've been doing this for six months now and I've been measuring POSIX compliance
over time. And most systems in that time scale have not done much, so that is basically how
it sounds aligned. I admit that most of them. But as a signal in every graph here, so you can see
Open BST and Free BST, they actually have a bunch of recent improvements on the latest releases.
But the most interesting thing is the happy operating systems of upcoming systems.
For instance, my software system in blue here has been increasing steadily. And then, my friends
from Managram, yellow, have overtaken me, which is just annoying. So, we are having a friendly
rivalry and redox is also just accelerating. So, this is measuring how my project is improving
and this is the referendum room where everything is off topic, but I want it to be on topic.
I can tell you that referendum exists everywhere. This is portable. So, yeah. So, my idea is to have
enough useful data to have a critical mass. I can't just go and pop these thousands upon
thousands of bug reports because I don't have time for that. And you know how this, you end up
filing a bug and then you have to debate everybody about it and say, I am right and you are wrong.
But now, because all of the data is just published, it's self-service people can come and just fix
our bugs. I could just keep writing tests. So, in the end, OS tests will just make every system
slightly better. And OS tests is a community project. So, if you want to know some
portable data, like, if you have a .t file, you want to know if that works everywhere.
But just send it to me. I'll just run that .t file everywhere. And if I find anything interesting,
it's a test case. So, we are using it in the continuous integration for
sautics, mannequin, redox, and hopefully more systems, I heard a room of that, Apple might also
be looking at it, which is incredible because that's in public system, that's not so well.
Yeah. So, in the end, I'm hoping that we can make new quality plastic systems and improve
the existing ones. So, what's the best plastic system? It's muscle lip-C, I'm a lean
external, no doubt about it, but a lot of them are really close to getting up there. So,
if people go and fix a few meters, feature macro, saying a few small things here in there,
Gilepsy can probably beat it. And if you want to try out the latest
plastic 2000-24 features, then sautics actually has 90% of them. So, thank you. I'm Jonas.
And of course, with presentation of sautics.
Any questions?
Hello. Thanks for the amazing presentation. I want to ask, have you noticed
like development, accelerating for these OS's live, and a graph to say for my
S's? I'm in the Managom development teams at a say, and we have definitely been trying to use
this a lot to try and beat Linux at their own games at a say, but I was wondering if you saw
other OS groups or other development things, just basically seeing development live. So, to say,
when you add tests? Yeah. So, the question is whether I've seen a signal in other systems
that have been increasing over time. Managom, Sautics and Redox, is the primary one spot.
When I started doing OS tests a few years back in the original form before the major work,
OpenBC did find it and they did improve a bunch of stuff. I did also contact the muscle
and G ellipsi communities, and they've been using my blog posts and so on to actually improve
their systems. I also know that by only G ellipsi has also been using these blog posts. So,
I think it's just a matter of time before I see more stuff in the data, but over time,
I will be tracking everything. So, I can come back next year and say, yeah, POS is 24.
It's now complete everywhere. Is there a certain criteria to any operating system just be sent
to over and get tested for two or how does that work? Yeah, I had the slide, which basically said that
essentially just need to make an SH plus, of course, a compiler. I want to be able
to SSH into systems so I can do things automatically in a lab and I'd deal about some kind of
experience for being able to install an upgrade with system. So, those are the minimum requirements,
but some of the upcoming systems where I've just found ways to execute it, otherwise,
and then I'm just able to put the data down from the CI, even for I'm not able to automate the execution.
Thank you.
What are the difficulties of implementing the new POSX thing? Is it because there are a lot of things,
some functionalities that are complicated to implement? Yeah, so the question is, what's the
difficulty is of implemented projects? Partially, there's just a lot of it, and without a public
test read, it's really hard to know how well you're doing. Oftentimes, you just pour some piece
of software, you see what compilation errors you get, and then you fix those compilation errors.
And then, at runtime, you see what system, of course, are failing and you implement those,
but you don't really have a coherent big picture about it. And that's what those types
are supposed to help you with. It's just knowing how well you're doing, how far you've come,
and then that's just a ton of corner cases that you have to do, but that's for life of implementing
operating system. Thank you. Thank you, another again.