Good morning.
Welcome to the first am lightning talks.
Great to have you all here.
I'll have a round of applause.
Thank you for coming.
So, my name's Sam.
I'm going to talk about testing no mobile images.
I guess everybody here has a smartphone, right?
Anyone not have a smartphone?
Nobody.
And you're probably aware that you're probably aware that the US tech industry controls
most of the available phone operating systems at the moment.
So, we're quite interested in alternatives, right?
In fact, there's a whole devroom happening this afternoon,
the fossil mobile device is devroom on this exact topic.
So, hopefully, I'll get you interested to join in there.
So, what alternatives are there?
If you want to have a phone which is controlled by you,
and which is all open source,
there's a few different technology stacks.
Probably the most obvious is Android itself.
The core of Android is still open source,
even though a lot of the bits you need actually are close source,
and owned by Google these days.
But the core of its open source and the people working on
replacing the proprietary Google bits with something open source,
called MicroG.
So, one option is Android, and there's things like EOS,
and GrapheneOS, which you can use.
And then, the rest is kind of, you know,
it's a, in the open source world,
we always have different flavors, different tastes.
So, if you like things that begin with G,
there's a G option, which is GTK based, post-market OS.
Let's you run either upstream, no more,
let's you run Fosh, which is,
and there's some, the Purism Phone Shell,
which integrates nicely with Nome.
And then, if you like things that begin with a K,
there's a KDE stack, which is Plasma Mobile,
which is also very good.
And then there's a couple, which also keep based,
but are independent.
So, the KDE plasma mobile thing uses KDE libraries.
Selfish OS is something from scratch.
One of the original open source phone OS is,
actually, even older than Android, I think.
And a bunch who touch or Lumiri, as well as an option,
which is also cute based.
So, is anyone running a G based phone?
Ah, fantastic.
Amazing, that's more than expected, actually.
So, there are many reasons you should try it.
Mainly that you want to have fun, I think.
You, you don't have to write any C++ code,
which for me is a big plus as well.
You can use it to make phone calls.
I have the honor of receiving,
I don't know if it was the first phone call,
but one of the earliest phone calls made from known
in the 2018, that's my colleague Bob,
and he successfully phone me from a Libre M5,
while in Spain.
So, you can use it to make phone calls.
And it's very volunteer driven development, let's say.
There's not a huge amount of investment going into the
known phone stack at the moment.
So, you get to help out and be part of the QA team.
A lot of good stuff happened over the last six or seven years.
No, obviously dates from before this month or even existed.
So, the original designs didn't even consider the idea
of having a tiny vertical screen.
And a few people have done excellent work
adapting GTK and building what began as Libhandy
and is now Libhad waiter, which gives you,
in short, it gives you adaptable widgets.
So, that's a really simple example, right?
If the screen is wide, everything fits on the screen,
and then the same app of the screen is narrow,
bits disappear, and the buttons appear vertically,
so that it fits on a phone screen.
And Libhad waiter is also the core of a lot of no maps.
It's quite new, so non-corrupt developers have rewritten their apps
to use Libhad waiter, and it's the recommended way to build no maps these days.
So, as long as you use Libhad waiter and you make a no map,
you kind of get adaptivity more or less for free.
It's not quite for free because you have to test it,
but you get it more or less for free.
You don't have to maintain two different code paths.
On the shell side, no-m-shel does work in progress
to make it phone adaptable.
Not a lot of it's merged yet,
but there's also a flash that I mentioned before that.
It's a fork, no, it's not a fork of no-m-shel, is it?
I'm lying.
The flash also integrates nicely with no-m and it's built on the same stack.
So there's a lot of good stuff, but we still have the problem
that NGOs are doing a lot of the QA, right?
I mean, do you find non-phone users?
Do you find any bugs?
One or two bugs, maybe.
So, we're still not fully there in terms of testing
in neither in the desktop world nor in the phone world.
On the desktop side, the last 10 or 12 years
there's been some innovation.
The big, the big win has been open QA,
which is a tool that began at open-susa,
and it's now used by several of the major distros.
So open-susa, Debian, and Fedora are all testing with open QA.
And the great thing about open QA is it's an end-to-end test.
So it actually boots the full operating system in a virtual machine
and interacts with it like an end-to-user wood.
So it sends keyboard events, and it reads the screen output
and does image matching to check that the rendering actually works.
So that's very different from a unit test,
where you mock the rendering pipeline and you mock the keyboard
and you mock more or less everything,
and then you test that your mocks work,
and they do, which is great.
But then in the real world, you integrate things together
and it doesn't.
So, end-to-end testing is fantastic,
but it's also kind of the most difficult and awkward to do,
because you are testing everything in a VM,
so it's quite heavy weight.
So in 2021, we set up a simple open QA instance for no,
and I did a talk on that.
Last year or the year before, so I won't go over that again.
But then last year, we ran an internship.
These were two interns that we had Dorothy and Tanju,
and they looked at a few different tests.
One of them they looked at was adding a test for mobile form factor.
And when I spoke this, I thought, how hard can it be, right?
You just make the screen smaller, and it'll just work.
So, firstly, if you set the screen size very small,
or rather, if you set the screen size to a number
that isn't a multiple of 16, then open QA breaks,
because it'll only encode the video if the resolution is a multiple of 16.
So that didn't work.
So we thought, okay, we'll work around that by setting the screen to double the size,
which is actually close to how a phone works anyway.
I'll go into that in a bit.
And we'll set two times scaling.
Guess what? That doesn't work, because mother, the composer, the back-snome shell,
has a hard-coated limitation that it won't do scaling
unless the screen is a certain size.
Which I guess the mobile forks are working around that,
but we're testing the upstream version,
and we were limited to that at the time.
So, couldn't work around that either.
So we thought, well, what if we do a hack inside the OS,
and we pass in some stuff to say, I'm a phone?
That didn't work either.
So we did add a mobile test suite in the end.
I'll show you how that works.
I want to show you a little bit what's remaining to do.
What are the next steps?
My aim for this talk is to make you interested in fixing some of these issues,
so that you don't have to be the test department anymore,
and we can automate it better.
So the first question is pixel density.
This took me a while to wrap my head around, actually.
But it turns out that most desktop monitors,
I think this is true, are around 100 DPI.
So for every inch, unfortunately, it's in inches,
for every inch of screen, you have about 100 pixels taking up space.
For example, if you have a monitor that's,
this is a 24 inch monitor, my example up there.
And so the width is 20 inches.
The x resolution it would be designed for is 1 9 20 pixels.
And if you divide that by 20, you get 96 DPI.
On a phone, you don't get that.
So on a phone, the screen is usually much more dense.
There's an example here that I got from my colleague Adrian,
who actually Adrian did most of this work.
I should credit Adrian with doing most of the investigations I'm going to talk about.
You can be here today.
But thanks to Adrian for doing all of this.
So for example, you look at the Librum 5.
Actually, the UI is scaled by 2.
And the way you do this in the real world is you'd set a display descriptor called the E did.
And you put in the physical width and height of the screen.
And then the rendering pipeline would say,
oh, we need to double.
We need to have twice as many pixels right because the screen is the size.
So QMU at the moment just hard codes 100 DPI.
You can't set the screen width and height on the command line.
It just guesses what it is.
Whatever the x resolution is, it fakes a width based on 100 DPI.
So that would be really nice to fix.
It sounds easy to fix.
Does anyone in here work on QMU?
Ah, fantastic.
Does anyone want to work on QMU?
Very friendly project, right?
Lots of people happy to review patches.
So it's a great first patch if you want to work on this.
Adding widths and height parameters to the display.
And then we'll be able to simulate a phone screen much more accurately.
The next thing, the SM BIOS, which is some data structures that exist,
usually in hardware, and they pass information into the operating system of different types.
The SM BIOS has a field called the chassis type, which lets you say,
oh, this is a server, or this is a handheld device.
And there's some code in the moment that changes behavior based on this value.
I think it doesn't give you power saving warnings if you're on a handheld device.
Something like that.
So that would be really useful if we could set that in QMU as well.
And again, we're almost there, right?
It already has a flag to pass in the SM BIOS Type 3.
We can set the manufacturer, set the version, but we can't set the chassis type yet.
So this is another great opportunity to contribute.
What else is needed?
So to get this to where we can fully test how people really use a phone.
QMU patches are probably the low-hanging fruit I would say.
We also need to think about how to do multi-terge gesture input.
Open QA doesn't have any supports for that at the moment, and it's maybe a little bit more complicated
because we have to think how to record and play back gestures.
I think the QMU bits are there, but making it work nicely in a way that it's nice to all the tests.
It's not there, so that sounds like a fun project.
And then the end goal, which would be fantastic, is actually testing on hardware,
having a hardware rig.
This requires people who want to host and maintain a board farm over time,
which is also really fun.
So that's something, I don't know how we're going to make this happen.
It's going to require investment, right?
Probably from companies that are willing to host these.
It's going to be difficult if they're hosted in people's bedrooms.
But in the long run, I would love it if we have hardware farms distributed around different companies
that work in embedded and work in this space.
And we can actually run tests upstream on real phones and say,
Ah, this commits in whatever library has broken this device.
That would be the dream.
For most of that testing, I'm doing a talk tomorrow afternoon as well
than the distro's devrim.
And I never told you how the tests we worked at the moment.
So we did get something to work.
This is running right now in the Nomoc and QA instance.
And what we do, it's a little bit dirty.
What we do is we boot the machine and desktop form factor.
And then we run some JavaScript code in NomShell to resize the window
so that it's the size that it would be on a phone screen.
And then we take a screenshot of that.
So we're testing the adaptive code paths in LibreWater, at least.
But it's not what I envisioned when we got, when we started working on the test suite,
I hope we'd get a little bit further than this.
And maybe we will this year, still early in the year.
But we are testing stuff at least, so we've made some progress.
My contribution to this, I don't have too much time to dedicate to it.
But my contribution is trying to get people to talk to each other,
which I think is one of the hardest problems in software.
So I run a monthly call that takes place once a month.
Happens, the details are in this wiki page.
Or if you search for, that's quite a long URL isn't it.
If you take a screenshot, then later you'll be able to see the page.
One minute.
So it takes place, all turn it.
But European mornings in alternate European, afternoons,
so that people in Asia can participate in people in the US can participate.
And the idea is to break down the walls between distros and upstreams and different companies,
maintaining test suites and just talk about how can we make really good tests,
how can we improve open QA and so on.
So please join that if you're interested.
Confine me after and I can give you more details.
And also, of course, as the fossil mobile device is different, I'm sure you spotted,
which is this afternoon.
So join that as well.
Thanks to everyone that worked on this.
And that's everything from me.
19 seconds left.
Thank you very much for watching.