It's better if we're seated, if we're not, I'm going to get in trouble with the staff
from Faustem.
And there's plenty of Spencer's good.
Our next speaker is going to tell you how we manage crashes in Firefox probably.
Enjoy.
Hello, I'm John Carlo, I'm a local, I have a little bit outside of Belgium, and I'm
managed the OS integration team in Firefox.
And today I'll be talking to you about your probably least favorite Firefox feature,
and that's what happens if you have a crash.
I'll skip through the agenda because we're a bit short on time.
Straight to definitions.
What's crash rate?
It was 100 people in the room.
Alex here in the first room has a Firefox crash five times.
We say, OK, there's a 5% crash rate today.
The other metric we use is crash incidents.
Say, Alex and Silvester, the both have a Firefox crash three times.
So we say, we have a crash incidents of 2% because that's
tuner that had a crash.
Which one of those is more important?
It's a bit arguable.
With for various reasons, so which will become clear during the talk.
Most of my data here is going to be about crash incidents.
And that's the number we track most closely.
What can cause Firefox to crash?
Well, the first thing we'll stop this one is just Firefox bugs, programming
errors.
That's probably what you thought this talk was going to be about,
but plot list, it's largely not, it's about other causes.
An important reason for Firefox crashes is running out of memory.
The browser has to execute a JavaScript on the web page,
and do whatever the web page says, so it's somewhat limited in its
freedom, and this can cause it to run out of memory.
There's various machine configurations, which
can make this worse, like running on a 13-bit OS.
And the last category we have are harder problems.
And I use quotation marks because I see this very broadly.
I'm not talking only about bad memory or overclocked machines,
but also, for example, like Windows installation,
that has had three different 20-virus products and a bunch of
fruit cuts on malware on it, from my perspective,
these are all hardware problems.
So who is crashing and why?
We wanted to dive a bit deeper to see how those three categories
are divided among our users.
And one of the most interesting ones I wanted to look at was,
like, what percentage of our failures are hardware related?
So one way we thought about this is we know the crash rate
goes up as the machine's age.
So as a bit of a question, can we estimate the age of a machine?
I think probably the most reliable thing we can get access to
is the bias date if it's successful,
but we currently don't collect such telemetry,
and I didn't have it historically, so it wasn't really useful.
But what I do have is the exact CPU model.
Now, if you look at the release gains of the CPUs,
you see that Intel, for example, has a fairly good cadence,
with, like, most CPUs only on sale for a two-year window,
three-end-year slightly larger.
And we say, OK, we just estimate that the machine
was made in the middle of this window, which
gives us about one year accuracy for the machine age.
That's obviously a rough estimate that it's good enough
for what I'm going to show you.
Now, what we can do is we can take a certain set of hardware,
say, when those release a machine with a gigabyte of memory,
and pull the crash rate over time.
And this gives us some idea of, like, how
the crash rate evolves over time.
And then we need a baseline, which is the amount of failures
that are actually caused by software problems.
This we can't really know.
But again, I made an assumption that the machine
from one to three years old, so not an entirely new one,
never crashes due to a hard-for problems.
So that's obviously going to be over estimating
our own crash rate, but it's fine.
Because what I really want to get is this graph.
The black line that's still over all crash rate,
you could sort of see the back of the curve,
as it says, so the new machine to actually
crash a little bit more, and then it starts
going up as the machines get very old.
Don't read anything into this graph that's not there,
like you see suddenly in terms of user distribution.
There's a few gaps here, but that's because of the CPU
book I think I just explained.
Most interesting part for me is this.
So this blue line is like our estimate for hardware failures,
out of memory problems.
Also goes up as the machines get older,
because older machines have less memory.
But here, we are at 0.3% crash incidents,
which is what we estimate our own software failure level to be.
But the total crash travel, and around six years of age,
also starts to go above 0.6.
So what this means that if we got a crash from a machine
that six years old, it is as likely to be a crash
from a calls external to Firefox.
And if you now look at the same graph,
there's actually a pretty long tail here of our user base
that are on such old hardware.
And this confirms something we had been seeing
for a while is that if a new crash arrives from the field
and we put an engineer to look at it,
sometimes it's been one or two days really digging into the crash
and they come back and they say, I don't understand
what's happening, this crash is impossible.
And I would sort of know why that is starting to be more
important to filter out those issues,
because we cannot fix those feelings.
How do we get to learn about crashes?
We have both crash reports and telemetry.
If the main process crashes
like all of Firefox shuts down, which is a bit the worst case,
you get this window, which is an external program,
we recently wrote in Rust to have mother and translation
support, IDPI, like this center, basically.
If you get this dialogue, we see about 70% of our users
replying positively, so submitting the crash report.
If a tap crashes, but Firefox's a sales stage running,
you see this in your tap, this report is done
by Firefox itself, and here we see already
there's quite a drop only about 30% of the people
who get this submitted, are presumed the rest just closes
in quickly and goes on working.
And now in the final category, is a utility process crash?
I think if you saw an exam or stop,
you have some idea that there's a lot of utility
process in Firefox, and these are invisible crashes.
I'm going to see if I can make the invisible crashes
visible.
Yeah?
So did you notice it?
It's pretty subtle.
So if I didn't tell you this, you might have just
thought this was like an advert glitch, but this
was actually Firefox, entire data decoding, stack,
crashing, and immediately restarted.
We see basically nobody submits these kind of crashes,
and that's obvious, because I lost my clicker.
No, you have to focus on the video.
Ah, is that it?
Yeah.
If we would report such a crash, it would be rather confusing,
because it might have been something that happened
in the background, and suddenly you have this
pop-up asking you to submit that.
It might be more annoying than the crash itself,
which just looked like a small network pitch.
So like from a UX perspective, we should never try to surface
those, which is why we don't get reports for them.
So how do we get to 0.1%.
Some people, and I love you all, have this very deeply
hidden setting enabled, which automatically
sends all crash reports.
So we do get to know about some of these.
But that's a bit of an issue, like 0.1% is not enough
to decode a given example of a media decoding or GPU.
So how do we deal with those?
So a crash report is basically a capture of the process
memory state.
You can add your URL or any comments.
But the memory of your process might
have passwords or rather private information in it,
so we can only collect those if you explicitly agree to it.
But we do also have telemetry.
And what we can do with telemetry is add to the telemetry
thing, basically saying, OK, this Firefox has had a crash,
and it was in this Firefox function with this call stack,
which doesn't really tell anyone anything,
because I hope most Firefox functions have more than one
user using them.
And it's not perfect for debugging,
because it only tells you where we crashed.
But often, it's already quite something to go on.
It's better than nothing for sure.
With a history, this is a graph of Firefox crash
creates over crash incidents over the last two years.
There's a very, very, very big spike.
This is called FoxDuck.
You can Google it.
We did a post more among it.
This is an interesting one.
Somebody asked me about this.
I think somewhere in 2022, in October or November,
you had the biggest ever hit of solar gamma radiation
on Earth, like 50 times bigger than ever before.
And yes, can you see this in Firefox crashes?
I was like, yeah, I can see it.
And I looked into what the crashes were,
and like a major Windows on the virus
thunder released and update on the same day.
So unfortunately, I have to conclude
that galactic data rays have not been
as on the virus software.
And there was a little bit of a depth here.
We shipped a new crash reporter.
We didn't get it right first time, but we fixed it.
OK, what are we working on right now?
The new crash reporter GUI already talked about that.
We can detect some instances of hardware issues.
If a crash doesn't make sense, but flipping a single bit,
would make it make sense, we tagged those crashes,
because it's an indication it could be hardware.
If we crash with an access violation read,
but the instruction was a right, we also tagged that,
because that's completely impossible.
We changed our telemetry work on a new backend.
Basically, this means we supported on Android as well.
And there's a nice new dashboard if you want to look at it,
which is public.
And in the future, not only the GUI, but also the backend
and the crash reporter, right now it runs in the same process
that has just crashed, which can be rather interesting
if that process has scribbled all over memory,
and sometimes you don't manage to recover from this.
So this is moving out of process.
We're going to try this a bit of a gamble.
If we, the moment we show the crash reporter dialogue,
where we can do a quick memory test,
and maybe detect some more machines that are seriously broken,
so we don't waste our time analyzing those.
This shoot land this week, next week, so we'll see if it works.
And remote crash collection, I'll show this,
because it's a bit more illustrative.
So in case we see from telemetry that people are crashing,
but nobody has actually sent in a crash report.
We want to prompt like a small number of users,
a bit more explicitly if they want to send in that crash report.
And then hopefully we can investigate it better.
I hope we can even explain this to like literally include the bug you're seeing,
which I think would alleviate like the concerns
that this is maybe a fishing site or something weird that's going on.
And I hope if we do it this way,
that also people will submit those crashes,
and we can debug them.
All right, that was all.
Thank you for crashing, and.
Thank you for your time.
Thank you.
So I really think the key takeaway is please submit your crashes.
Yeah.
They don't have enough work.
They won't more.
Do we have questions?
Great.
From my perspective.
Have you seen a substantial decreasing number of crash
when you introduce trust into the modular overall?
I would say those kind of things.
I can actually be able to go back to this graph,
because I've skipped over a lot of details here.
So this looks kind of flat, but it's only because this spike
was so ludicrous, they're huge.
But you can see it's actually a very spiky signal.
A lot of these are like third party vendors,
or third party software that does broken updates
and messes with Firefox.
Here it starts going down.
That's not so much because of trust.
We could a lot of work on improving out of marriage situations.
Here it went down again, because we moved Windows 7 to ESR,
so the ESR is not in this data.
And here it starts going up again,
very slightly, because a lot of Linux trusts
switch to A-land, and the waitant ecosystem is not as
material as everything else.
But from the rest, I mean, the rest would be so gradual
that I wouldn't be able to visualize it here.
Yes.
We actually make some benefit of rest in our crash reporting
tooling, because we have some updates, which
catch every error condition in the crash reporting,
and then surfaces to us, to close a few black holes.
We have, and for that, Rust was incredibly useful,
but it's very hard to skip on the error handling.
A collection about privacy.
Is it safe to send crash report?
Is it safe to send crash report when Firefox crash it?
When I was filing the bank formal government form?
Very easy.
Why we have this distinction between the crash reports
and the crash telemetry?
The reports do contain, like, a dump of your memory,
so there could be confidential data in there.
We didn't measure that's extremely restricted.
I like you need to take an actual course
before you can get access to that data.
But if you send, if you opt into sending the crash report,
you do send that data to us.
So there's a bit of a decision for you, whether you're okay sharing that or not.
Also, I can only confirm that we do apply the highest standards for that kind of data,
and that's definitely not publicly accessible.
Where's the telemetry, yes, because that's just like a stack of Firefox functions?
Can you say how many people have access to the crash to two of the
accordance, not so many, right?
Not so many, you know.
I would guess which is 50 people or something one hundred.
And something that the process is complicated to get access to.
Okay, other questions?
No.
Thank you very much.
All right.
Thank you.
What?
Take a minute for a few more.
So, thank you for joining the modular.