As a first speaker, we have Oscar and I'll let him introduce himself.
Okay, so it seems with the smikest working, that's great.
Hey everybody, thanks for the introduction.
And yeah, let's get started.
I'm here to talk with you about the fast and disparities,
which is what I've kind of called the various congestion control features
that we've been experimenting with for Firefox's quick stack.
As Max said, I'm Oscar, I'm a working student on Firefox's networking team
and over like the last 10 to 12 months.
I've mostly been focusing on quick and on congestion control.
So yeah, first of all, it's a very short primer, what congestion control even is,
so we're kind of all on the same page.
And inherently congestion control is trying to solve the problem
that if everybody on the internet would be sending as many packets as they could,
just to get their stuff out quickly.
All the routers would get congestion and we would have packet loss
and basically the internet would stop working.
So what congestion control is doing is trying to algorithmically
find the ideal centrate without having any knowledge about the actual path
and about who else is on this path and sending data.
And one sort of algorithm is cubic, which is what neck view implement,
neck view is Firefox's quick stack.
And cubic works with an additive increase and multiplicative decrease approach.
Let's quickly look at that. If we look at this graph,
we can see that here the congestion window that is the allowed centrate
is over time slowly additively increasing.
Until it is at a point where we're actually congested
and we experience packet loss and then we have a congestion event
and we multiplicatively decrease the allowed window again.
And with that we get kind of this zigzag pattern that you're seeing
that homes in on the ideal centrate ideally.
And one more building work of congestion control that I want to quickly mention
is slow start and for that we're going to zoom in on the very start of the connection
and that is this exponential ramp up at the beginning of the connection.
So we're kind of getting quickly getting up to the available bandwidth
and not just like very slowly ramping up.
So the name is kind of funny, but yeah, that's slow start.
Having said that and now we're all in the same page.
I want to give a very quick shout out to Huwis, which is an amazing open source
web-based visualization tool for quick.
And for example, all of my congestion window plots that you see here,
basically just screen grabs from Huwis.
And so if you're working with Quake and you haven't checked out Huwis yet,
you should, I encourage you to take a look.
With that, let's get into the first feature.
And that is Paris congestion event recovery.
So let's unpack what that means.
And to do that, I want to go back with you to foster 2024.
Huwis Manuel, I'm not sure if he's in the room.
But Huwis Manuel, who back then worked at Firefox networking,
and he showed the talk about H3 upload speed and Firefox.
And he showed this graph and he said something along the lines of
packet reordering causes too many congestion events.
When he was talking about stuff that we are yet to more thoroughly look into.
Ah, there is Manuel.
So yeah, let's further zoom into this graph to understand what that means.
So here, all those orange dots are
extended, are coming in for packets.
And here we see this reordering.
So those for green dots, we didn't get an act back for them,
but we did get an act for packets that were sent later.
So we are giving those as lost.
But then just a few milliseconds after we're seeing,
we do get actually do get the act back for them.
So this was just a packet reordering, and this loss was spurious.
The problem with spurious loss is that it does lead to spurious congestion events.
So the congestion controller treats the spurious loss just as it treats
any other loss, and our send window is reduced.
And this is happening despite the spurious loss,
of course not being any indicator for real congestion
happening somewhere in the path.
So yeah, if this happens more often, then this kind of course
like heavily degrade performance.
And of course also completely unnecessary,
because it was just a packet reordered.
Luckily, the cubic RC also suggests a mechanism for both
detecting those and for recovering from those spurious congestion events.
And so at first, we implemented the detection logic,
because we wanted to get a feel for,
we wanted to get a feel for how often we actually see those in the wild.
And if it's actually like, if this never actually happens on real connections,
there's no need to care about it basically.
But 5% of connections actually do see those spurious congestion events,
and we found out that those that see them see quite a lot of them.
So heavily degrade performance for up to 5% of connections
that doesn't sound good.
And so we implemented the recovery mechanism,
and this is actually now merged.
I think this Monday with the latest naked release
into Firefox nightly.
So let's take a look at some measurements,
and take a look how this actually works.
Before we do that, fun fact.
My home Wi-Fi is one of those lucky 5%.
Lucky, because this was a great testing ground for implementing this.
And so all those measurements that you're seeing have actually been made,
sitting at my kitchen table in Berlin.
So first off, this is kind of like the healthy connection.
So no spurious congestion events happened,
and we have a 7 second completion time for a 50 megabyte upload.
My Wi-Fi is also slow, but let's not get into that.
But yeah, this is kind of the healthy connection.
We can see that in the beginning of the connection,
the violet graph, we have the ramp up of slow start,
and yeah, this is how it's supposed to look like.
Now this is how it looks like with spurious congestion events.
So we have a 13 second completion time,
almost taking twice as long.
That's of course not very good.
And if we look at the graph then we understand why,
because the never was a slow start phase.
Again, if we look at the healthy one,
we have this ramp up in the beginning.
Here, apparently there was a spurious loss.
Right in the beginning of the connection,
we accepted slow start, never ran up.
We're never getting up to the proper bandwidth,
and throughout the whole connection,
we never have actual loss.
We never have actual congestion.
We never actually touching the bandwidth that we're supposed to use.
And then this is how it looks like with the recovery patch.
We're back at 7 second completion time,
and if we look at the graph side by side,
the healthy one and this one,
we see that they're kind of looking almost the same.
And so it can be said that with this recovery patch,
we were able to basically completely emerge
just performance degradation, at least in my home network.
And if we zoom in further,
we can kind of see how this recovery works in action.
If you look at where the error is point at,
you can see this dip in the congestion window.
For example, I think it's best seen on the one on the right.
We have a congestion event with reduced the send window,
and then after just a few milliseconds,
we see R, this was spurious,
and we basically just restart of state
as it was prior to the congestion event.
So yeah, we have this pretty strong anecdotal evidence
from measurements,
and we also have some singular from CI simulations and benchmarks.
But the question is, what about real world data?
It would be really nice to be able to say,
yeah, we're also seeing that this is actually impacting
our users, we can prove that with data.
The problem is the higher level metrics,
like throughput, for example,
are very noisy for changes like that,
because Firefox has such vastly different environments.
So like all the different Firefox installations,
and all of the network environments,
that's a huge melting pot,
and it's very hard to see any signal through that,
especially if like in this change,
we're only changing something for a smaller subset of users,
in this case, those 5% of connections.
So yeah, even though I think this change is quite impactful
for connections that have packet reordering,
it's very hard to actually prove that with telemetry data,
which I think is a shame.
But having said that, let's move on to the next future,
which is called alternative back off with ECN.
To understand that, let's first have a quick primer,
what is ECN?
So explicit congestion notification.
The idea is basically that right now,
how our congestion controller works,
we always have to have loss,
it's kind of part of this design.
But loss is not good,
like we ideally wouldn't want to have loss.
So ECN is a mechanism for having a signal,
to adjust the send rate,
without relying on packet loss as a signal.
And that's basically working through the middle boxes,
like for example, a router,
and notifying the sender, Firefox in this case,
if a queue starts building up,
and if congestion might happen soon,
so that the sender can react.
But this also means that the path,
the whole network path,
has to be capable, of course,
marking, like notifying,
and also passing along the signal,
and not stripping it.
So the internet has to play along for this to work.
For that, let's look at the state of ECN right now.
And before we talk about this graph,
just a very quick disclaimer,
because this is the first time
that we're now actually looking at Firefox telemetry,
and so to say real world user data,
that of course all the data that we do collect
for Firefox is anonymized,
and it's also not like we're collecting this
and like doing our finger with it,
but all the data is public,
and you can actually look at this exact graph,
if you click the link beneath the slides,
that I will upload after my talk.
And yeah, I think not too many people are aware of that,
so I just wanted to shout that out.
But yeah, state of ECN,
we can see that about 60% of connections are ECN capable.
So this is promising,
and it might be an indicator that it could be worthwhile,
actually optimizing what we do with ECN.
And this optimization,
alternative backup with ECN.
So how it works right now,
is that ECN just triggers the same congestion event
that was also triggered, so the same reaction.
But because ECN is an earlier signal,
so ECN is saying,
there might be congestion happening soon,
but it might be a little less saying there's congestion.
We could have a smaller decrease,
and this would then make,
if we have a smaller decrease,
increase our average congestion window
and lead through higher overall utilization.
So this makes sense in theory,
it looked in some simulations
and quite honestly,
it also wasn't that big an effort to implement this,
so we implemented it,
it would be nice to know,
how does this behave and how does it look like in reality?
And in this case, again,
is usual the higher level metrics too noisy,
and lower level metrics,
there were some that we could have looked at for this change,
like for example, packet loss,
we could have looked at how many ECN reactions there are,
and the problem was that this got shipped together
with other features,
it also could have influenced those things,
so this makes it very hard to isolate
as features impact,
so while we were able to say that this didn't
like, while we would read something,
it's very hard to say that it,
like, reason about actual impact,
apart from, yeah, it makes sense in theory.
And then another thing,
we wrote another probe,
which is actually,
which we didn't have before,
and now just recently added it,
which is checking how many of our congestion events
are actually caused by ECN,
and now despite there being 60% capable paths,
it's only like 3% of congestion events are caused by ECN,
so if we knew that before,
yeah, the impact of this feature is contained
to just 3% of congestion events,
so it also makes sense that we're not able
to see anything in metrics.
But then the question is,
how do we fix this data problem?
If we cannot see anything in the higher level metrics,
how can we do anything about it,
and how can we hopefully better proof
that the stuff that we're doing had a big impact?
One idea would be having more specialized metrics,
so if the higher level metrics are too noisy,
then it might make sense to,
for example, in the case of spurious congestion events,
have like a built-in filter in metric collection,
so for example, only collect throughput for connections
that do at least see one spurious congestion event,
so with that we would hopefully have less noise,
and then maybe be able to say something
along the lines of, yeah, we see this amount of throughput
for connections that have spurious congestion,
and those are this amount of connections,
and that would be much easier understandable
than, for example, all of this likes that actually before,
and would be a nice statement.
One other thing that we could do is running AB experiments,
which is a great way to isolate features,
but even though we have tools for this,
and it still needs a lot of time to just instrument,
have the extra code instrumentation
to be able to run those experiments,
take time to set up, take time to evaluate,
and so even though it gives the most accurate signal,
yeah, where it's always kind of a trade-off of time
that is available.
Having set that, what's next for congestion control
in Firefox, and that's, at least for what I'm working on now,
slow-start exit, so quick reminder, slow-start,
the beginning of the connection where we exponentially ramp up,
and now look at the problem with slow-start exit.
So if we look at those graphs, again,
we see this exponential ramp up in the beginning,
and right now, slow-start always just exits with loss,
and because of this exponential ramp up,
we can overshoot our actual ideal bandwidth by so much
by the time that we get feedback from this loss,
we actually have multiple losses right after slow-start,
and then if we're unlucky with those multiple losses,
we land at a point where we're quite beneath that,
the actual saturation point again,
and then we have to very slowly ramp up again.
So this all doesn't sound great,
but because we had this before, is it actually worth optimizing?
So I implemented a probe to check how many connections
in Firefox actually do see a slow-start exit,
because Firefox as a browser isn't necessarily sending
like big amounts of data like a server would, very often,
but we often have very short connections.
So intuitively, I thought this is like 5% or 10%,
but as it turns out, it's actually 20% of connections
that's do see a slow-start exit,
and that, in my opinion, is then definitely worth optimizing around.
So what we're doing is right now working on a research project
to compare different slow-start jurisdictions,
for example, high-start plus plus,
which is what, for example, TCP does, in Linux,
and search, which is for which is a newer algorithm,
that aims to exit slow-start without having any packet loss,
by, for example, reasoning about RTT measurements or about egg bites.
And of course, while doing that,
we're going to take all of those learnings that we just talked about,
so we're designing some specialized metrics,
and also in project planning already scheduling time
for AB experimentation.
So hopefully, we're going to get really nice data out of this,
and also maybe be able to publish those results in a paper,
and, for example, help advance emerging standards like search,
with some real-world data and help out the academic
and scientific community in that way.
So I'm personally very excited about that and looking forward to it,
and yeah, who knows, maybe we'll talk about that next year.
And with that, yeah, thank you for your attention,
and if you do have any questions, I think we have,
I have, like, a minute left,
also talked to me in the hallway, there's my mail,
and my matrix, and yeah, and remind again,
feel free to play with our data,
and also check out Cubus.
Thank you.
Very cool, so I think we have a question for,
a time for question one or two,
but maybe fill this, would you mind already getting set of,
during, and I'll unplug myself here.
Yeah.
I was just curious, how many of these, like, other toggles,
switches for some of the features that you were talking about today,
is it easy to, that say do a comparison as a consumer,
to say hey, what about if we switch off some of these features?
For those, it's not super easy, certainly.
This is part of the learnings.
For the slow start, we're implementing,
like, a plaggable framework, where you can,
for example, through a config,
shows which algorithm you want to use.
Yeah, more questions.
So you, you talk about Cubic,
and optimization for Cubic, so any,
are you looking at more modern alternative, like BBR?
We're not planning to implement BBR,
but with L4S, slowly rolling out,
we are with our ECN capabilities ready
to at least echo ECN for L4S,
and be a receive path for L4S,
but we're not planning to roll out BBR.
Okay, thanks.
Thank you.