I think we're about time to start for the next talk.
So I'm from Amplores, I work at number zero where we build Ira and we kind of embraced
quick for this and specifically like last year I talked about, last year I talked about
how we wanted to go from normal single path quick to quick multi-path and so this year I
want to talk a bit more about that experience and what we sort of learned along those things.
I want to talk too much about Ira but essentially the goal that we want to have is like that
we want fast and reliable connections anywhere and we want them to be paid based or like
anyone can accept connections in common connections regardless of where you are on the internet even
on your home advice at home or something. So this means you have to do we want to do paid pay
and paid pay means we want to do whole bunching. That kind of push just towards
whole bunching is kind of a lot easier on UDP so that kind of push just towards UDP and then
uses don't really want the unreliability of UDP so and to quick I guess. So that's kind of the story
of how we ended up on a quick story.
All right sorry. Yeah and the basic the basic architecture kind of of of
Ira to some sense is like we do have like inherent in multiple paths which is why we kind of
inherently we're drawn to like wanting to put multi path because because you
be want like if you have like to pull set up you have like to divide it in the home network to
something you can't establish the direct IP connections to it and normally normally those take
require whole bunching. We're not I'm touching slightly a little bit on whole bunching at the end
but mostly not but the really connection is sort of like you make if each device makes a connection
to the relay server we do this via GPS upgrade to web sockets and then we can like transport
UDP data grams via the relay server to each other and the general idea is that we use that to
kind of establish the connection if you can't establish it immediately direct and then we do
whole bunching coordination over that and then you get the other path so we have like inherently
several paths typically these days you got IPv4 IPv6 if everything goes well but the reason we want to kind
of go to multi path is that each path has its own properties right just talked about congestion control
each path has its own congestion controller and with single path you don't have this like you
start on one of the paths the traffic goes through the path and then you switch to another path
and then you congestion controller has to like we start figure everything out again if you switch
back as you move through networks the congestion controller loses all that state there's a
loss detection at such a loses all that state so that's not ideal so that's kind of my drives us do it
another important part for us is that quick also has inherently this protects against
linkability between different paths so if you have like if you send in packets via via for the same
connection via two different paths or quick tries to like avoid showing that these these
dispatch are actually related to the same connection that is something we didn't have in the
single path quick version so that was also kind of the reason for us um so multi path is
kind of an ITF draft it's actually like I think maybe the last version of the draft by now they
even have the iron on numbers now this is a total title for it but it kind of only specifies the
wide protocol it only says like how do you build packets how do you send them how do you decode them
on the other side it doesn't really talk about how you um about how you send data over the various
parts or how you decide to do that that is kind of very much the left up to the application
so it doesn't solve all all the problems but but it gives you tools to do you have an interoperable
kind of why a protocol at least which is a very good start before we go into kind of multi path
we need to talk about paths in and and create itself out of c9000 so quick already had the notion
of a path in a way and it's mostly exists because it's mostly existed because usually you have
like a client or the original design of like HHB3 it was like you have a client behind the
mat and you have like discovering your data center that's not what we want but that's the original
design so the client the math could get rebound like if you can actually as I look for like a minute
of something and then you suddenly start sending packets again um it did not could have read
um good of like already decided that that flow was already um the first flow was already finished
so it has to rebound and you not and not mapping um and then suddenly the services like
valid packets come in from somewhere that isn't the client but it looks like it is the client
um so what the server had to do is like send this um you have to validate that there's actually
the real same client um and and validating that means like knowing it's the same TLS just it can
still control the TLS identity the reason to validate is that this can also like be an on path
a dark or something or why you suddenly pretend you're sending to something from somewhere else
and then you can like potentially redirect a lot of data through some interesting
victim on the internet somewhere um and that's kind of why this part challenge kind of mechanism
already exists um and and and and not everybody is this kind of the the primary reason for this
and and part challenge is kind of like identifying that that there's still the right there you're talking
to and and they are now just on a new address um so that looks something like this like
assuming that your your client is originally sending um from from like the first address um
it's sending data then at some point the client is going to be unaware of this so
at some point it just starts sending again like streamframe in this case like application data
um and uh it just going to appear from a new path so if server has to do this part challenge
thing client has to send the path response the client has to make sure that this comes back from
the same same address that it receives from for the important in a second but right now the
client doesn't really know about this yet it's just blissfully unaware um um and but I just
tend to a response uh this this packet has to be patterned to like uh 12 on the bytes payloads at least
you get like the IP header over at home that on top of that and that's kind of to make sure that
the path is actually capable of carrying traffic and um that is like the data grams that is big enough
for for to be usable so it's like gives you a full full idea that the path is actually usable
only when this path validation has succeeded can the server start sending one of the new path
um and uh like for our implementation of multi path we are writing everything in rust and we
we started from the twin rust library and this is kind of the migration that it that it's supported
uh in inside the data structures um there is however like even still in in RFC 9000
uh that that is not the limit of of what you can do like only the client is allowed to migrate
but um only the client is allowed to migrate but that doesn't mean it can only do in volunteer
migrations it can also actually knowingly migrate for example like a uh a commensituration is like
when you have a mobile phone with um Wi-Fi and mobile data interfaces
in theory what what the client is allowed to do is like it can start if it is aware of this it can
actually already try and verify validate essentially the um the other path so it could try and
validate the path for the second interface if it's originally been sending on on on the first
good rest we can try and like said it's also dressed as the second interface and it can do a
part challenge um as long as it only puts like part challenge and a few very um a few very small
uh amounts of uh allowed friends in there it it it it only counts as uh this thing that is called
like a probing packet and probing packets don't actually switch the path to to the new one like
you saw in the involuntary migration this means that um the client can already verify this path
uh the other thing to to notice is that like I added this part challenge on the way back
like the server can also like um it's we'll see to the probing packet so it knows it doesn't
need to switch the path to the client doesn't need to switch but it can also already start to validate
this path by putting its own part challenge in because only the person that the side that receives
the response actually knows that it's validated and both are a feature know that it's validated
so you need this three packets before you can fully validate um
then if you have like two parts validated or you could even like pre-validate like more
part if you wanted um i'm not sure if anyone actually implemented that fully um but then you're
still sending on the first path and then you like switch to the next path by sending something
that is not a programming packet um in this example ping but it like it doesn't really matter right
um but that switch is kind of like happens like suddenly you just like switch to the new path
and now you're on on that path um and now you're you're probably when you do that you reset
your congestion controller states etc and then they they they build up their knowledge again from them
from there um this kind of brings us to like how multi-path model things because in multi-path
um uh they bring in the notion of a path ID and the path ID is essentially just extending
the packet number spaces so in in original quick you got like three packet number spaces like
the and and the packet number spaces literally just like the packet number like you start at one
it keeps incrementing with every packet you sent quick and ever reach out to the same packet
um also in in case of loss uh it built a new packet with the data that was lost in it but the
packet number will always increment um the initial hand shake is like the spaces are like very early on
most of the stuff is in the data space and that packet number just like keeps incrementing so for
quick to be able to like really get good loss detection signals and congestion control signals
they wanted to have the the packet numbers for each path separately which is why they decided to
just split the packet number spaces up path ID um it also affects the crypto crypto um how you
have you encrypt the the packets originally also is just on with the non-spaces like just from
the packet number and now now we'd include the path ID um but the combination of this basically
should also never be reused for for for the cryptography as well because um if you were using non
uh that's not very good um so this this spaces and packet number spaces are actually like
quite difficult to manage in a way um so the the reason is that the mapping kind of was
was very difficult uh for like from from before multiple to after multiple uh like just this
the space is like if you if you kind of skip over initial and handscape hands hands shake space
um because you don't really spend very long in that and uh but the the data you kept per path essentially
and the path is is um like ideally you want to keep your congestion control straight and all that
per for two builds like IP import source and destination
um but also like the the space ID also contains the crypto keys so each um especially during the
handshake you have like so you have like a cryptography of keys to to encrypt the very first packet
in quick um that are just like for the client hello then the server hello from TLS is kind of
encrypted the different keys and then finally you have the final keys that the rest of the connection
is kind of encrypted with so so you have to derive this three these three sets of keys
at the beginning as well and they are still like um this this whole motion of like multi parts
split the packet number spaces into more packet number spaces doesn't really match fully
one to one to that because um the cryptography keys for the entire data space stay the same
even in multi path um so we had to like it's you end up having to split this data in in in
very awkward or or at least in in in in the implementation we had in very awkward ways
the per connection data like path data which contains congestion control, etc
it only had like the version now and like the previous version and the reason to have like the previous
version is to like protect against basically someone faking a migration
so so the thing that you did there was like when you observe the migration as a server
you would still kind of try and path validate both a new and the old path and then if the
old path still validated then you know that someone was trying to fake a migration
and you can sort of recover quickly so that's that's why you have like this previous path data
stuff as well um
this thing is like to connection identify this kind of an image of big thing in the you need to be
aware of in multi path because the problem is like right where do packets come from um and
the way to do that so because of this migration right you have this involuntary migration that
can happen um packets could suddenly come in from a different IP address so you can't use that to
know which which um which connection the packet belongs to because that can suddenly change
so quick solves this with CIDs connectionities um it's essentially uh a small byte array in
in the packet here there are somewhere fixed offset um depends on the header used but essentially
on some fixed offset and and that means that you can use that when you see a packet
coming in from your sockets you can look at that connection idea and go out out belongs to that
connection that belongs to that connection um but this means also that because it is on the
receiving side that you do this the sand aside uh has to know the CID that you want and so you have to
issue the the the CIDs which is like with a new connection ID frames um in in multi path that's the
new apart from your connection ID and the headers actually are completely unchanged in multi path
so the way the way to do it is basically your issue site at CIDs for each path ID so so the
states of the CIDs kind of gets um multiplied by the path you want to have um but the variance
like one of the interesting consequences in implementing this is that CIDs the way um when you
had a single path the way they were issued you only have an issue a um you you always have a CID
essentially from your peer because the peer is sending to um the way the way the connection is set up
you could never end up with a CID okay usually you wanted to rotate CIDs like when you have been
idle for like a minute or something you might as well grow well let's rotate the CIDs because why not
and in the worst case you couldn't do that if you didn't have more spare CIDs in multi path now
if you want to start sending on your path um even when you receive the packet on that new path you may
know there but it's sent because you just might not have CIDs yet and that that fallibility kind of
it was as also quite a lot of impact in the implementation um so which brings us to like
opening paths like managing paths I guess so opening paths now we know how you can like send
the packet on a specific path and and all the sort of consequences they're committed like
opening path is is is the state the state uh that you have to kind of track
purpose kind of triggers well in in uh and not yeah um firstly like only clients are allowed to
open path which is kind of uh easy from like that that carries on from like RFC 9000 thread
you were only clients were allowed to migrate and it kind of simplifies it simplifies a lot of
design but it's a little bit annoying for us but we can generally work or because we want to be
paid payer right we want both sides to be equal but we can sort of work around this by the the client
is always the one that starts a connection um but so so to open a path a client has to send well
or and a side must have to send or receive basically um uh a packets on a new path ID and that's all
it can even state still be on the same 4 tuple although you probably want to avoid that um but that's
not enough to like fully have uh uh the open path because each 4 tuple that you actually send or send
and receive them all has to be validated on before you can start sending on it so if you if you
as a service only receive a packet that happens to use the CID of a new path ID that's that wasn't
used yet uh you have to then go and validate and only when all of that succeeds then then you can
start using the path um abandoning is kind of also um both sides can abandon the path for
whatever reason coordination wise this is also tricky because if you end up with no paths you don't
have a connection anymore um so you you have to yeah you have to be a bit careful there as well
um so that kind of gets you to like uh the the the last I don't know the last challenge of
of most part I guess that I'll talk about which is like um package scheduling is like well what
you actually like if you have data to send you have to put your your your pending data that you want
to send you have to decide where to send it to there's depends on whether there's any congestion
control window and things like that um what do you have CIDs um uh this is kind of like awkward
to decide like the the way the way we decided to do this for now initially at least we will
improve from this later it's the but initially we we only want to send to uh as a single remote
at a time like as the primary um this pathway to spots uh back up available is kind of a mechanism
in the uh in the in multi path that is kind of entirely advisory and and we send it but we don't
really use it uh I'm being told my time is up so um yeah package scheduling is sort of like you
have to balance all those things and and and not to too many checks upfront but still kind of
managed to send uh send your package on on the path that can actually send them or decide to
wait um there's a few trade-offs there uh the thing that I didn't get ran to uh how we then use
multi path with uh quick natural versatile it's basically using part-changes just to do whole
pinching but all the kind and skip past that part of this yeah okay uh so things like great
talk for us uh so we don't have any time for questions but maybe uh you can just catch them uh in the
hallway or uh some other time um