Okay, thank you, everyone, and now we're welcoming my use plan for resumable uploads on the web.
Good for users.
Thank you.
The last talk was on security.
That's always a bit tough to swallow.
This one's going to be a bit easier to digest, I promise.
It's about resumable uploads.
My name is Miles Kleidel.
I work as a back and engineer as stream currently.
I'm building a lot of back and services for cloud services.
But this story isn't about me.
It's about file uploads.
And we're going to start where to all begin with the mighty humble input element, the file type.
As far as I can tell, that was first introduced in RFC 1867 in 1995.
I'm not sure if it existed before that.
I wasn't born though, I wasn't on internet, but at least that's what I found.
And it's a really powerful element, because for the first time you really had the opportunity to transmit files from your user's device to the server.
And you have to do it like you wrap it in a form element.
You add some submit button to it.
And then it's submitted as a multi-part request to your server.
That's, yeah, you see the ink type, multi-part form type.
That's how it's transmitted to the server site.
And it's nice that works great until it doesn't.
And there's especially two situations where you will run into issues with this.
The first of all is if you have unreliable networking.
We all know this.
Move mobile cellular data.
We constantly run into situations where we don't have receptions.
The receptions incredibly slow.
You know, I'm not sure who of you has reception down here, I don't.
So we constantly run into these issues.
And of course if the upload is ongoing and suddenly the network disappears, then the dialogue load is basically fresh.
And this gets especially worse if you have large files.
Files sizes are constantly increasing.
You know, there's a video recording being made here.
I'm not sure how big that video recording is going to be.
It's going to be in the gigabytes, terabytes.
And if you transfer large files and then suddenly the network disappears,
users are going to be frustrated because your upload is broken again.
And all you can do is then again upload the same file again.
Try it again.
This waste resources.
This annoys the user.
This is just a really inefficient process.
And like these two situations, they don't.
They aren't distinct, right?
Many people record videos on your phones and try to share them with their family.
Upload them to the cloud.
So these situations, unreliable network and large files, they constantly happen together.
So how can we do better, right?
We want to have a great user experience.
So how can we make uploads better?
Well, there's a zoom will upload for this.
And this isn't a new idea.
Resumable uploads have been around for basically ever.
This isn't a new problem we just thought about the last year,
but this has been around forever.
So how can we improve the situation?
So let me try to illustrate this.
On the left hand side, this phone is supposed to be a client.
On the right hand side, this cabinet is supposed to be a server.
And there's a small video between that that we want to transmit from the client to the server.
Now the upload begins.
Everything's going fine.
But suddenly, we have an interruption in our connection.
And the connection is that severely halted that it even TCP retransmissions can't help anymore.
Now with normal traditional uploads, this will be lost.
We have to start from zero again.
With resumable uploads, we change this by making the server keep track of how much data it received and what data it received.
This is illustrated by this small cloud there.
In our example, the server received about 50%.
It retains the data because this allows the client to resume afterwards.
There's one important step before I can resume.
And that is the client has to offer the server how much data did you receive.
This is an important step because the server might have received less data than a client sent.
Now in our case, the server received about 50%.
So it tells the client, hey, I got 50%.
I only need the remaining data.
I don't need the first half of the file.
And that's exactly what the client does.
It just uploads the remaining 50% as indicated by this half error down there.
And then we're all happy.
The upload succeeded. The file was transferred from the client to the server.
And we didn't have to retransmit the first half of the file.
No more duplicate transmissions. That's nice.
This is a vast improvement for user experience.
It allows uploads to automatically recover from transient failures.
I've talked about connection drops that are usually a common one.
But it can also be because by networks which is right.
Right now in the Wi-Fi you go outside of the Wi-Fi range.
You phone switches to cellular and then the network can drop and the upload can halt.
It can also be that the server crashes, that the proxy crashes, that your client crashes.
It can also be a user error that I'm accidentally turning off my phone while my 10 gigabyte video upload is running.
And then it's lost again.
This is all annoying and can be avoided with presumably upload.
But besides this error recovery and this resilience aspect,
there's also another aspect that it offers new capabilities to the user interaction.
Can also allow the user to simply pause and resume the upload as they want.
They usually know best what their network conditions are going to be.
They know best when it's a good time to upload.
But it also gives the device itself more capabilities.
For example, when you're entering a video call and you're already on a connection that has,
let's call it says a less bandwidth.
You don't want to have uploads in the background like from your backup.
Be running and ruining the video call.
So you could have the operating system or the browser automatically pause uploads that are less priority.
Then, for example, video calls.
And therefore, again, improving the experience of the video call.
But it's also possible to pause the upload space on network conditions, right?
Like, I know if my device recognizes that it's which is from Wi-Fi to a meter cellular connection.
I don't want to be wasting my entire volume on my meter connection with upload.
So I can also just help me in the background when I'm back home again.
Dresumable uploads allows us to do it.
It can say, okay, just pause this upload.
I'll resume it in an hour when I'm back home.
And all of these benefits can be brought to you by resumable uploads.
As mentioned, this isn't a new idea.
People have been doing this for quite a long time.
But in the early days, you usually have to choose one of the lesser eagles.
Either you go and implement your own proprietary solution for this.
This is what has been done by big platforms like YouTube or Dropbox.
They all have their own implementation of something like this,
but these are not accessible to you unless you work there.
There are, and where are few open source projects that targeted exactly this problem.
But they were usually focused on either a specific platform,
such as the web.
But that didn't make them usable on Android on iOS,
or when you wanted to integrate it with an external API that you provide to your customers.
Or it was specific to a specific stack like PHP on the back end.
But whatever you're running Python or something else, then you can't use it.
So there wasn't really a standard for this.
There wasn't a common way to do this.
And we ran into exactly this problem in 2013,
when we wanted to offer reasonable uploads on our platform.
And we thought, okay, what can we do?
We build our own solution that is proprietary.
But we decided to do something else.
And we started an open source project called TUS.IO in 2013,
with exactly the goal of improving the situation.
Instead of creating another specific implementation,
we started with designing an open HTTP-based protocol
for reasonable uploads.
The designing a protocol is actually pretty simple.
All you do is you write down a spec that can be as loose or as vague as you want it to be.
You publish it on GitHub or your own domain.
And then that's it. You're a protocol designer.
Besides that, what the benefit of a protocol is,
that it is the couple from the implementations.
So the protocol can involve and then you can have different implementations of these.
And all of these implementations will work together if they obey to the protocol.
And that is exactly what kind of came out of this.
We fostered a community that created many implementations.
We also create a few implementations.
But this is really the benefit of having a protocol and then different implementations.
They can all interrupt.
And one of the best way I think that the can be shown is
the TUS project and its implementations are not only used by applications internally.
But the upload protocol is also used in external API from different organizations.
I'm not endorsing these companies.
For example, if you upload stuff to cloudflare,
you may be actually using the TUS upload protocol.
And that's kind of the power.
It's not a specific implementation that we're building,
but we're building an open protocol that can be used by anybody.
I want to give a brief shout-out to transloaded who have been sponsoring a lot of this.
I'm very important quite a lot of money in there and enabling people to work on open source.
So enough talking about this.
Now how does this protocol actually look like?
We haven't discovered this yet.
So let's get back to the example from the beginning.
We have our client.
We want to upload the video to the server.
How does this work?
The client initiates the upload by announcing it to the server through a simple post request, for example.
The post request includes some additional metadata.
For example, the file size, but it could also include the media type information for authentication or association of the uploaded file.
The server then has to create what we call an upload resource.
This upload resource keeps track of the data that has been uploaded and how much data has been uploaded.
It's kind of the thing that the server needs in order to retain information.
And it includes the upload resources location in a response back to the client.
So the client can be aware of the upload resource.
Now we imagine the upload continues.
The client is sending data, but then suddenly the interaction happens.
Now the client knows, okay, I want to resume this upload, but how does it do that?
It starts a head request to the upload resource, right?
See that we have now the same endpoint as the upload resource.
And this query is the state of the upload resource.
The server responds by saying, okay, I've got 50%.
We call that the offset.
So the offset of 10,000 would be the half of the 20,000 that is totally uploaded.
The offset is then the point at which the client can resume the upload.
And it does so.
It sends a patch request as a follow-up to the upload resource, saying, hey, I want to modify this upload resource.
By appending the remaining data that I have from transmitted yet.
It announces again the offset, just to be sure that we're both talking about the same offset and we're not accidentally corrupting the file.
But once the remaining data is completely uploaded, the upload is fine again.
And this is the rough schema.
There's a few more details to make this work properly, but this is the rough layout that you need to have resumable uploads as implemented by task.
I now see upload with a post request, optionally resumed by ahead, and transmit all data using patch.
And all of that is already implementable in the browser.
You just use the file API and either the fetch API or XML, XML, sorry.
XML HTTP request if that's the only thing that's available.
And yeah, we have libraries for that.
They already implement this.
We not only have an open protocol, but we also have many implementations.
Many of them are built by the community for the community.
On the client side, we have a JavaScript client.
The small astrogus year like this is because it's not only for the web, but it's also like for anywhere.
It runs in Node.js, it runs in the browser, it runs in DNO basically wherever you have a fetch API available.
We also have clients in Java, if you enter that kind of stuff.
Android and iOS, of course, as well, because those are the major platforms where you will be running on unreliable networks.
On the server side, we have a implementation in Node.js, for example, in Go, Java, and.net.
But there's many more community contributed ones in Python and Ruby and Rust in many other languages.
And they all interrupt because they all speak the same open protocol.
But the thing is that we're implementing this based on the APIs that the environment provides us.
So we're fundamentally limited by the capabilities that this environment brings us.
For example, in the beginning, I said that, hey, would be nice if we could call pause uploads in the background when, for example, real-time video is appearing.
That needs the upstream bandwidth.
In the web browser, of course, conduit is, because I'm not sure what the other web pages or applications on device are doing.
The only one who really knows is the operating system at the end of the day.
So we do are pausing an upload because more priority or higher priority traffic is, or occurring, isn't possible for us with implementations that are happening, I can use our space, so to say.
For that to happen, we actually need to implement it on a lower level.
Maybe directly inside the HTTP or networking stack that your device is running on.
And, of course, another problem is every application needs to bring their own client that's kind of annoying, increases bundle sizes.
So what can we do about this?
This has actually a question that Apple was faced a few years ago.
They wanted to bring resumable uploads into their HTTP stack because they realized that this could be beneficial for the platform.
Then they saw the test project and they were like, hey, this is kind of nice, but we can't implement test in our own HTTP stack. It's not official enough, right?
Because it was just a document that we published on our web page.
But we said, okay, we can actually improve this by making it more official.
And by making it more official, I mean that we reached out to the ITF, the internet engineering task force, which has the working group on HTTP, which is, you know, taking care of HTTP standards that are out there.
And we proposed idea that we should have a standard on resumable uploads for HTTP.
And they accepted it, we got it adopted.
With the idea of bringing resumable uploads into HTTP implementation correctly, meaning inside clients, inside browsers, inside proxies, so that you don't have to ship them anywhere on your own.
It also helps us get more stakeholders involved, right? We're mostly application developers. We're not developing on proxies or infrastructure, but they have their requirements and their concerns as well.
So by going to the ITF with this, we get a lot more valuable feedback on how things should be working the proper way.
It's currently working good bus call, so fingers crossed that this will complete soon. There's already implementations of this. So if you have a current iPhone or Mac, it's actually implemented already in your networking stack. So you could use it nowadays.
It includes many nice features that we discovered would be great while doing tasks. For example, it can transparently upgrade from classic uploads that I initially mentioned to resumable.
And two resumable uploads. This happens by one XX response, but they're not available in user space by the Fetch API.
So let's think about bringing it inside the browser, right? Why don't we have a resumable property in the Fetch API for example?
Then we don't have to bring the own resumable upload client anymore in your application. The browser will take care of everything.
It can take care of automatically enhancing the upload to being resumable if that's available.
It could pause the upload to prioritize real-time traffic. It could pause the upload when the device awakens after it has gone to sleep network has disappeared.
All of these things will be available if it was in the Fetch API, because then it's handled by a lower level that has all of this information available.
But this is a proposal, isn't it? If you think this should be the case, then comment on the good update there.
But yeah, that's all I have to say. You've probably heard a lot already today, and we'll hear a lot more tomorrow. So here's a few key takeaways for you.
Resumable uploads offer a great improvement for the user experience. You can use them nowadays using existing open source solutions just head over to test.io and use any of the available implementations.
Support is hopefully coming to more devices next to you soon.
But besides that, I also want to just say, you can just start your protocol, right? You don't have to be a protocol engineer.
You just write a spec, publish it, and then you're done. It can really iterate easily.
But if you want to have it more mature and get people from or input from more people, then head over to the ITF, W for C or any other standard organization.
Where you can usually get a lot of helpful feedback that you've not discovered before.
But yeah, that's all I have. Thank you for your attention. I'll be outside if you have any questions of that as well. I think we have some time left.
Thank you for your attention.
Thank you.
And we have five minutes left for questions. There's one there.
Hello, great talk. Thank you.
Maybe I missed it, but how do you make sure that the file being uploaded is actually the same, the exact same file as it was before.
Like, is there like a hash you're comparing it against? And it's so is the hashing algorithm part of the spec?
The responsibility of making sure that the same file is resumed is usually the client's responsibility.
So they have to make sure that they're uploading the same file that they, when they resume, they have to make sure that's the same file as before again.
This is also made sure because the upload resource is unique to the file you're uploading. So for every file you have your own upload resource.
So if you're uploading multiple files, you will have multiple upload resources.
And this is the client's responsibility to make sure it sends the data to the right upload resource.
Thank you.
Thank you. It was very digestible. So great talk.
What I want to ask is whether when you started this project and you wrote that's protocol, like I guess whatever you wrote down back then.
Did you already had a mind that it was the idea to be standardized or to be more official?
No, we didn't come from that background, so we had no experience with the idea of whatsoever.
And for many years, we didn't really consider it until some people reached out to us and were like, hey, if you would make it more official, this is the way to go and we can help you with that.
So it took us a few years to realize that this is the possibility.
Okay, thank you.
How reliable have you found it to be uploading the file after just given an offset without any sort of hashing?
Like does that, do you ever run into scenarios where part of the upload, where something I corrupted near the end of a stream, for example?
We haven't discovered much, it's just with that actually, no.
It's the, again, this is puts a bit of responsibility on the client to make sure it actually retains the same file.
If, of course, the file is changing because it's a document you currently editing, then you will run into issues with this.
But then again, this is kind of more of the upper layer issues.
But yeah, it's a very important, the offset is basically the most important element we have to kind of prevent corruption.
There was another question somewhere here.
Hello.
What about the data retention policy for incomplete uploads on server sites?
This is a very good question and usually depends on your application needs.
What I usually like to recommend is that the upload resources kept around vong enough to allow the client to retry once more.
Because there's a specific failure mode where the upload is complete, but the client doesn't receive the lost response from the server.
Then the server already thinks, okay, my upload is complete, but the client doesn't know it.
So it has to do one more round trip and ask the server like, hey, did you actually get the entire upload or not?
So I usually recommend to keep it around for a short period of time, but you don't have to keep it around for hours.
If that's not what your application needs.
Sorry if this was mentioned already before, but do you have any insights as to why browsers haven't already included this?
It's actually the question that I get a lot.
Sorry.
Why are you, no, no, no, it's not that.
I get the question often, like, why are you still working on this?
It seems so basic, why haven't you done this?
It hasn't been done years ago.
And I think this boils down to is that if you control the client and the server, you can build your own thing and it will work just fine because you're in your wall garden.
So I think the pain to actually go out and start to interrupt their protocol was not too high, so people have just not done it.
And they're kind of okay with, you know, using implementations that work in user space, so to say.
But I think now there's been some momentum to actually get it to a stage where it's integrated into HTTP networking itself.
So let's hope this actually happens now.
Earlier in the talk, you mentioned that standardizing on this also allows proxy to support it.
Do you see any role for them in the resumability beyond just passing the traffic on to the server?
So there's a specific mode where the proxy can translate between resumable uploads and normal uploads.
This is especially handy if you have legacy applications, for example, that don't support resumable uploads or you don't want them to support it at all.
Then you can have a proxy that recognizes how resumable uploads work and automatically takes care of concatenating data together.
And that is something that could be implemented in proxy.
Well, thank you so much.
Times running out, but if there are other questions, I'm sure you can discuss.
All right, thank you very much.