All right, everyone. Good morning. Still the morning. I want to thank Yaron for sitting
of the stage today and organizing all this open media dev room. My name is Roe Mao,
Boksis. I'm a software engineer. I've been working in the field for about 10 years,
a little bit more. And today I'm going to try to talk about how you bridge the gap between
web browsers API and what you do in the server and the backend. So that's all planned.
So yeah, nowadays the web in the past 20 years has become very, very rich in features
and particularly in media. And it's also becoming the number one software that people really
have installed and where they do most of their work. It's getting more and more rare to
install third-party applications. So we can do a lot of things in brothers, browsers, but
not everything can be done. We have a lot of API for manipulating media listed here. But
there are some interesting challenges when you want to do a really state-of-the-art in
brother specific experience. And so what I'm going to be up today is very specific examples
that I've learned in the past six months. He's installed in my current position, which
I'm going to explain in the next slides. So my background is I've installed media streaming
in college. I was at the course on trial where VRC was at the time. Jean Baptiste also
will talk after me and was there. Then I did some open source contribution with liquid
soap and I'm trying to contribute with a fempeg. But all of that was my non-professional
career and then the other part of it was web. I did a lot of web-sealed websites building
rich application and other things. But recently I started a current role where I got hired
to work on a web application that is actually also very media rich and was very exciting
to be able to bridge those two gaps and see how these two worlds can interact. And that's
where I've based all the topics of discussion for this talk.
So this script, the website that I work for, is a online video editor. It's basically a web
application you can go. You can upload your files, you can compose them, you can add text,
put some effects and export that as a video. So it's the same space as a premier or the
things but it's all web. It started originally as a text base so you would upload a file,
audio file and edit the text and the file with the audio would be cut and then evolve a video
and have a lot of features in particular. In recent years they incorporated AI research
team and started having their own effects that does like, you know, background removal,
I contact you follow the camera and those effects are done in the background typically.
So you're starting to see that we have an application that's very rich that does another
things and some of it has to be done in the server, some of it has to be done in the client.
The other thing I want to highlight is that specifically to the web, a web application
functions essentially very differently than a normal application. And one of the things you
can think about is that you have a whole class of execution context. You can have different
browsers, a different OS so you can be Chrome on iOS. You can be Firefox on Linux.
You can be Safari with my Quest too. Also, you can have different apps. You might
want to do the same projects, open into different apps or different browsers. Are you
going to want to collaborate saying that I'm using Chrome on Linux but I'm sharing that
projects with my colleague who has, you know, Safari on Mac and I want both of them to have
a good user experience in the brother and that's where the real complexity comes in terms
of doing that. So yeah, that's the full stack media editor and how do you make that work
in a context of browsers. So I'm going to focus on a very subset of the features because
there are a lot of features. I'm going to focus on how you ingest media, display them
to the user and export them at the end. So the first thing we're going to start looking at
is what are the capabilities in a browser to ingest media. So imagine that it's a really
user-oriented application. Most of the users are not software developers. They don't know a lot
of media. They might upload very large files of very exotic formats and they expect everything
to work very rapidly. So we have all these APIs that can be used. The web collects. We have
the web audio API. We have the WebGL for composition. But how do you maintain a consistent
UX user experience across different browsers platform and collaborative stations?
So typically here's one example. Codex support is very different across browsers. And we
want to support all of that. We want to really our user to be able to upload the Matroska
files or their mob files or their A, C, Opus, whatever they have. We're going to support it
in the browser. Unfortunately, the range of things that the brother which support is extremely
unpredictable. Chrome on WebKit, Mac OS, my support things it doesn't support on Linux.
So we're going to receive all these files and have to decide what to do with them. On the
contrary, you can see back-end things like FFMPEG would be able to process pretty much
anything if you compare it correctly. On top of that, the set of codex that we can use inside
the web codex to decode hardware is even more limited than what you might support. Some
of them are supported in WebKit. You see, but in WebKit. So it's just a lot. But what
we really want to try to do is try to take advantage of the native anchoring API when
they exist so that we can at least leverage the part of the web API that are relevant
and then use in the back-end when we can, or when we need to. So it's not just codex.
It's also containers. So remember, you have the way you encode the data for what you
can do, but then you have the way you store it in those big boxes. Metroscopes move a lot
of them. And same thing, they're not all supported. And we're worse than that. There's
actually, I mean, there's actually no support for demoxing in the browser. So let's say
you receive a Metroscop, which is not supported or an MPEG TS, which is not supported. But
you still have H264 in it that you can decode natively in the browser. You still need
to be able to access that packet of data, send it to the browser to decode and then display
the frame back. So how you do that is a question that we're going to address typically.
That's one of the example where we can bridge gaps between browsers API and backend API.
So first of all, you get to think about the web codex audio decoding and video decoding API.
So those are native to the browser. And they use array buffer that are basically chunk of
memories that you can decode natively. The same exist for video. That's the one we're
here. And that's what we're going to use. But what we wanted to be able to do is how
can we access that chunk of data that's stored in a mattress cafe, when we don't have
the native support for it. And that's we'll see that in a minute. So yeah, that's one of the
problems we have is your user starts sending a lot of files in your application, a very
exotic format. Some of them we could decode natively, but they are in certain form of containers
that are natively supported by the web API. How do you bridge that gap? So that's one example
we're going to see right next here. And the solution, as you can see, is to leverage
the wisdom, compilation of technologies that are essentially back in, but to lift them
into the browser. So you can start executing, bridge the gap between what the web offers
and what you have typically in the backend, which is FFMPG, demurccing capabilities, but
this time you lift them in the client. So that's was an presentation of the kind of technologies
that are in the browser. So the next, oh yes, sorry, I forgot. The last thing we need to do,
and it's also important, we also need to do renders. So you have your video, right? And
you're composing those videos, you're adding different layers on it, you want some text,
what we're going to do, we're going to leverage the WebGL composition. I might do a little bit
faster on that because of time limitation, but that's the last part of the application. And
this also is challenging, particularly because remember, we want to add text, a perfect example
of something that you would think is simple and actually it turns out pretty hard, because when
you add text, you have things like that. I say we want to be able to do a collaborative session.
So I am working on macOS and I'm adding images on my brothers, renders with the local system
phones. But my colleague is on Android, on Windows, and has different emoji phones and it starts
looking differently. How do you do that too? So I'm going to run some example of how we did that,
it's going to have to be a little bit quick because of time limit, but I hope this will be
insightful, at least I find that interesting when I was working. So the first challenge is how do we
seek when the user upload the file? So let's say I'm uploading a file that comes from a screen
recording smartphone videos. The files can have a wide range of codec parameters. And one of them
is it's going to have a very unpredictable frame rate. Remember like the keyframe. So that's what
I mean. So you have keyframe space at very different places and you want to be able to seek very fast.
So the user expands in the application is I'm editing my video and I want to zoom to 4.8
second, 5.2 second and I want the application to react very quickly. But how do you do that in the way
that's the best possible? What you have to do really is that you have to know where the keyframes
are exactly in your media. So if you want to go to 5.2, but your left keyframe is 4.8, you need to
be able to know that this is your keyframe, seek right into it and decode immediately there to
give yourself the best chance to be as fast as possible in your seeking. The browser can
do that because the browser doesn't have demoxing capacities as I was saying. So how do you do
that without having the native API is a good example? So what you would do is you lift up the
wasm build of a back end library. Here we can say FFMP. So you open your file with FFMP. You scan
all the keyframes inside the file so you know that that's where your keyframe is and then you seek
with it using FFMP. You move past and you get exactly to the frame you want and that's
the point where you're going to send me to the browser. So first step, you extract the timing.
Then you seek to that keyframe at 4.8. Second thing, you read the packets until you get to your
frame and then you take over with the native browser API, render it and eventually display it
on the frame on the screen. So that's the last step you can see. You have this packet that's
available. You can put back to your JavaScript memory. It's that packet the data at 914. Pass that
to the video chunk decode it natively. That way you bridge the gap between what is available in
the browsers and what is available typically as a back end. Another example, and I'm probably
just going to have time for that is when browser processing is not enough you want to upload to
a back end infrastructure. So as I was saying, we have the users throwing a lot of media at us and
some of it we can decode and have that kind of experience that's just described. Some of it we cannot
because it's a codec that's not supported because many reasons. And in this case what we want to do
is we want to be able to send that to a server and retrieve transcoded media immediately that
are usable. Typically they're going to have a high keyframe rate. We're going to also shrink the
resolution because if they're editing on the screen they don't need a 4K video. And we can
that's the time we can add all the service side background, removal, eye contact, and etc.
The second piece of technology that we do in this case is this scenario which adds a lot of complexity
to the application. Basically it's a user application that we want to be friendly. So we really
want things to work as quickly as possible. So the way this scenario works is you upload your files
and you put your files in the web application and I start displaying using the native capacity
of your browser if possible as we just described for seeking. Meanwhile we start uploading in the
background. And when the file is fully uploaded we are ready to hit it up with the back end server
and we switch immediately to chunk of media that have been transcoded on demand just for you
and are really like in a codec that's supported in encapsulation container that's supported
and then your experience becomes better. But that has a lot of complexity because it means that you have
a first kind of degraded experience in the application while you are uploading that we kind of
do what we can and immediately switch to what we fully can do with the application on your file
is uploading. So that's the second piece. If you can't really bridge the gap in the browser
you bridge the gap with the back end service that kicks out as early as you can. That's the one
we call this media transform server. Yeah so that server is implemented and works it's a tiny
FFI layer on top of FFMP the libraries. It produces MP4 segment. Yeah it's dynamically generated
because there's a lot of parameters that change as you edit your video but we also experimenting with
pre-segmenting for caching. Yeah that's the second piece of the puzzle. And I have one minute
to talk about the last piece which is more of an open question. I remember I was saying we use
composition with GL with WebGPU or WebGL to do the final composition of the canvas to a
rendered frame with text and everything. One thing that is still a challenge is how do you do the
composition in a way that remember is described in the browser but needs to be rendered in the back end
if possible because in the back end we may have higher quality. We can do the rendering while
the client is doing something else but the client has been describing their video in the browser.
So when you start peeking it out in the server you have different font but you also don't have the
web API. So one of the challenges that we are thinking of at the moment is how do you do that
translate that and one of the ideas we have is to use the WGPU stack because this stack that's
natively in pre-mediting in rust that support the WebGPU and it exports to native volcano in Linux but
also WebGPU on the web so the kind of challenge we're thinking in the future is can we rewrite the
rendering so the composition is described in this stack in a way that can be rendered both in the
browser and in the back end in a way that's consistent and looks the same for everyone. Thank you
there was a lot to present so appreciate I hope you'll be happy to have a big problem.
What FFI? It's there is a node module that's called FFI-Dush-RS or yeah what FFI do
you use to interface with FFI-Dush-RS or the opposite and it's a rust tiny layer on top of the
lib FFI that provides like a type script safe way to interact with FFI.
No yes the binding itself is linear but all the rest of the stack is type script so we get a
plug right into that. All right if there's no more question we'll yield the floor and thank you guys
again.