Hello everyone, I'm happy to see this many people this morning, also a bit intimidated.
I'm just filling in for somebody else, so this is going to be a bit shorter.
I'm going to talk about why I wrote it in P4MUPSA and why it's kind of painful, but also fun, maybe.
First about me, my name is Dennis, I'm a software engineer from Germany.
My day job is working at Twitch, but more importantly for this,
and one of the main contributors to OBS Studio.
I've also done a couple of things for, for example, another open source project.
And I do know what an MP4 file is.
If you don't, MP4 is probably the most common most widespread media container,
so it contains audio and video streams you can click play, and works a little bit of history.
It kind of is based on what Apple created in the 90's called,
and file format or MOV.
And then the ISO, the International Standards Organization, they took it,
and extended it into MP4, and then now abstracted it further into,
what's called ISO BMFF for based media file format and the CMAF, which is not relevant here, but it exists.
But also what is OBS?
It's free and not so a select streaming and recording solution, GPLPQ.
We have first party packages for Windows, Linux, and MacOS.
And also, tens of millions of users at this point with a massive variety of skill levels,
which is going to be important.
And that also means it's mostly used without backup recording,
so one of our highest priorities is avoid data loss.
And also we have tons of different use cases.
Game streaming is a big one, of course, twitching all that,
but these days it's everything from features recording lessons
to we heard about doctors recording ultrasound machine outputs or all kinds of stuff.
So that brings us to the problem.
To produce faster useful for most users, we need MP4,
to avoid data loss we can't use MP4.
And the reason for that is the structure of MP4.
It's going to be simplified, but the basic thing is,
MP4 consists of these boxes or atoms in the Apple terminology.
They contain different things.
The most important one for us is the media data box, which has all the media data in it,
but it is by definition unstructured.
You only can get meaning to what's in it from the movie box,
the MOV box at the end, which contains descriptions of where all the samples are,
how long they are and all that.
And it is at the end because that's how most MP4 files are written,
but that is also the main problem,
because if you stop writing before you write it,
like you lose power, your storage is for whatever,
you file is toast.
You can't really recover it without a lot of work.
There are some methods, you can do some horistics to detect all the video data
and try to get it together, but it's very difficult on most users,
I'm not going to figure out.
So since we can't use MP4, what have we been doing?
Yes.
So until 2019, we've been just using FLV,
because that's what we throw over our team,
for live streams, why not dump it to disk?
I can't believe it took until 2019,
it's a format that is supported by nothing.
And I also didn't support back then,
didn't support multiple video tracks and all that.
So in 2019, when we also wanted to multiple audio tracks,
we switched to MKV, which is much more modern open format,
but it also has this use, Apple and Adobe.
They don't support it.
And that, again, makes the problematic for all users.
They also cut a lot of things,
so you can't drag it into disk or whatever.
It's a nice format of theory and practice, it's just not really for us.
Now, MP4 has an extension or an update to it,
called the Fragmented MP4, which sounds like it would be great for us.
In this case, instead of having the movie books at the end of all the data,
you have a smaller one at the very beginning,
that only contains the absolutely necessary data
to initialize here to the coder,
but it doesn't contain any information about the actual media samples.
Those are all stored in the movie Fragment box, the MOLF1.
And this one is the sort of every fragment,
and every fragment is essentially decodable independently.
So this means if your file is now concatenated,
this dice or whatever, you can still play it back up until that point,
so you only use a couple of seconds.
But, again, we have problems.
Windows.
Microsoft doesn't really support it.
If you open a Fragmented MP4 file with the default Windows media player,
it'll play it like a stream.
You can't see it just play it like a live stream.
It also doesn't show the duration and explorer and whatnot,
but it's not just Windows.
So it's also not supported by some other software.
The venture resolve used to just spread up crash to the desktop,
if you even navigate to a folder that has one in it.
And it's also really slow,
because in order to get the full duration,
the player will generally read every single Fragments,
or the Fragment header, at least.
And if you own a network share, or a mechanical hot roof,
that is really slow.
Yeah.
And that's also a good end of work.
So it's a terrible user experience.
And we tried and we reverted back to MPV.
But this is where I had an idea.
What if we just, you know, do both?
We write it as Fragment header,
let's say, let's do that before movie box.
Everybody's happy, right?
Well, no.
Dispect us agrees.
And they are right, because you can only have one movie box.
So I came up with a bit of a trick.
You have that free Adam there at the beginning.
I didn't explain it earlier,
but it's essentially a placeholder.
FFMP users, so when the data box in the normal
and before gets too large,
you need to have a longer header with an extended size field.
It's essentially sacrificial eight bytes.
So you can overwrite to get the larger size header.
But we can abuse it a little bit here and get creative.
Because again, just overwrite it with an M data header.
And turn the entire file into a giant data box
that by definition has no structure.
And then slap the movie box at the end.
Now, we have the file that works normally.
If writing dies, okay, we still have the Fragment
that file you can re-max it with FFMP or OBS.
Get it normally before, but if writing succeeds,
we have a normal looking MP4,
and the players will be normally wiser.
So, why do it ourselves?
Why not try to upstream the TFMP?
That's painful, sometimes.
But I'm mailing this shavings side.
It might not be accepted because it's kind of niche.
It also could take a wall before you get merch,
like we talked a month, maybe years, until attack us out.
Into Ali, et cetera, I think there.
And we ship Ubuntu LTS packages,
and we want to have this available for you
in less than eight years.
So, doing natively,
we can have a lot more flexibility.
We can do a lot of experimentation.
It might be fun, and I might learn something,
which I'm not too afraid of.
So, this is a little high level of view.
It's just a piece of code, just for illustrative purposes.
So, I'm very much still followed the FFMP example,
because what they do is fundamentally very simple.
And, you know, that makes sense here for us as well.
We just have one method for each box type.
It's largely based on the ISO spec.
I had to look at FFMP and G-pack,
which is another open source project for dealing with media files.
Because I don't have the latest ISO spec,
it's all pay world, I'll rent about that later.
And, yeah, so I had to use them as references for some stuff
that's not documented or poorly documented elsewhere.
But, I strictly limited it to what we need.
So, it's only the code, so it'll be a supported only
experimentation on keyframes.
And, also, use the opportunity to get us some new fun features.
No, we have like chapter markers, so people can hit the button
and then in the editor, they see like the marker
and for highlights and stuff like that.
So, the end result was one big aspect request.
It's about 2,900 lines for just the mixer,
the rest is like your eyes, stuff, and all that crap.
Took a few days, then the review took about a month.
And then it was mostly done with a couple dark fixes,
but it's been mostly stable.
But, one fun thing at hand, like this was made 29th to this later.
Somebody submitted a patch to FMP to the same thing.
Pure coincidence, I've not talked to Martin before,
and he cited Apple having apparently done something similar.
After a bit of deliberation on mailing,
it was really quickly and renamed to also use hybrid,
which is, to tell him I came up with, I didn't lobby for that.
But, I think I have all the OBSs so huge.
People are going to call it that.
Let's just do the same in FMP.
So, yeah, you can do it in FMP now.
I think it's maybe in the 7th point, one tag, I'm not entirely sure.
But, yeah, if you set that, it'll do the pretty much the same thing.
Small details, but yeah, you can do stay recording with FMP as well now.
So, some fun ideas, what to do with that.
I wanted to have a demo, but I ran out of time.
So, the first one, somebody on FMP is not really hybrid,
because it's not both at the same time.
So, I think it might not do it anyway.
So, what we can do with streaming with HLS.
We have this thing called Bidrange.
We can point into offsets into a file,
rather than just having entire thoughts for each of the segments.
So, what we could do is have an HLS playlist
and point into the hidden area of the file,
and stream it like a normal HLS file.
But at the same time, you can use the file for progressive download,
and it'll be a normal NB4.
You can also do server-side concatenation.
It's something I talked about with a friend who had a use case of
essentially recording directly to update storage,
but still wanting a file that works well.
In these cases, it's a bit more complicated,
because it's not using Google Cloud Storage,
but with Google Cloud Storage, you could just write the new header
and tell the Cloud Storage to concatenate the server-side,
and you just get a new file that is progressive for all intents and purposes.
You could also do this on the fly.
I think it's not exactly the same,
but Vimeo does something like this,
where they only store video-in-set-mended formats.
But when you download it, they on the fly do concatenation,
which is pretty cool.
So, what to come?
Well, I'm also going to do MOV or QuickTime support,
because ProRes, people want ProRes,
at a time go track, and a couple of fun things in there,
not too important.
And last night's audio is hard.
We already heard about that earlier.
Priming samples are one example that I went into,
so most audio encoders like AAC or Opus.
They will have the number of samples at the beginning
that are silent to prime the encoder decoder.
And it obviously didn't really care about that,
so we had some audio-desync with some encoders.
They had to fix.
I also found a similar bug-in-effort fan pick,
that provided some solos at least.
Videos are too.
We already heard about frame-reordering earlier,
which is the devil,
and makes all of this more complicated than needs to be.
And before it's pretty simple though,
the spec is huge.
There's a lot of crap in there you don't need.
There are things in there like,
hey, why not use an edit list to cut out part of the video
and reference an entirely different file for these media samples.
Things that nobody really implements,
but they had higher aspirations from writing the spec, I guess.
And yeah, also I learned that days of hard work
can be summarized in like five minutes,
one minute if somebody knows what 94 is.
All right, some thanks.
At the same time back of course,
they have a lot of cool stuff.
They document the undocumented.
I wish they had some more comments in their code,
but hey, it's what it is.
GPAC is also great.
MP4 box in general is super great for dealing with MP4.
It was a bit too much for what we wanted to do.
But they probably have the most complete feature settings,
including some of the more niche things that are in the spec
that nobody else implements.
Apple's old documentation is also great.
They have some good examples for how priming samples work and how to deal with that.
And then I don't want to thank the AI so,
because the paywalling of these specs is really annoying.
MP4 alone.
You basically need three different documents
that are all like 200 euros or so,
maybe 300 euros,
just to get audio and video working.
And if you want to stuff like PCM,
it's a separate document.
If you want to have,
I don't know,
CMF and Dash compatibility.
That's another document that's expensive.
It's a whole mess and it's really annoying that
what is arguably the most common,
most important format is not accessible.
It's a little fun fact.
If you want PCM and MP4,
the spec for that is so short.
The free preview you can do,
or not it's actually an entire thing.
So I recommend doing that.
And with that, I'm already at the end here.
If you want to complain about a house or MP,
you can reach me there.
That's another thing I've been involved in,
that people don't like.
But hopefully, overall,
I think I gave you an idea of how this works
and why we did it.
And hopefully, we solved some uses issues.
And with that,
I am at the end.
Thank you.
All right.
Any questions or remarks?
Sure.
I just like to add that I am the Martin that you mentioned.
Oh, nice.
And it's not that you're making,
you know,
I've heard this meant for this whole feature,
because having also done the same thing,
it's nice that somebody can present it in a nice way.
Thank you.
Thank you for also getting it into FFM pick.
I know I was so that,
Oh, repeat the question.
Yeah. So the person here is Martin
from who contributed this to FFM pick,
who said,
it's nice that I explained it here.
And I will also say,
it's nice that you contributed to it to FFM pick,
because it's just,
you know,
it's what everybody uses.
And I didn't want to deal with the politics and mail-in-linguists.
Anything else?
No.
Cool.
Oh.
If you have a little minute,
can you go back and explain a little bit more
to the actual trick that you used for it?
Oh, sure.
Right.
I regret making this animated now.
All right, let's go from this.
As I mentioned,
so actually let's go even further back,
because I think the graphic for the full thing is super important.
Where is it? There we go.
So as I mentioned,
in the MP4 file,
as it is produced by FFM pick,
and actually recommend it to be done this way,
by the Apple spec,
instead of this free box here.
And the reason is that the MDAT box,
by default, only has a 32-bit size.
So you can only stop to forget about its data.
But if you want to have more than that,
you would have to either over it,
entire file or the trick that came up with is to have
the free box here that's eight bytes.
So you can then overwrite this with the 64,
64-bit size header,
to store up to,
I don't know how to memorize it,
I can't memorize it.
And what I figured is,
why don't we overwrite the,
got them it?
Why don't we overwrite
the free box with an MDAT header,
but just in a fragmented file.
So all of this becomes an MDAT box.
And now everything in here is essentially invisible,
unless you are partially doing something that it shouldn't.
But then we have the movie box that still indexes the samples
that are in here.
And that allows us to have this resilient format
that is still fail-safe.
And one of the fun things about this,
the way we do it,
the feedback doesn't because we don't
interleaf samples as much in this way.
This actually has negative overheads
once you go past a certain duration,
because the movie box is smaller.
And so if I read it out there,
it doesn't go by the hybrid format,
and it can read that.
Yeah.
Yeah, as long as it's spec compliant,
it just reads the movie box,
and then it gets later,
and it knows everything that's not in there.
Yeah, so the movie box is really the main part of the,
and before it contains everything
from the codec description to the samples,
like the offsets into the file,
the duration's and all that.
It doesn't matter what else you have in the file,
it just goes to the start of the data.
Yeah, like you can,
I guess I said,
they have some acceleration with the spec.
Technically, you could reference URLs
in different files as well,
and pull data from there.
It's just that I don't,
I think it doesn't implement it,
and I think GPAC doesn't even,
so you can't really use it.
They also find other fun things in the spec,
like compressed boxes,
because those things get quite hefty,
like four megabytes is not unheard of.
So they introduce compression,
but Chrome doesn't support it,
so it doesn't matter.
Okay.
Cool.
I'm happy to give you a bit of a longer break then.
I'm going to have a longer break then.