Can you hear me?
Everybody can hear me?
So hi, I'm Anton Kirnoff.
I'm, or maybe, I was, and I've been back developer.
I work with FF Labs, or maybe I used to.
And past year, I implemented multi-view decoding in AD codec.
And in the FF and back CLI, Transcoder 2.
In this talk, I will tell you, what is multi-view?
Why might you care?
Why might you care even if you don't care about multi-view?
And some technical interesting aspects, hopefully interesting, of this work.
It was sponsored by Vimeo and Meta, so thanks for making this possible.
It has wide implications beyond just multi-view, but multi-view in itself is also quite interesting.
So, to start, what is multi-view?
I think the picture really says it all.
You have two or maybe more video streams that are kind of independent, but not really.
So, they are independent in the sense that you treat them as a two-parallel video streams,
but there's a lot of redundancy.
So, if you squint really, really closely, you probably can't see it from there,
but they are actually not the same.
They are actually different.
And the canonical example of multi-view is to your scopic 3D.
So, if this is the left-eye view, the right-eye view,
so, yeah, that's the way people generally use this,
but you can do other things with it.
So, now you want to code this thing.
The naive way you code two video streams.
This is very simple and obvious, but your bitrageous doubled.
So, you don't want that.
So, what you do want is to make use of the redundancy,
and somehow predicts one of the images from the other,
and just encode the differences.
You could use some kind of hacks like, well, maybe you put them side by side,
and use intra-frame prediction,
but, or you can interleave the frames,
and put them into one stream,
but these are possible.
People sometimes do them.
I think, but it's quite hacky,
and, for instance, it forces you to decode
always decode both of them,
which you don't always want.
Maybe you sometimes want just one of them.
So, multi-view is a set of tools to deal with that.
So, the thing I actually implemented is called NVHVC,
which is multi-view for the HVC codec,
also known as H265.
As you all know, H265 is a successor to ABC or H264,
which we all know and love, the best codec ever,
objectively true.
A and H264, they used to be a thing, which was called NVC,
which was multi-view coding for H264.
I think it was used in 3D blue rays,
and we had a longstanding feature request to implement that
an AD codec and that never happened.
For a bunch of reasons, which I will elaborate on later.
But so, yeah, it existed.
It was used a little bit in the wild,
but it's not really supported very much.
So, in HVC, there is a similar thing,
which we now call NV, which people call NVHVC.
And it is a way of doing exactly where I show you
in the previous slide of packing multiple semi-independent streams
into a single HVC bit streams such that the streams
can predict from each other, but otherwise you can sort of
tweak them as independent, which is exactly what you want.
It is based on multi-layer extensions.
So, I think in H264, all of this was multi-view,
was a separate thing, scalability, was a separate thing,
other dancing animated ponies.
It was a separate thing.
In HVC, I think that the unified,
there is sort of a general multi-layer extensions
specification that, and then it's specialized
into multi-view, scalable encoding, alpha,
things, some kind of depth, texture,
thingy, I didn't look into it.
But there's a bunch of things of purposes
that can be used for, but generally,
people care about multi-view, about alpha.
I think some people care about scalable, who knows.
If you remember what an all unit had
or looks like, which you should, there is a field in it,
which is always zero.
And if it's non-zero, you scream and run away,
and the point of this work is that you don't scream,
you don't run away, you face it,
you don't, and do something useful with it.
The first specification is insanely complex,
because all of these things, they can be used together.
And you can sort of have a multi-view, scalable stream
with alpha, which has up to 63 layers,
which one of them is the base one.
This is the layer ID zero, which can be
decoded on its own, by a decoder that
doesn't know anything about any multi-layer,
just ignores everything, and just decodes the base there.
But the other layers sort of predict from it,
and there can be a complex dependency graph.
And as far as I know nothing supports that,
even the reference implementation,
like there's the base one, which only does base layer,
and there's like three forks of it,
one of which does multi-view, and the other one
does scalable, and the other one does 3D,
and I'm not sure which one does alpha,
maybe somebody knows, and they are separate,
and none of them can do all of these at once.
But in principle, per specification,
you can do all of these together,
and if you look at this specification,
which I highly recommend, it's just completely insane.
So we decided to not support, of course,
any of that, we only support two layers,
with the second one, depending on the first.
Although, with alpha, being interesting for people,
maybe there will be a use case, where you have
a multi-view stream with alpha.
Somebody should create it, but I don't think
the demand for this is driven by VR, I think,
is so, probably has to be hardware for the does this,
so, probably not, but it will be fun.
But so far, we can do two layers,
and that's it.
Why do you care?
So, one possibility, you care about your copy 3D.
You have VR glasses, or Oculus Quest,
Apple Vision Pro, one of these things,
and you really like to record and watch videos on them.
So that's one possibility, that's the canonical use case.
You might care about alpha, so I didn't implement that,
but I opened the door to that, and somebody else,
already wrote the patches.
So, that will be possible soon, probably.
But more generally, multi-view, the coding,
the reason why it was never implemented for issues,
or one of the reasons, and why it was hard to implement for this,
is that it challenges a bunch of assumptions we make internally,
and also in the APIs about how video is decoded.
For instance, you have a single input packet,
like the coded HVC data that you sent to a decoder,
and that decodes, that contains all the views.
So, it decodes into multiple frames.
Two in our cases, but we don't make the assumptions in the API.
So, in principle, end frames.
So, that was not really supported in a bunch of ways before,
now it is, and that has implications,
so maybe it allows some things which were not possible for.
And the other thing is that now you have a single decoder,
which produces frames for several independent stream,
which, again, it has implications.
It might allow some use cases which were not possible before.
So, you might care even if you don't care about 3D.
So, what was hard about implemented?
So, first, inside the HVC decoder itself,
so AD codec has like generic, generic code,
which is the code independent, and then below that,
there is the decoder specific stuff.
So, this is for the decoder specific stuff.
The main thing that you encounter, or the first one,
is that a bunch of state that used to be per codec,
is now per layer.
So, you have your decoder context, which is a big struct,
which with a bunch of state in it,
and now a lot of that state is per layer.
So, you have, you need to have multiple of these context,
and for each layer we want to decode.
A common approach, I don't know about your project,
but a common approach that people very often do,
is that you add a bunch of children of,
or a bunch of copies of the same struct inside it,
which saves you, it seems like it saves you work.
So, it's because you don't have to really do anything,
you just do that very simple thing,
and from now on, some things are per codec,
and some are per layer.
This is a horrible, horrible, evil,
obfuscation method, which you should never ever do,
and if you do that, please stop.
Because immediately, when you do that,
you lose the information, which fields from the struct are,
are meaningful in the parent,
and which are meaningful in the child.
Now, everybody who's reading the code later has to reverse
the engineer, the struct, check all the places where
it's where some field is used,
and only then you discover, which is which.
And you might think, oh, but if I document it,
surely this will fix it, haha.
Of course, nobody ever documents things,
and if you do it, it will get out of date,
eventually, because somebody dazed the code
and doesn't update their documentation.
So documentation helps a little, not that much.
And it's, but also another problem is,
you have a bunch of dead fields.
In the parent context, and in the children,
you have a bunch of fields that are just there,
waste memory, waste cash, and don't do anything.
So, and in the end, the amount of work it saves you,
it's very little.
It looks like a lot, but not really,
and it's, it's work that's very straightforward.
You don't have to think about it.
In the future, probably, you charge a Pt will do it.
So, please never ever do this.
The thing you actually should do,
you check all the fields.
You find out which one are actually per layer,
which you have to do anyway in the end.
You just, in this approach, you just do it more systematically.
And then you add a per layer context.
You move the things in that per layer context,
one by one, and hopefully you're done.
If you work this was the majority of work by patch volume,
but it was really straightforward mostly.
If your code is really crappy and entangled and spaghetti-fied,
this must be not trivial, because moving one thing
can depend on some other thing,
which happened here to some extent,
but not as much as it could have.
For instance, the H640 coder has more history.
Let's say.
And doing the same thing there will be more complicated.
If you feel like second that problem would be prepared for some pain.
So, that was the biggest thing I had to do.
Another thing was frame output logic.
As you all know, H2HVC,
and also ADC they have frame reordering.
So, when you decode a frame, you don't output it immediately.
You put it in a decode a picture buffer,
and then maybe depending on some conditions.
You look at the picotic picture buffer.
You select some specific frame from it,
and then you maybe output it.
One factor that complicated is that there are things,
which are called sequences.
A sequence is basically a segment of coded video,
which has the same parameters.
Like a single coded video that was encoded in one at once,
for instance.
And this can change at a time.
So, you can concatenate a bunch of videos,
and you get two sequences, or multiple sequences.
And whenever you switch the sequence,
you have a bunch of frames buffered for output later.
And so, you could be decoding one frame,
but still be a frame for months sequence,
and still be outputting frames from a previous sequence,
or in more pathological cases,
you could be two sequences back, or 16 sequences back.
Probably not six, I think, 15 is the limit.
If you really want pain.
So, and we had a lot of complicated logic to handle that.
And actually, and what I had to do right now,
because now we have not only do we have to handle this,
but we also have two views.
And when you switch sequences, you can also switch the number of views.
You can switch from single view to multi view,
and back, or you can switch the positions of the views,
or the properties blah, blah, blah.
So, this had to be added on top of that logic,
which would be very complicated.
And because I'm not smart enough to think about all that,
I noticed that we actually don't have to do any of it.
But all that complicated logic is only there,
because we have the constraint that a single input packet
has to output at most one frame.
But we don't have that constraint anymore.
So, I change the logic to output multiple frames at once,
which we can do, and all of that horror goes away.
So, this work actually simplified a lot of things,
even though I now, it has to interleaf,
still interleaf frames from multiple views.
Now what it does is when it encounters a sequence switch,
it just flushes the decoder picture by for completely,
which we can do, which is great.
I can also notice frame threading is inefficient for multi view.
If you care, you might want to fix that.
That's welcome.
Now, moving a layer up in a decoder generic code,
there was also a bunch of bunch of issues.
We, as I said, we need a single input packet,
and we need it to produce multiple frames, which need to be output,
which is fine as far as public APIs concerned,
because since the new API, it's 10 years old at this point,
which was added by the infamous WM4.
Exactly for this, to handle M to N arbitrary packet to frame mapping.
So, on the public API level, this is fine, but internally,
internally frame threading did not support that.
A frame threading was working on the old API model.
So, it could only do one packet to add most one frame.
So, I had to change that.
I had to port frame threading to the new API, new, then your old.
Actually, I started doing that back in 2017 for my work on MVC,
which I never finished, but it was never really,
most of the work was most of the work in theory was done,
but actually polishing it was quite complicated,
because there was some unnamed decoder,
which abused frame threading quite a lot.
It did a bunch of things wrong.
It did that thing, which I told you not to do.
It did exactly this.
So, I had to reverse and generate and undo that.
It did that for slide threading, and just to make it readable and possible
to understand for myself, I had to fix that,
which incidentally made it 4% faster in single threaded decoding,
which I didn't intend, but yeah.
But also, it had some hacks like generic codec independent frame threading code
would have code like, if the codec is this, do something insane.
And by insane mean, reduced the number of threads by one.
So, if you had two threads, it was running single threading.
And there were also some races, and by thread sanitizer, and so on.
So, I had to do, in order to implement multi-view for HVC,
I had to fix this decoder, unfortunately,
or maybe fortunately, if you care about it, because now it's faster,
now it doesn't have any races.
It's faster in single thread, it's faster in frame threaded mode.
Now, frame threading is actually always faster than sliced threading,
which makes sense, it should always be faster, otherwise there's no reason to use it.
So, yep.
In order, one thing that helped me a lot here is this new API,
we have which is called draft struct, which was new.
It's a year old, I think, about now.
It was written by Andreas, thank you, Andreas.
It's great, it's recently became public, so you can use it.
It's an API for reference, counted structs with very little overhead,
and very little boiler plays on top of it.
So, it's very convenient.
So, I heavily recommend it, it's great.
So, I had to fix that, and then finish this batch,
and then frame threading is now finally able to handle multiple over frames per packet.
All that for just a small thing.
Another challenge or a bunch of challenges is the public API part.
As I said, the output part is not problematic,
because we do support multiple frames, multiple output frames per packet.
We did that for a long time.
I think many colors actually don't get that right.
An example recently that assumes that one packet produces at most one frame.
So, all these colors are broken, but that's their problem, unfortunately.
But it's not ours.
The problems we do have actually is that all the multi-layer properties are per sequence.
So, if this never happens, but it had to be implemented properly, of course.
So, in principle, you have to consider the case where a multi-view video is concatenated with a single view one,
and so, or you can have a multi-view video with different properties.
So, you need to tell the color, what view ideas there are, what view positions there are, which view is right or left,
and this can change dynamically.
So, for that, I use the GetForex callback, which is named that way for historical reasons.
It actually is used currently to configure hardware acceleration,
but now it's also used to negotiate multi-view properties with the color.
So, when that color bike is cold, the color gets the information about the stream,
and can tell us that it wants either one or both views to be decoded,
but in principle, as many as there are, which can be up to 60 feet, or in the API there's no limit.
In Max.
Also, I added every type of options, because we want to export multiple view ideas,
and multiple view positions, and previously we didn't have a rate-type option.
So, what was done was we communicate by using comma-separated strings,
and parsing strings in C is great fun, everybody loves it,
but because I hate fun, I took that away from you.
So, yeah.
And this will also be used heavily in other places, like in any filter we do that,
and we do it everywhere all the time.
And the frames that are produced by the decoder have side data,
which tells you which view it is.
So, that's quite simple, and then the color can deal with it as it likes.
I will skip that, because I'm actually going very slowly.
That was a repeat of my last year's talk, which you can look up.
There is native support for multiview, and the CLI, and the Tonscoder tool.
It's not originally intended to just, well, have the codec output all the frames in to leave,
and then let the user deal with it, which was completely painful, because the users don't know anything.
And so, because I do, I extend it three specifiers, which everybody loves,
into view specifiers. So, now, before you could say, well, I want to decode the fourth video stream.
Now, you can say, I want the left view of the fourth video stream, and you can pipe that to an output stream,
or to a complex filter graph, typically you want to put the frames side by side,
or, or marks them into different streams, or different files, or whatever.
So, that's up to you. That can be done with, with the new view specifiers.
One feature, people might, might be interested in.
So, now, a single decoder in the CLI can produce multiple streams.
It is, it is technically possible.
This could be generalized, for instance, to support closed captions, and other features like that.
You could have a video stream, which has embedded closed captions.
And before, we supported extracting them by using insane hacks like, you use the Aby filter device pseudo-demuxer,
which uses a movie video source, which opens a file, and somehow decodes the video internally inside the filter graph,
and extracts the video stream and the closed captions, and then gives it back to you as a demuxer,
which sort of works, but just know.
And now, the CLI can, in principle, it's not implemented, but it could, it could in principle be done straight forwardly.
And other things like that.
Okay, I think I'm done. Thank you.
Yes, Kevin?
The splitting out closed captions notion does, wouldn't that mean that a decoder?
You know, right now we have video decoders, audio decoders, and some type of decoders.
You don't have a decoder that can produce both video and some type of data.
And not on Abycodec level, on Abycodec level, I imagine it would give you a frame with side data.
Oh, sorry. I have to repeat the question.
Could you repeat the question, sir?
The question was, you know, if that back today, the coders are typically categorized as video audio.
Right, right, right.
So the question is, in Abycodec, the decoder is a video or audio or a subtitle decoder.
How do we make it out with closed captions?
And the answer is, we don't do that in Abycodec.
We do that in the CLI. That's my point.
Right, so a decoder remains a video decoder.
It gives you a frame, and the frame has side data with closed captions.
And then the CLI pretends that it's actually two streams.
It's a video stream and a subtitle stream.
The decoder is actually producing multiple up and stream.
The decoder decoder isn't. The CLI decoder object is.
Those are different things.
Are the questions?
Yep.
I'm wondering if you plan to expand this multi-use report to DVD angles.
What about that?
So the question is, am I planning to extend this to DVD angles?
I don't think it's...
I am not sure how DVD angles actually work,
so I am not sure that this would be applicable to them.
We do have a lot of activity on DVD demuxing right now.
Maybe you should ask the author of that code.
Sorry.
Have the questions?
Yeah, Victoria.
Can we do internalized in MVHVC?
Sadly, the windows do not open.
Yep.
So the question was, how do we accelerate with MVHVC?
I don't know.
I didn't try.
Yeah, there's an encoder set at least on this curve thing.
So enabling with the MVH module to encode already into the content.
For decoding, in principle, I think the low-level hardware
doesn't really care about multi-view.
It only gets the reference frames.
So the difference in decoding in the actual pixel level
or macro-block level decoding is that you have more frames in your reference
pictures, lists, or set, or whatever it is.
So if the high-level code just adds one more frame in that,
the hardware doesn't care in theory.
I didn't try.
But I would hope that it can work.
Yep.
I was wondering what kind of requirements
for milk to view?
Does two inputs need to be the same size?
The same aspect in the late, you know,
are they always left the right or can they be up and down
or in the corner?
So the question is, if there are restrictions on dimensions
and aspect ratio and positions, so the answer is kind of.
So positions are just metadata, right?
So it's just a field.
It's actually optional.
You don't need to have a screen that doesn't tell you what the position is.
Actually, the spec doesn't mandate that it has to be somehow oriented.
That it has to be really to eyes.
It could be anything.
The interpretation is really in some metadata, which may or may not be present.
I think the allowed positions are left right and top bottom.
I don't really remember if it could have more complex orientations.
For formats and resolutions and aspect ratios, in principle,
they don't have to match.
And actually there is this spec allows you to have different size of the different views.
But we don't support that, because that is not same.
More questions?
Thank you, then.