All right, let's talk at the open media room, guys.
I'm going to be talking about open source video mixing with real time control and also
most specifically syncing multiple videos.
I'm going to start with the project that this was done with this misserver.
It's source media in a certain way.
There is a shared memory page in them that has three metadata.
Now for each track for the audio track, the V8 track potential, multiple track, there is
another page in shared memory that stores information about the track.
And it has references to more pages, even more in shared memory that have roughly
a minute of media data each.
And what we do is that the right media also comes into one of these pages at some point
that fills up.
And then the other pages get deleted and we keep like a small stack of them going so
to speak.
This works great for media because usually you want to be able to go back a little bit
stuff like that.
And this lets you keep stuff in memory without needing to continuously rewrite stuff.
But it doesn't work so well for the raw video.
So for raw tracks to do something else, if you have a raw video track, we don't do all
those extra pages.
Instead, allocate a large ring buffer of raw pixel frames in the video metadata page
in shared memory.
And then we overwrite those in a loop.
This lets you do raw tracks without needing to continuously create files in shared memory,
which tends to get very inefficient at high bet rates.
So with that in place, we can now do encoding or decoding to a form of raw formats and keep
those in memory the same way we keep normal streams.
We have some ways to do this.
One of them is Ms. Program KVExec, which is literally the thing that lets you pipe a non-roll
stream a metlosca format into a binary virtual choice and it expects the raw format out
again in metlosca formats because you can actually send raw pixels of metlosca in case people
didn't know that.
We also have a version that effectively uses the top one to call FMPEG with the right parameters
and do that.
And then the last one, Ms. Prof. KV, is an integration with the FMPEG library, so a lip-a-v codec.
And it does it more directly so that you don't have to send things over a pipe.
That's the fastest one, but it requires compiling with the FMPEG libraries, which makes
distributions a little trickier.
We also support your reading the raw pixels directly from a webcam with a video for links.
So now that we have a streaming graph format, we can do fun things with it.
So we came up with a compositor process, which takes these raw video metadata buffers from
shared memory and will then read from them directly.
And the neat thing is that if one of the stream thing goes away, well, we simply make
the pointer invalid, but we can keep rendering and not really care.
If one of them falls behind, well, we just use the newest stream we have.
If one of them goes ahead, well, we can theoretically go back a little bit, but at some point
the ringbar for them's out and we'll get newer data and we want it, but at least we'll
have video.
It will place something so that's neat.
The compositor receives configuration in JSON format from the control process and then writes
it to another pixel buffer.
With speed and can do the opposite to turn it back into something useful, like S6X4,
ATPC, everyone, whatever.
And it lets us create cool little things like this with multiple feeds in a grid, but
I don't have to be in a grid.
You can put them in any kind of free form control.
This is what the next slide should be about.
Yeah.
We also have a fancy weapon to face.
I know it's a little small, sorry.
There are bigger versions if you install the software.
They're in the top you can select what sources you want, assuming that they're in
a well format, which this thing can do for you as well.
You can pick if you want a grid, you want a free style if you want, like one of those
old versions, security camera views, and then you can make it happen.
As a little trick, we also added in support for it.
It's hard to see here, but we can actually put titles in the videos, so you can actually
see what they're named or something like that.
You can put your own text in there, we too can open the main IBM phones to do that.
And the neat thing is that if you change this configuration in the weapon interface or using
JSON yourself at API calls, it updates and he'll time without anything going down.
It immediately moves on to the next one, which brings me to the next problem, that none of
that is in sync.
Like I said, it takes the last frame for everything, and everyone can take the last
frame for everything, but it doesn't mean that they are synchronized to each other.
If you have two cameras in the one thing or three or four cameras, there will be a slight
decync between them, because some cameras are faster, some cameras are slower.
Maybe one of them is coming over the network, and the others are not.
Maybe one is going over the different media protocol than the others, and it's not fun.
So how do you solve this?
Does anyone have any ideas?
No?
Okay, well, this is the solution.
This looks really stupid, it's going to this.
This is a QR code that basically holds a UTC time in milliseconds.
It's just a unique time.
Finally, encode is so that it becomes a small code as possible.
You can change that frame right at me, you can actually do this in a real time.
You can make this go faster, you can make it go slower, and each of these codes contains
the time code.
That's welcome to topics you start for debugging, you don't actually need it.
What we did is that if the compositor detects in one of the feeds this code, it knows
ah.
When I see a frame with the code in it, that is the time that that frame should be displayed
at or is intended to be displayed at.
It then looks at all the codes that we see in all of the feeds.
Takes the slowest of them, slows them all down to that pace, and boom, we're in sync.
So what does that look like?
Well, I have clearly recorded the demo video.
I would have loved to do it here in a cool time, but the setup alone would take 15 minutes,
so that wasn't going to happen.
So here is a little demo video that one of my colleagues recorded, that is sitting
down the back.
So this is Miss Surface Interface, for people who are familiar with Luke's familiar, we're
showing off here that we have a camera feed that it is decoding to raw, or I think that
it is going to be set up to be decoding to, yeah, we're here for adding a decoder.
And then the raw pixels are available for putting in the compositor, so here we have
a compositor set up currently with three feeds, and we're going to show off that's all
of them show the same thing, but it's going to be from angles, and that they're not
in sync.
He's going to a clef in his hand, and you'll see that they're all not in sync at all.
I mean, they're close, but they're not quite there.
So then the next step is we're going to show that code on all the feeds.
Yeah, he's quick showing off how that interface works, that's not really important, not
right now anyway.
So we're going to fill up the QR code generator, and we made that a progressive web app,
so you can use loader on your phone and install it as like, and a little template icon
that we just fill up.
It doesn't require internet, because it's all local.
So you show that to the feeds, and you'll see that at the top of every per debugging
in the phone, so at the top you can see some text, I hope it's readable probably, not because
it's tiny.
It's currently says it is guessing the time, based on the Seafetime.
And as soon as I've managed to scan that code, which, depending on your camera, might
take a while to get it not blurry, everyone has experience scanning QR codes, that's, but
once it becomes not blurry, there we go.
It has now synced it by QR code, so it knows that the time is seen.
You'll see that the clock that is running there is also synchronized to the clock that
we see on the phone itself.
And we do that in all three of them, and once all three of them have the time codes, then
they'll hopefully be in sync.
I'm not that it'll be, because it's pretty recorded.
And you can already, if you are able to see this, it's hard to tell because there's
nothing moving, those two are already in sync with each other now, and the third one
will join them as soon as the code is scanned.
Of course, you might not have the ability to get all your feeds in sync, and if you don't
scan a code and have the feeds, that only used to be latest frame best effort.
But now we have all of them, and now he's going to do the hand clap again, and we'll see
that it is much more in sync.
There we go.
I'll sync up.
And now, for bonus points, we're going to add a webcam to the multi-fueler, which will
heal time update it.
And here you can see some of the interfaces well, how you can position it, like with
a nice dragon drop thing, and you can still yep, and then go back to the feed, and it
will be right there, normally it doesn't take that long to load, but it actually shut down
because no one was watching it.
And you'll see that the other three are still synced, because they didn't forget their
time codes.
And now, my colleague is realizing he forgot to put the debug info on that one, so he's
going to turn the debug info on.
There we go.
And then once that is enabled, we can see that it's also in sync.
You can see it's already set sync by QR codes.
And now we're going to do one final hand clap with the fun fact that this webcam is terrible.
You'll see the frame rate is absolutely, absolutely, it's telequality and still, even though
the frame rate is wildly different from the others, it is synced pretty well, as well as
we can do with that low frame rate.
So yeah, that's the demo video.
When can you use this right now, two days ago, we released 3.10 of Mr. and it includes this
feature.
It's public domain code.
So anyone can use it, anyone can steal it, you can reuse our sync codes, we don't care.
It'll work.
This brings me to the answer, which is, are there any questions?
Yeah, I go ahead.
So you've shown us how you synchronize the clocks on the various media sources at one
question.
How do you deal with the clocks drifting up to that?
That's a good question.
The question is, we've shown how the clocks are synced but how do you do with drift afterwards?
Right now, we don't.
We found that in reality, most video sources tend to not drift that much.
It will be like maybe a few seconds per day, which for most purposes if you're doing
a broadcast of an hour or two, you will not notice noticeable drift.
If it does drift, you can simply show the QR code together and it will update its sync.
But it's not ideal, but it works.
So if you have like a particularly bad karma that tends to drift a lot, just show the
code to it after 30 minutes or something and you'll stay in sync.
Of course, you can also put it in the feed somewhere in a corner or something and then
normal mode.
Hopefully.
Yes, but the next time you're synced, you will have to be smaller than some of the streams
and all you have to synchronize the code together.
So the follow-up question is, want that, stop some of the schemes?
Yes.
What we actually do is we rewind all the other schemes if necessary immediately.
So you'll see some of them repeat the small section to go and think.
But it will keep going and the frame rate of the output of stay steady.
So that's pretty nice.
Anyone else?
Yes, go ahead.
Do you vote on some piranhas?
So if you think your code is worth it in the latest event, so are you going to sign this
order?
Ah, that's a good question.
The question is, are the QR code signed?
The answer is, no, not in this version.
We did think of that and the plan is to put a prefix in front of the code that you can
determine yourself and only if that prefix is visible, then it would work.
That's not a quite signing, of course.
The idea is that you would show these codes to the camera before the stream goes live.
So other people don't see your prefix.
And then theoretically, if someone is walking around with a sign that, you know, in 1970 or
something, then it will not ruin your life.
Of course, we could add signing in the future.
This is just V1, you know?
And if people say that this is going to be a problem, then yeah, we'll definitely add
signing to it.
And I bet that if this becomes a popular technique, we might have to.
Yeah, we don't think of that as well.
Anyone else?
Yes, go ahead.
Have you ever said that a sign can buy audio, which is coming like, obviously?
Yeah, we thought about thinking, sorry, let me repeat the question.
Have you considered thinking by audio?
Yes, we thought about that.
For now, we figured that video was the important thing to have in sync in this case.
We want to add more audio support to the composer as well.
Right now, we can copy and audio feed from one of the streams that we can, for example,
mix them together yet.
So if you have an audio and two of them, you want to mix them, you can do that yet.
That's coming, but not quite.
And at that point, I imagine it will become more important as well, especially if you have
like, are you only streams that you want to keep in sync with the video?
And we were thinking of putting, so some frequency signals in there that effectively
do the same thing as the QR code, but then with audio.
But if you have cool ideas, then share them.
Anyone else?
Yes, go ahead.
Are there a visible time that you are allowed to see?
What was that, sorry?
I already said on the amount of time that you are allowed to see, because if you are using
the memory somehow.
Hi, so is there a limit to how much these sync you can have?
Currently, we store, I think, from what I've had 200 frames for every row of it, we
figured that was a nice in between of not having two enough frames to sync within a reasonable
time, and also not having so many because you are not a wrap all the time.
That is configurable as a compile option.
So you have completely compiled the app if you want to change it, because that Vingberfer
was not made to be dynamically resized.
200 frames gives you depending on your frame rate a few seconds, which tends to be enough,
but if you're really far behind, then you'll need more.
What we're planning to maybe do is integrate the decoder more, because we could theoretically
tell it, hey, when decoding, fall back this many seconds before you decode it, and then the
Berfer can stay small, and we just slow down the decode a little bit.
But we haven't had to do that yet, we haven't found cameras that are more than three or four
seconds behind.
So hopefully we don't need to do that, but maybe in future.
Yeah, go ahead.
That is a good question.
We'll be able to detect, of course, if you see the code again, but other than that, it will
be pretty hard.
So there are some ways that you can detect whether a camera is in sync or not, like if
there's a big change in the light, you may have noticed that when the guy clapped in his
hands, we also turn to light on and off to make it more obvious where that point was.
That works great, but you can't rely on the time that the light changes intensity that
that means you want to check sync, right?
It could also be that maybe a lamp was aimed at a camera at some point, but not at the
other cameras.
So that's not reliable.
We would love to do something that checks on the fly continuously, but we have not
been able to come up with a liable way.
If you know one, let us know.
All right, number questions then?
Cool.
Well done.
What was the end of the talk?
I'm the end of the open media there for them.
I hope y'all have a good time.