I'm happy to be the first one here, and thank you for being awake that early on the Sunday.
My name is Roman Buxis, and I'm going to present today a couple of ideas I put together
after working on managing dynamic content for an automation streaming software that I developed
with Samuel here years ago.
Leave it up myself, I leave it in the audience, I've been a software engineer for over 10 years,
maybe 15 soon, and one of the things I love when I do software is to think about navigating
between the higher level, the clean abstraction we can create as software engineers,
and what goes on under the hood, where we actually deal with a lot of complex problems
that we know are to deal inside of bugs and present something outside of the bugs that
is neat and very usable for people and solves those problems internally.
I'll give some example of that.
What I'm trying to do today is after having dealt with the hard problems looking back and
say, what can we do that simplifies it that would be useful for other people?
So a little bit of the context here is liquid soap, it's the software that we have, it's
been created around 2004 by Samuel and David Baal, where students at the ENS, the theoretical
computer school, it's a language for media streaming, it's functional, statically typed
within for a type, there was all the hype, but we're not here to talk about that today,
and since then, because that's been a while, it expanded to a free feature language with
a free type integration with FFMPEG, and that's the integration that triggered what
we are going to talk today about, especially because originally it was audio and now it's
doing audio and video mixing and streaming, which brought another level of complexity in
terms of the kind of content that we handle, and the kind of questions we have to answer
when we create stuff.
So I'm going to look at a script in liquid soap, that's where we use, that's the way we
use to do it, take a minute to read about it, but essentially you create a playlist of
audio files, you're putting a jingle, a file here that's used when you're the playlist
is empty, we're extracting the audio, the metadata, and the track marks, we're taking
a video of a playlist of videos, we're extracting the video track, we're remixing here
audio video and metadata from each source, we're adding a little text on it, and we're
exporting to HLS re encoding.
That's the way we use to do it, where everything is decoded to a role content, being PCM
audio, you for 20 video, something that you can actually manipulate and add data to it.
But one of the layer we've added since then, since the first time we came to present here
in 2020, is that now we're able to do something very interesting, which is instead of re encoding,
we have a copy and color that takes the role binary and coded data straight up of the files,
basically informs every single decoder here that they need to not decode and just take the
packets of encoding content and repackage it for HLS, and it looks minimal in terms of scripting
but in terms of CPU and machine consumption, it's day and night, because you can have a 4K video
here that's being processed and re encoding a 4K video is vastly different from just repackaging
the encoding video.
And so this is where the complexity arise, how you transfer what we do in streaming to the
world of directly encoded data.
Whoops.
Yeah, so liquid soap can schedule how the place can support life source DJ that connect at any
given time, it can have multiple output, and basically what it means is that it can handle
dynamic and coded content and at any time you might switch between the stream and another
one, you might change your audio, you might change your video, and you still want to be able
to compose all these content and create a stream that's readable and compliant with
the format that you're exporting to.
And that's what we're trying to solve here and I'm going to explain how we did it.
It's related to other needs, I think that it's not just us and that's why I want to step
back and say we've done it, but how can we help other people do it?
Like in FFMPEG, if you read a RTP stream that's coming from an external encoder and
the encoder starts, currently you're not going to get something very successful because
FFMPEG can do some limited content, concatenation, but you cannot really do it in a dynamic
way where you suddenly change the source in the middle of the encoding.
And so I'm hoping that by extracting some of the knowledge from here, we can create maybe
a library of functions that can be reusable outside of all project itself.
So I'm going to do a quick demo and let's hope that it works because that's demo for you.
So I have the script, let me put that here.
I have the script from, and that's not going to work, I don't know.
I have the scripts from the, oh dear, I have the script from the last time, no demo,
do it another time, sorry, it's just the two screenings to confusing, I'll do the demo
again.
Back.
So if we don't do a demo, I'm going to explain what is streaming, how you create a streaming
loop.
What you need to do is that you take a frame duration of 0.02 seconds, you start at a certain
time, you use a clock to measure time, it can be your CPU, but if you're using other
things like SRT input, the clock synchronization is going to be given by SRT.
If you're using your sound card, it's going to control your latency, I'm not going to
get into that, let's not fit today.
But for today, we take a unique get time of day, we measure the initial time, then we run
our program, now program is going to generate 0.2 seconds of content, send it to all
it put and say, done, that's our ending time, and then we have two scenario, if the
lapse of time that we have used to create and not put all that is less than 0.2 seconds,
we are good, we can slip a little bit, we come back later, and we can create content in real
time.
If we have done more time than that, it's going to be a problem, because we're late,
maybe we can catch up, maybe we're temporary, but most likely it's not, your system
is not able to handle streaming.
But that's how you run a streaming loop essentially, it's another complexity under the
hood, but on the higher level, that's the AD.
So in the liquids of the way we used to do it is that we used to have a static frame of 0.2,
0.0 to second of audio PCM data, so think of these bars as little zero loading points
PCM audio samples, and we used to partially feel that we would pass it down to every single
operator that creates data, like a file decoder, which fill up your frame, pass it on
to something that composes it, all the way to an output would take the samples that
had been generated and output it, and the way we would do it is this way is because this
way we didn't have to relocate content on every cycle, we just have one static frame
that we pass around every cycle, and we would mark how much of the frame had been filled
up, like you can see.
But that was not a very useful format for content when we started adding video frame, because
we want things to be synchronous, so we were constrained to have one frame, one video frame
per content frame, which forced us to put first of all to do a sampling rate in audio,
that was compatible with the frame rate, so that we could have one single check of data, that
was on the same boundary level, and we would assume a fixed video rate, which is not
the case, and a lot of content that are encoded, and it would force us to have like 0.0
for second, because of that, it was not a convenient way of representing content when we
started adding video.
And so what we did in a recent version is that we started thinking of content in a different
way, which is that you have an empty frame and you want to fill it up, and so the intuition
of filling up is that you're going to add a chunk of video content to it, and it's not
filled up yet, so you run it again, you finish with another chunk of content, you do the
same with audio, and at the end of the day, you say boom, right here, that's my 0.0
to second of frame, and so instead of passing around a pre-allocated frame of PCM array,
we pass an empty list, and we just dump content into it until it's filled up enough, and
what we want to do also is that if we have a little bit too much, we slice it up right
here, we keep that in our buffer for the next cycle, and we pass that to the output to
an code, and that's so much more rational way of thinking about it.
But now the question that arises, how can you programmatically do that, and that's what we
can't talk about it, because what I want to introduce here is this idea of unified content
composition API.
Can we have an abstract API to compose content, to consider what is a chunk of content?
What is inside your chunk of content?
How do you take a chunk of content, and another one, and concatenate together, that's your
first operation, and the second operation is how do you take a chunk of content and slice
it up, because if you remember, here we want to slice up right here, so what we're going
to explore is, can we have an API that is working with both decoded and coded, so
the content that is still a row MP3 data, a row H264 data, can we create abstract chunk
of content that can be concatenated and sliced?
And we want to encapsulate the system where, from a user perspective, I'm going to say,
oh, I have a chunk of content, concatenate those two or slice this, and I don't want to really
know what goes on under the hood.
Yeah, so let's go.
So the first thing we need is content elements, so what are our content elements?
So, and as something that I want to take a minute to think about, because the audio content
is vastly different from video content, and you don't manipulate them the same way.
Typically, if you think about decoded data, an audio content would be a PCM array, which
is here five samples of audio floats that represents digitized point on a curve that's
like a sign or frequency based.
But for video, you have just a single video image is your base content, an Azure is vastly
different.
In video, you can do nearest image resampling, most of the time it's going to work because
of written-in persistence.
With the audio, no, like you can't just manipulate elementary samples.
They just need to come together with the significant enough amount of data.
Welcome back to that, because the same problem arises in anchored format.
So let's go into it and say, what is a packet?
So I'm going to use the terminology of FFM peg here, which says the frame usually talks
about decoded content.
Packet is anchoded content.
Packet is anchoded content, meaning it is going to be a natural boundaries for it.
So for instance, if you have MP3 data, there's a natural notion of MP3 frames.
It's in the spec of MP3 data, and so that's like a fixed amount, I think, 10, 44 samples
of audio.
That just gives you a chunk of the anchoded data.
For all you have a notion of packet, that's the packetization in the mixing system.
For video, most videos format will say, one chunk is a frame, which is an image, but they
have different nature.
If you're from it, now with it, you have a knife frame.
We can be decoded separately of a p-frame, we need to have a knife frame to be decoded.
But what's FFM peg does here is that it abstracts away, what is the packet and it just
tells you?
Here's a packet.
That's just a opaque and ambreakable amount of data.
It holds a little piece of your anchoded data, but you can't really look into it.
You can't really break it up into a smaller chunk.
So the second thing we need is to create a chunk.
This is our timeline, and here at this time in point is two ways of describing it.
If you would decoded content, you could add a p-ts, that's the time at which you need
to be presented to the people watching the video.
If you're an anchored world, it's the decoded timestamp, that's the time it needs to be
given to the decoder, and we'll come back to that.
You have a duration for your chunk of data, and that gives you right there your elementary
first chunk of data.
That's the most simple element.
You have a frame, it has a sort of duration.
Boom.
You have a little tiny bit of content.
You have a packet, it has like an MP3 frame, it has 10-44 samples of audio, 44,
k, rate, boom, that's a little bit of a amount of data.
You have one image, it has a little duration.
Now you want to compose your stuff.
You have two chunks here, and because you know, remember that we have duration for all these
things, so you imagine that those are no longer samples, right there like frame, packets, little
elementary.
We have a duration for it, and we have a last pts or dts, a last timestamp for that.
So what we can do is look at the other chunk here, that's the first timestamp, and so
we can basically say the last timestamp is going to be the first timestamp of the first
chunk, the first content element in this chunk.
And then one by one we can adjust based on the duration, which would be the timestamp
of number two, number three, and now it's going to flow, right, because the first timestamp
of the first content element here can be placed right here, and we have a concatenation,
that's simple, and this will work at the mixing level.
Now the second problem we have is about slicing content.
Remember I was saying, we want to slice this, we want to be able to divide this content
into a smaller chunk.
So the way we're going to do it is we're going to take, again, a chunk of content, five packets,
five frames, and we're going to say, well, here's an abstract offset, and here's an abstract
duration, and we're not going to do anything.
We're just going to say our content is three elements, the row content, an array, or
list of the data that's inside, and just an abstract offset and duration.
And then if we have another one that's like here, another content, two offset and duration,
we can say now, I would just want to say they are placed next to each other as a composition.
And what we're going to do is once we send that to, once we compose our final content
chunk, we're going to extract the content elements that are relevant within the boundaries
of our offset and duration.
So when I want to create the final larger chunk, I'm going to take everything that's in
the yellow and orange area, adjust the timestamps as menu content.
That is on the surface, what we've done.
We've implemented a notion of content in Nuke, so that extracts chunks of data from the decoders
that are packets from a fifth-impag or frame from a fifth-impag.
They are placed into arrays.
When we want to do a smaller one, we just adjust the offset and duration.
If I want to concatenate, we just list them together, and eventually the encoder, the output
is in charge of recomposing that content description to generate the data that's going
to flow out.
It works, but it doesn't work in all situations, because, as I said, the beauty of an engineer,
good engineering work is about having that exposed to the user, but internally being like,
it's a bit more complex, I'm just not showing you that.
If you think about it, that's already what F-impag does, because F-impag gives you an API that
says, get a stream frame, returns a packet, you don't know what the packet is.
Decode packet gets a frame, you don't know what the frame is, send a frame to a Mixer,
you don't know what the Mixer does.
That's what we're trying to do here, is the same IDs that under the hood, F-impag knows
that if your Mixer is an MP4 container or an MPXTS, then they're going to do different things,
but you don't need to know, and so what I've presented here is really nice, it's elegant,
it works, but the other things that I'm hoping we can solve that the user doesn't need to know
that are pretty complex, so this is what we're going to talk about now, is the hard bits.
First of all, let's talk about what we just saw, which is slicing off a smaller chunk
of content based on a larger one.
It's not always possible, it's not always possible because you have limitations, chunk
have a minimal size, so if you take one sample of PCM audio, its duration is one over the
sample rate, and you can't really slice it to less than that, so here typically if my
chunk was PCM audio, I would be incorrect to assume that, because I can't really take enough
set that in between samples, it doesn't make sense, my digitization level is, my granularity
is at minimum one over a sample rate, you have the same problem with video frame, a single
image is one over 25, one over 30, whatever is your frame rate, you can't really slice it less than
that, and then for I could it packets, it's even more complex than that, because we don't
really know what's inside, it just tells you that, so packet, you can't really slice it.
So here's one thing we've done here, that I'm going to show some of the ideas we have,
I'm going to expose other problems, and then we can talk about it later, so one of the solutions
we found for the PCM audio, if you think about it in your system, you're going to have audio
and video that flows inside internally, and every computer system is, is internally,
digitized in a specific sample rate, there's no continuous, so your application is, it's on
master rate that is then converted into audio rate, and most of the time the audio rate is much higher
than the video rate, it's like 44 points, you know, it's 44K sample per second, 25 frames per second,
so what you can say is just set your applications main rate to be the same as the audio rate,
internally you flow everything in 48K, your application handles everything at 48K, and that means that
your offset is always going to be on the sample boundary, you can't really mess it up, that's a solution,
in videos you can do the nearest image resampling, as I said, original persistence works,
and for anchored packet, it might work for some format, you might be able to do something,
some other you can, and so typically for image, I say that an anchored frame can be a single image,
so if it's a single image, you can always change its location in the timeline, you change its timestamps,
written in persistent, make it possible to put it at any of set you want, so that's going to work,
the other thing you have is that if you think about the fact that that's an example of content
that is not at the exact boundaries, then you have you're going to have a little bit of delta in
each case, but what you could do when you do your APIs is compose your content and keep a delta
that's within acceptable boundaries for instance, granting that the average delta in your, you know,
little boundaries that we're not exactly where you want to slice, it's still on the average zero,
or within one of the accepted delta between audio and video, so those are different standards
depending on Europe and the US, but how much the audio can be laid compared to the video.
So if you keep that within the boundaries, you can still have a very high level API
that allows for content composition and hides a little bit of this complexity that you can,
you cannot actually slice properly, but overall it's going to function as if you had done it.
So the last bit that I want to show before we finish is, of course, I've saved the most important
complexity for the end. This is a timeline of video frame, but in anchored video, usually you have
a knife frame, which is a frame that can be decoded separately. You take this frame, you decoded,
you get an image back, and then usually you have frames after that that needs the first eye frame
to be decoded. And you might as well have frame before that, who also need to be reference,
referencing the eye frame after them to be decoded. So that's the timeline in presentation time
stamp, that's what you want to see in the final video. But that means that the timeline in the
decoder way is going to be way different, and it's going to be completely jumbled, because in
terms of the decoding, you need to give the eye frame first to the decoder, so that he has it,
and then you give the first blue frame that I'll be for, so that they can be decoded first,
then you give the frame after, so that they can be decoded after. And so that means that your
timeline in terms of anchored content is going to look very different than your timeline in terms
of decoding content. And so if you don't pay attention to that detail, and you say, I'm going to take
a chunk of content that's right here, and just ignore the eye frame, and take that's my chunk of
content, it has the two blue frame to green frame, well, in and off itself is not going to be
decodable, because you don't have the eye frame that refers to it. And so one of the problems we
have to solve is that if you're handling dynamic content, and you're composing from two different
stream, you have to ask yourself, like, is this stream the same as the previous one? Then in this
case, I can probably concatenate whatever I want, because I'm assuming that whatever I frame was
required to decode that chunk of content was already given to the decoder in the previous one.
But if you're switching streams from a different source of anchored data, and you want to
compose the content, and you're going to have to have a limitation that says, I want the first
chunk that comes out of the new stream to start with a knife frame, so that we don't have a glitch
to the user. Or you can want a glitch, but that's a presentation for later. So that's another
problem that you have to solve when you want to create a nice composition algebra.
And there are more questions that we haven't solved. Can you compose, so again, those goes to
problems that are very low level and dependent on your mixer and could it format.
For instance, if you look at H264 and MP4, or MPXTS, can you compose, can you compose two
stream that have different frame size? Can I change the frame size dynamically? Can you change the
pixel format? You've for 20 years, you've for 22, it's part of the standard, it's not part of the
standard. Can you change the audio sample format? Another problem that you have is what in FFM
ping is called extra data. So basically, when you have anchored content, you need certain
data to be able to decode like estimate tables for compression, and depending on of your container
in MP4, this is going to be in a global header at the beginning of the file. In MPXTS,
it's going to be in every frame. So if I'm composing content dynamically, I need to be able to
detect that and say, no, no, no, no, no, you need to convert your H264 from global to local data,
so that you can compose dynamically and be able to decode. So typically, I have FM ping as what they call
bit stream filters that can be used to manipulate that extra data and go from a global header format
to a local frame format. Again, I'm hoping that if you write an extra API that does these two
things, co-contination and slicing, it can also detect those things and do it for you. The same
way that when you compose content with the FFM ping API, it's able to do it on a file like if you
write an MP4 file, it's going to do a global header, if you write MPXTS, it's going to do it for you.
Yeah, and you know, the other thing is that it's not always going to work, so what are the
optimal parameters? How many I frame do you need? Those are like the hard questions.
So yes, I'm finished now, and I think we're going to forego the demo, I wish I could have done it,
but to recap what I was trying to present is if we want to manipulate content, and especially
manipulate, well, I encoded content, binary content, can we create an API that allows
to consider your content a little bit of pieces that you can slice up, patch together, send to an
output, and create a stream that is a valid stream without having to re-encode everything,
and also abstracting the complexity. In the demo I wanted to show, it was a 4K video,
well, a bunch of 4K files and some audio, and basically I click on something button,
and it switches the file on the main, and the content keeps being decoded without any glitch.
And I think that there's a lot of value to that, I think that a lot of the
computing power that's required to re-encode to make that work in most application could be saved,
if we can work on specific expert knowledge about all these binary formats and export that
in a very neat API that would solve the problem for the people. Thank you. So, yeah, maybe there
are remarks or questions.
Yes. In your experience, have downstream receivers with going to accept the level of
AV synchronization that your innovation of the solution? In our current solution,
so what we're solving at the moment is a very simple subset, meaning that it works if your system
has been preemptively designed to work properly. So, I haven't done a lot of testing in
specific situations where the AV gets, like the idea of maintaining an average AV disynchronization,
we don't have it yet. It's ideas for future development.
Well, if I have a couple of minutes, I can try that demo. The problem I was having is that the screen
is not screen screen screen screen, or is it? You know, display display. There you go.
All right, arranged? No. Now, the mirror, mirror. Is that, can you see? What I'm seeing? Yes.
All right, so let's try the demo here. This is my script in your quitsup. I'm going to start it.
It's going to create a HLS stream. Yep, the files are studied. So, the music is mine,
and the video is free. So, we're going to look at the term it interface, and we have two
common that says audio next. What's the next audio file and audio skip? So, I'm going to skip audio
two times. All right, and now I'm going to do the same with video. So, that's the next video is
going to be number one. Then number four. All right, and now let's listen to the result.
That's the video we switch. Two times, and now let's do the audio, and now let's do the video.
Ah, wait, it's a 22nd loop. There you go. So, yes, what we're seeing is that we're switching
seamlessly from the different files and they're all 4k. I'm not re-encoding them, but the content is
about is we're able to do that streaming loop where we do 0.2 seconds of it. We slice them up,
we compose them, and there's no glitch, because the iPhone is properly placed first. All right,
I hope that makes more sense now, and I want to thank you, everyone, for being here today.
Oh, sorry. What do you mean exactly?
No, again, and I am not doing a lot on the video side yet. It's other side. I should have
made more clear at the beginning. Liquid slope is more high level. So, I'm not myself an expert
in all the different binary content. I have been working straight up with the, what a fifth
MPG, it's me as an API. And part of the reason I'm giving that talk to be honest with you is
because I'm hoping that more expert knowledge in the binary formats, in the different encoders,
in the different mixer, could help inform that. The set of questions I had at the end about
about things like, you know, when I was saying this, I don't have the answers to that.
I'm actually seeking them myself, like, can you change the frame size, can you shade a pixel
for my, you know, what are the limitations? We have solved the problem in a very specific case,
we're very strict. When we compose, we want the XI same codec, the exact same frame size,
the exact same pixel. And it works if you tested it basically. But thank you for your question.
And then we, yes, absolutely. Yeah, I would love to learn more, absolutely.
All right, I think that I'm done. All right, yep, times up. Thanks everyone. You're all have a good day.