Hi, good morning.
Welcome to my G-streamer talk.
First of all, hi, I'm Tim, I work on G-streamer.
G-streamer, you might ask.
What is G-streamer?
It's a multimedia framework.
It's cross-platform open source, free software.
It does anything you would expect from a multimedia framework.
It does audio processing, video processing, subtitles, captions,
metadata, and you can do exciting stuff with it.
And you can chunk it, stop it around.
You can edit it.
You can decode it and code it.
You can stream it, read it from file, write it to a file.
Mixer, do apply effects.
Whatever your heart desires, basically.
Run analytics, that sort of thing.
All right, so what have we been up to?
I've got way too many slides, by the way.
This is more like the speed dating equivalent of the talk.
Yeah, we released a new major feature release, 1.28.
We have one of those every nine to 12 months.
Yeah, and this one is brand new.
Was released just a couple of days ago.
Release notes are on the website.
If you haven't read them yet, you're not the only ones.
But you can feel loads of features.
And I'm going to work through some of the bits we've been working on.
There's way too much stuff.
And I should also say, I'm keeping it very, very high-level,
just sort of to wet your appetite to see if there's something that's of interest
and then you can dive into the race notes.
Or you can check out the Gstreamer Conference talks.
The videos are all online, and you have lots of great talks about all of these topics here
in much, much more detail.
Yeah, number one codex.
We've got support from our codex.
H266.
There's a latest impact codec.
Then we've got enhancement codex.
LCEVC, sort of like a multi-layer codec thing.
We've got support for that.
We used to have H264 already.
And now we've got enhancement layers.
In combination with H265 and H266 as well.
And we can decode and code that.
An impact H audio is a new audio codec from the front,
over people.
And we've got support for that.
So we're now plus maxing and demuxing into the MP4 container.
There has been a whole bunch of work around and salary metadata.
So that can be captions, sort of in broadcasting captions.
That is the bottom of your TV.
But it can also be generic metadata.
All kinds of metadata is sent around.
That's sort of quite important in broadcasting workflows.
So if you watch a football game or something,
you might have, for example, the game score as a metadata
in the stream and things like that.
You can put anything in there.
And we've got whole infrastructure around that.
So you can attach it to buffers in the pipeline.
You can split it out.
And you can write it into the MXF container,
which is used for film and reproduction workflows.
You can capture it on sort of decling capture cards.
And you can send it out again.
And you can send it over RTP with a new payload.
That's in the payload as we have.
RTP is sort of, it's a really old protocol.
It's kind of the workhorse for, you know,
ingesting a stream into it like a streaming server.
Unfortunately, it just doesn't go away.
It should.
But it doesn't, everyone uses it.
So the container format that's, that's used underneath is FIV.
And it is really bad.
And old, but, you know, as I said, it sticks around.
So, yeah, but people, you know,
it only has a very limited range of codex support.
But people want to continue using it.
So, you know, there's like an enhanced RTP as a V spec now.
So you can actually put more than video codex into FIV.
Like, ASUS X5 and everyone.
And you can actually have more than one audio track.
You can have multiple audio tracks and multiple video tracks.
So it's all very exciting for those of you who still use RTP.
And we support that industry one now.
Or, I should say, I think the AV1 support
is not actually merged yet.
I think that's in, in GitHub.
But it is a rebase on top of the, um, the latest code.
You can create HLS streams with the streamer.
Basically, just, you know, write stuff into FIV directory.
FIVs and playlists.
And you can point your web server to it.
And just, you know, server to web browser.
Yeah, we've got support for single media file.
I'll put now in, which is quite, you know, quite useful.
If you just create a stream for video and demand.
It's not a live stream.
And then you don't have to manage like, you know,
a million little stream fragments files.
But, you know, you can just have one file.
And the playlists will just sort of point into the, um,
the chunks of the file, which is much, much easier to handle.
And you can also buy it.
I frame only playlists for, um,
seeking and, you know, like, like,
trick modes, fast forward, fast reverse playback.
Thanks like that.
There's a lot of work going on around, um,
speech to text, um,
conversion text translations into different language.
And also, uh, speech synthesis.
So you can, you know, you can pick up the,
the, the audio converted to text translated into different language.
And then have that generated, again, into audio,
and you'll feed it out.
Uh, you brought it into a file or stream it out,
basically.
So then you've got to translate it, uh, audio track.
Um, you can do exciting things without,
well, exciting and slightly scaring.
But you can do things like voice cloning.
So you can clone, you know,
voice, and then translate it, and then he speaks Spanish.
And it sounds like to improve just speaking for, like, Spanish.
Um, yeah.
So we support lots of stuff, um, in that area,
and it's unhappy development.
And also cool stuff, like, you know, if you translate things,
and often it takes more time for the translated language,
that it takes in the original, right?
And you can't have a problem because you go out of sync,
or you kind of have to drop stuff somewhere.
So we can, we can speed up the audio if you,
you detect, like, it goes out of sync because the translated language
just takes longer.
Um, then you can speed it up and catch us up and you,
you know, you get back into sync again.
It sounds a bit funny, but, you know, works.
Um, yes.
Also, if you enter karaoke, um,
we also have an audio source separation,
and then it's based on DMUX.
It's like a library engine, um,
that's often used for that sort of thing.
Um, that's used for the karaoke,
but also if you have, like, a, you know,
the stream where you want to filter out the background audio
on you just want the voice right so you can then run speech detection
on that voice and translate it.
So it helps with that as well.
Um, and then it takes, you might have, um,
being to Daniel's talk yesterday in the, um,
of media dev room.
Um, yeah, we've got a lot of support for, you know,
object detection, inference, all of that.
We've got, yeah, look, I mean, that doesn't area
that's being worked on quite a lot.
We've got more inferencing elements.
We've got more support in, in on X in the runtime.
More matters, um,
tenser decoderers, uh, they can be auto plugged now as well.
And we have a batch meta, um,
plus a splitter and combined elements for that.
That existed already because, I mean,
with the analytics stuff, everyone, like used to do their own thing.
You know, Nvidia has, have their own thing,
Intel has their own thing.
Um, and as I'll have their own thing.
So, you know, the hope is sort of to provide basic,
infrastructure upstream, and then maybe in 10, 20 years,
you know, the other people can move towards that as well.
I hope that's last.
Um, well, can, well, can, is like the latest and greatest sort of
low-level video API, um, for rendering,
but it's also got, well, in video, um, which can be used
for hardware accelerated decoding and encoding.
Um, and yeah, we've got support for more decoders,
everyone in the combined particular, um,
H64 encoders, um, and the H65 decoder can do
HDR now.
Let's 10 bit decoding.
GTQML integration, um, but security took it, of course.
We've got various elements for that.
We've got QML five elements for that as well,
but QT6 is what we're working on at the moment.
So there's a new render source element,
so you can, you know, render a scene and have that, um,
be captured into the pipeline as a video, basically.
Um, and there's a GL overlay, so you can overlay something on top of
a video using the, you know, GL rendering, basically.
And you can, now, not only have like a scene-rath description,
but you can actually put a QT quick item in there,
which gives you more flexibility.
Um,
Weirland is basically what you render to on a modern Linux desktop,
hopefully.
Um,
yeah, so the colour, um, there's been a lot of work on the colour management side of things,
um, HDR support, although that, um,
very exciting also a new full screen output property.
Additionally, we haven't added these because we, you know,
think it should be done via the, um, API, but we've relented.
So there you have it.
Um, there's something called UDMA Buff support,
basically the, you know, way how you send video buffers,
very efficiently between, um, subsystems of the kernel.
Um, and UDMA Buff is basically like a user map,
a little variant of that.
And that's quite useful if you have, you software decoding,
like you might need to do AV1 software decoding,
if you don't have a hardware x-way to the AV1 decoder.
Um, and then you can avoid some memory copies,
if, if you can use UDMA Buff.
So you have support for that.
Um, if you're using GNOME, you might notice that, um,
um, I don't know, you probably know Totten, maybe.
Um, and it's finally getting replaced.
As a new midi player video player in the GNOME,
it's called Showtime and it uses all the cool news stuff.
We have, um, added in the last few years.
So it's all very efficient, uh, you know, zero copying,
output viable kernel OpenGL,
and, you know, UDMA Buff support.
And they use something called GDK4 Paintable Sync,
which is based on GDK4,
that the toolkit that GNOME uses, um,
that lives in the rust plug and,
so you can use that as well in your application.
And it's, it's all very nice.
And it looks very slick as well.
Um, compared to Totten.
Um,
AMD Hip.
Um, Hip is sort of,
it's very similar to Prudah.
It's like, you know, for, for graphics programming,
pixel modifications, basically.
But the idea is that with Hip is that it, you know,
Prudah is Nvidia specific.
So if you don't have Nvidia hardware, then,
yeah, it doesn't pipe anything.
So Hip is very, very similar, basically.
Um, but the idea is that if you run on AMD hardware,
you know, it uses, um,
the, the, the, the, the, the, the,
rock and underlying rock and engine,
and, you know, that worked perfectly.
And if you running on Nvidia hardware,
there would be like the, a very, um,
thin translation layer.
And then we would run on cooler, basically.
So you get like the best of both worlds.
And, you know, with two of the,
because, um, graphic cards,
then there's out there.
So we've got to plug in for that.
And we've got a few elements that were the usual things,
you know, like conversion, some scaling,
etc.
Uh, language findings.
So Gistima is written in C and Rust,
but you can use it from lots of different languages.
It's perfectly fine.
You can write your application in Python and Ruby.
And, you know, Rust here, of course, C plus plus,
whatever your hard advice,
um, because we've got language findings for most of these languages.
Um, one thing, well, first of all,
we've got new go bindings,
where newish go bindings,
but I think they're much more usable in the show now.
And they're called go GST.
They're externally maintained, but I think they're quite good.
Um, and quite exciting as well for some people,
um, who are not quite ready to jump on the Rust band
working here is the C plus plus bindings.
We used to have C plus plus bindings,
but they're all kind of bit rotten and I don't know what
it contains so many more.
We had two sets, actually, and they're both not contained anymore.
Um, but there's a lot of people who want to provide C plus plus.
Um, but they're now C new C plus plus bindings,
uh, based on a new, um,
sort of infrastructure, and gnome, called peer,
and they had us only bindings, and they, um, don't use the STL,
and all those are very cool.
Yeah.
Lots of Python bindings improvements,
uh, bindings, of course.
Please remember, we love Rust for a few love Rust.
Please come, um, trying to pull back source.
It's like an element that sort of allows you to, um,
to have a pull back source.
You have, have a primary stream, and then it drops out
for some reason.
The network goes away, um, your, you know,
graphic, your card doesn't deliver anymore.
You cycle goes away from your capture card.
Then you can sort of switch to background stream,
and then if the primary signal comes back,
you can switch back and, etc.
But before that only worked for raw audio video,
uncompressed audio video.
So now it can also handle encoded output,
which makes it much more flexible.
And you can, uh, you know, if you have multiple stream,
you have backup file.
You can select good streams you want to, um,
to use for the backup.
Um, so then here's a little bit, uh,
we've got some new element for VMath.
VMath is a, uh,
it's a framework by Netflix.
Um, and basically,
it tries to give like a perceptual,
video quality of the score for your video.
So if you, if you encode the video,
then you can, let's give decoded and see like,
well, you know,
how would that framework rate the encoding?
Um, and, you know,
how would a human,
perceive the encoding is a good,
it's not good.
You know, you can get some metrics for things.
So quite interesting.
We've got a new plugin for that.
Um,
a new webkit plugin as well,
that uses the, uh,
the embedded webkit platform API,
which I think gives you more acceleration.
Support, open gl,
and,
for a little bit.
And super exciting.
So I wrote it.
And you, I've cast think element with,
A.A.P support.
It's similar to shoutcast,
uh,
shout to send,
but shout to send,
they're not going to add A.A.P support ever,
because it's, you know,
a proprietary format,
or a lot of the price,
but royalty,
uh,
Petting Royc and come load.
So they're not going to add that,
but need to add a client,
and that's written rough.
And which else,
tracing in the bugging,
lots of work going on there.
If you've ever used,
um,
we have a mechanism to jump pipeline grass.
It's a sort of dot file.
And then you can render it into an SVG or a png,
and you can see like a beautiful,
you know,
what your pipeline looks like basically.
basically, which is very cool, but it's also very tedious. But, yeah, we've got a cool
new feature where you can basically just be in these things to a web browser and, you
know, then you just say like, I want a new graph and then it just pop up in a web browser
and you can click on it and yeah, lots of stuff there. Very cool. Also for those who are
writing plugins, we've got a sort of a logging system that's very, very chatty. And we've
got more mechanisms now, basically, to find tune the logging output a little bit. Because
sometimes you get a lot of spam, like some kind of warning is repeated like a million times
just because get some packet loss somewhere and then it's just, you know, clutter your
logs and you don't want that. Yeah, so we kind of have a mechanism now we can say like,
you know, don't print this more than once a second or only printed once ever, things like
that. So you can reduce that log message down a little bit. So if you, if you suffer from
that, let us know if we can look at the code in question, maybe make use of that
new API. And geistrum editing services is a library that basically provides you what you
would need to, for a video editor basically. And yeah, lots of work going on there in recent
times and lots of new features and also performance improvements. It's got new APIs to sort
of manage threads better so that, you know, video elements can share threads if you do
video processing and scaling and all of that. We across that form, so we don't just support
the nodes, we support Windows, Android, iOS, Mac OS, etc. On Windows, Direct 3D12 has seen
lots of new elements and lots of performance improvements. We've got a compositor, a Tish ID
wall, a UV remap and an interlaced element. There's also like a Windows IPC element so
you can actually send, you could use to send raw video buffers and audio buffers to
different process on your system. So you can split up pipelines and you can, you know,
have different applications that one independently and that one crashes the other
to the works. But that only used to handle uncompressed video audio and actually I'm not sure
if it handled audio, but it handled uncompressed video. But now you can send anything over
as basically that's been made generic, which is quite cool. There are about a 3D and
audio API from Windows and we support all of them. But now we basically worked on our
was RP support and made that really great. It's got dynamic audio device switching, it's
got more exclusive mode features, more negotiation features, so we're basically switching
only to that now. The device provider will only advertise me, if you use it in the
app. Apple MacOS and iOS, we've got more hardware curated decoding, we did decoding, more
encoding features, can do HDR now. And we have something called navigation event handling,
so if you have, you know, like mouse pointer movement across the video window, you can send
that through the pipeline and other bit of a pipeline can do things like render menus or,
you know, if you have a stream of web browser, then you can interact with the web browser
that you've rendered and find that. But we support that now. And right, the hardware provided
video that has seen some love. Several is sort of a matter of the system we have that
we used to produce binary packages for Windows, Android, MacOS, etc, because on these systems
you can't just install all the plugins and the libraries that we need to depend on, so
we have to build everything ourselves and put that. So that's how it's so robust.
Yeah, a couple of, well, we have a new Windows installer so it might not look like 9080s anymore,
which is nice. We've got an R64, we'll put on Windows now. We will provide Python
wheels on macOS and Windows so you can just install Gistreamer using PEP basically, which
is quite cool to get started and we also ship GDK4. So if you are GDK4 hacker or you want
to use it on Windows on Mac, you can use that now and use binary packages. All right,
so I have a few minutes. What's coming up? Lots of Rust things, as I said, we love Rust,
we add lots of new things in the Rust plugins. That's where most of the cool stuff happens,
you know, apart from the analytics things of course. Yeah, media over Quake or Mark, I don't know,
it's cool like that anymore, but it's sort of, you know, the ideas WebRDC is very, very complicated,
sort of the made new things called Mark and it's yeah, quite exciting. But I think the spec is
still very much in flux, but we have support for that. It's not merged yet, but it's a good lab,
but hopefully that will then soon. We're putting more things to Rust, WebRTCsack is going to
go Rust. Yeah, we've got the ISE bits already, the Rust library, which also has a C-refer,
and we're going to drive the new Rust RTP Stack further, which has been sponsored by this
or when Tech Fund a few years ago, more RTP payloads are going to be ported and in general,
we're trying to improve scalability so you can run more pipelines in parallel, like thousands,
tens of thousands of pipelines. Impact the ASCOS Rust as well. We're going to work in a complete
rebuke of the impact the SD maxer, which will hopefully improve reliability, especially when
they're sort of changes and the timing at the time domain, the clock changes. If programs
which has happened, or you know, we want to make that much smoother than or designs or sort of little
citrus, I think, should be fine, usually. And more little bit of stuff coming, especially,
you know, like video formats for sciencey people, geo intelligence all about, they like,
you know, fancy things, floating point numbers, complex numbers, find integer, so you can run
and I let you take some of these things. That's coming up, I'm just working on that.
And we have more speech and translation features, hopefully coming up, we support a lot of cloud
services. Some of those you can run on site, speech medics, but it's still proprietary and
you interact with it. If you think some API, it's just that the API sits on your own, that work
and not on some other network, but you know, still, I mean, we open source people, we kind of want
open source stuff. So I think, yeah, the new focus would be on local processing, so you can actually
run speech detection and translation, et cetera, on your own computer, on your local system.
So, you know, things like whisper, et cetera, I think, we'll be next. More formats, of course,
every two, well, it was supposedly supposed to be out at the end of last year, but
sure it's coming. HDR10 plus support is coming for a 6,5 and 81. Playback's luck, some improvements,
a timeshift element plus if you want that. By the year I input, so you can have not just, you
can, you can, you could have like multiple subtitles, for example, a lot of subtitles,
as input, so lots of changes there. Cracing and debugging, hopefully, well, there's support for
something, the rust world, there's something called the tracing crate, so we'll have more integration
for that. And also, hopefully, you know, we'll have something like, so you can do per pipeline,
the bugging, or per been the bug logging, because currently, if you enable the debug log,
it's just the bug logs for everything in your process. And if you run 20 pipelines, it will drop
20 pipelines, the debug logs, something you pipelines on you, and that's not always great.
So hopefully, you know, we find ways to fine tune that. More things coming up, TVOS,
what to ask in the vision of S support, and for me too hard to add, I'm in thought, and support for
digital design content. And that's all I've got. Thank you very much.
Thank you. Go, so any questions, can you put your hand up, action, and we pass the
microphone so that people online and being recorded can take this up?
So yeah, you mentioned we're adding support for a hip, but which is an abstraction of
kuda, but is there the equivalent of kuda stream, and are we planning to have an integration for that,
in the pipeline, when you have multiple island that are just, there's a change there.
So I think I'm going to quote half of the question, are you asking about the hip part, or the kuda stream part?
The hip part, are we going to have the equivalent of this in hip and,
and I don't know what we're going to ask. You have to ask them, I'm not sure what his plan is actually on.
Yes, hello. I have a question about the Vulkan video stuff, so we're using WebGPU, the RustWGPU
GPU, great to render, and right now we're letting G-streamer create an OpenGL context for us,
and we in the future would like to go all log, and so the whole pipeline is broken.
Is that possible now from 128, or is there still stuff we need to wait for?
I'm not an OpenGL, but I want to give you a wrong information, but
so who should we ask? Matthew Waters, Sebastian Dweger, Sunker, Sunker, Sunker, Sunker,
we'll jump on the matrix. All right, so that's good, she's there.
The other side of the room again.
Thank you, thank you. That's the link where all the G-streamer conference talks are for how
the browser was of course done. I thought the speech to text and translation
things running locally sounds very interesting, and from missing, is there a timeline for that?
I mean, I think the first bits are going to land very, very soon, like weeks, rather than months,
and then there's always more to, I guess.
It's quite, it's a little bit tricky because you have to, I mean, just, you know, on the whisperside,
I mean, it's not just the detection, but you wanted low latency as well.
Yeah, the tricky to make that right. So are there any plans to make simpler command line tools?
So for example, with FFMPEG, you just type of FMPEG input file and then output file with the right
extension, I would do the right thing with just saying default. I don't know if there is already
something like that with Gstreamer, I know that GST launch allows it, but with complex syntax.
I mean, some of these exist already, so there's something called GST Transcoder, which allows
you to, you know, basically, for the transcoding use case to have a simpler pipeline syntax.
So, you know, that doesn't work for you. Then there is the GES launch bit for the editing services.
So you can do everything, GST, the editing services library can do, but otherwise, you know,
write new tools for specific use cases and submit them. But I mean, I think it's a good goal.
It's just that, you know, don't have time again.
I found a question about the synchronicity of the different streams. So if Gstreamer can
accumulate so many streams, for example, captions or the metadata, how synchronized are there
with the real image data and is there some guarantees that Gstreamer cares for or how is it working?
We have a very, very good sort of timing and synchronization model. So it's basically, you know,
if your input is good, it will be perfectly synchronized. So, I mean, you know, we have the,
we have clocking, we have clocks in the pipeline, we have the time stems. You can attach
metadata to both of us. So, I mean, everything will be perfectly synchronized as long as, you know,
right times then, I put on things. And the, you know, the elements sort of honor the chosen
clock for the pipeline. So it should always be perfect. So if I, if I have a, for example,
submit a data, attach to specific frames, can I be sure they will arrive at the same frame
at the receiving site? Yeah, I mean, if it's attached to the frame, then it will always, you know,
be carried with the frame. Okay, cool, thanks. All right, probably out of time now.