Oh, so hello everyone, welcome to the 2025 edition of the Geistreamer State of the Union
presentation. So just in case Geistreamer is a multimedia framework, which is based on graph,
and that can be extended with plugins. But this is not a point of the presentation,
it's to update you on what's going on. So Geistreamer is actually 25 years old, which is kind
of nice because fuzz them is also 25 years old. So it always matches. It has had over 2000
contributors with the other little facts there. So quite a big project. And despite his age,
it's still pretty active. So on of the course of 2024, it's up till now. We have had around
50 contributors per month, which for a total around 300 patches per month on the project,
getting merged. And in terms of releases, so we've released three bug fix releases on 122
since last fuzz them. So I'm not released to release. I'm using the last fuzz them. 13 bug fixes on
124. And we actually released 125.1. Don't be fooled. It's not a real release. It's a
development release. You shouldn't use that in production. But that means that we are very,
very close to 126. Geistreamer rust. It also has the same releases, but it also has this
releases cycle on crates.io. So for that we had 14 releases of the Geistreamer bindings and 18
releases of the plug-ins there. Just know that not all plug-ins go forward because they don't change
every release. So we don't want to put out crates.io. In terms of GitLab, what does that mean?
So in terms of merge requests, it's 1200 merge request, which on the mono repo, which 400-ish
got actually backported. So massive amount of fixes that get back into the stable tree,
just to make stable better. And a couple of backports also in 128.
160 merge requests on the rust mining. 380 merge requests on the rust plug-in rust is a growing
part of Geistreamer is no longer optional. If you want all the features of Geistreamer
you need to include this plug-in pack now. And it's now well integrated in most Linux distribution.
It'll mention I don't do usually, but 230 merge requests on our build system this year. It's
getting a nice re-haul. It will get to that a bit later. So yeah, security fixes. It was a big year
in terms of security fixes. So security fixes comes in different banners. There are actually
from a bunch of researchers, which help us do that. So we have a revamped process actually to handle
these and make the patch and co-review those patches in a private GitLab protected GitLab
tree. So it's much better, I think, now and it gets through the CI and everything. And we managed
to handle 29 security fixes. 25 on the GitLab security lab banner. I don't know if it's all
that company, but and then two were reported under the Java and security notes and one under
the zero-day initiative, which has been there for a much longer amount of time.
And we get into what actually changed inside Geistreamer. So my organization is not perfect,
but it's an approximation and it doesn't cover everything there's too much. So let me start.
So at the very core of Geistreamer we have libraries, helpers, and stuff. And that a structure
and we're trying to reduce the memo lock that we do all over the place. And Geistreamer
DQ is one of them. It's a replacement for most of our usage of the Q from GeLab. And it uses
a vector, but it implements the Q logic. And it's using a multi-QQ, everything that Q has been
important to use that. The messages have always had a detail structure in there. It was just
hidden, so not with nothing anymore. Tag list, new tags coming. We now have the ability to place
caps structure into caps. Something we've talked about for years, but nobody ever implemented it
finally happened. So now you can have caps within your caps. And we target using that within the
AI domain where you need to negotiate the tensor format, but you still have the original picture
format and you need to match these things. That's a melting pot of serializer, the serializer,
so a lot of stuff goes through the pipeline parser, which is text, you need to transform that into
object and back into text. So there was some stuff that was not serializable. And there's this
new thing, the GSTID string, I had to read it because I have a hard time remembering, but it's an
optimization over the quark to try and get rid of the locking actually as much as possible. That was
implicated there, not getting into details. In terms of libraries, so we have higher-level libraries
for video for audio for stuff like this. So in the video library, there's the Ancillary Metadata,
which gained yet more, and so you did a metadata formats. Something that haven't been touched
for years, the navigation event. Now it has been improved to have mouse double click and scrolls.
The video converter is being worked to do better 10 bits support, has more 10 bit formats.
And one of very cool fix because it's a bug that has been around for a while is there's been a
rework of the deadline calculation with the QS event, which is the thing that decides to drop frames
ahead of time within your pipelines so that you don't waste too much CPU when you're late.
And yeah, now it behaves as we expect, meaning that if it's truly, truly late, it will at least
render one frame per second. So it's not going to drop everything and confuse users.
But every video filters, a few properties, this one got mentioned one 24, so you can
basically clip your data when it goes into video rate that goes out of the segment when you have
segment seeks and stuff. The audio cutter was calculating an audio level in RMS, but it was
keeping for itself. So now it's being exposed to the audio level meta, so you don't need to kind of
combine it with the level element in ways more cycle. Right-and-pitch has implementation for reverse playback.
Yeah, more format in video clips. The bangles of really have this new feature where they're
used for like subtitles or clock sync or stuff like this and you can actually tell it like what's the
on-included render delay for your display and you can compensate for it and can be kind of
useful if you do calorie care or stuff like this, there might be other use cases, I'm not the
world. But and then the volume, the volume always had this limitation that NX was the maximum
again that you put on the volume and we actually removed that because it's floating point
out and then if you're in and it's just going to cap. And now the audio converter can do more than
64 channel and there's a really interesting feature on scale tempo which is the fit down feature
where you can create buffers that are too large for the duration that they're supposed to be and
the scale tempo thing will resample. Well, we'll kind of convert them to make them make the things
peaks faster but fit the window. This is interesting for cases where you do a text to speech
but the combination of all these things like with translation from one language another but
one translated it doesn't fit so you can make it speak faster. I think it's interesting. I didn't
know about it before I prepared this slide so there's quite some work actually on adding analytics
metadata supports and also AI supports native in g streamers so that different frameworks can
change between each other and can negotiate between each other. So we now have design documents
so when we add stuff we can document it and we can actually challenge it with the others. There's
this segmentation meta that got added the tensor meta the tensor decoder which were already separated
from the execution of your inference is now separate but now it's outside of OIDNX so it's no
longer specific to one stack and we know that Intel on stack is starting to use that so that's
quite interesting so we have a second vendor that is using it. You can do some rotation when you do
object detection so that you can have a rectangle and squeak it so you reduce the surface.
Yeah we have this element that that was written it is basically converter that converts the
relation meta data from your AI research into on-vift meta data and then using on-vift means
that you can stream it over RTP and then you can take that back on a remote machine like your
server or whatever and you can put the logic somewhere else. So you do the AI you send the results
somewhere else and then you recover it through RTP. Original buffer it's an interesting one so
when you do AI you always have to scale down and crop to a square or something and you always have
the need to recover the original image to put back the meta data on the original image so that's
exactly what it does so it will save the buffer in its own meta data and then once you're done it
will restore the original image getting rid of the small one from the that was used for the AI and
then it will reapply the meta data and apply some transform to it of course if it's a resolution bound.
And there's a lot of work done on the Python binding to the ideas that Python is the
predominant language for AI stuff so it's mostly at the control level because it's not that
performance inside G streamer itself as a plugin. Stream containers yeah a lot is going on there
so stream containers that's basically impact TS is a went before MKV so a lot of future got added
into MPGTS it has a segment seeking where you can have a stop time and loop back on some zones it's a
bit difficult with things like MPGTS it's not meant for that there's GPEG access as being map
PS meta data I'll skip that one more and so every data including AES there's a it's not complete
but it's support for synchronous KLV streams there we have a custom just from a specific mapping for
VP9 so you can actually put some VP9 there it's not illegal but it's going to be ignored by
most player because there's no spec for it there's a provisional spec for AV1 that has been implemented
and VVC has been added and the I thought it was provisional that's what they said in the
comment message they lied I need to read more than the message than next time I saw MP4 it's a bit
confusing because it's being rewritten really in in Ross but there's one in C it has gained some
C I ain't simple grouping and you are going I don't know the laggerit lossless video support
more MPG video one mapping and I'll compress audio and hopefully before the release but I'm not sure
there was a lot of work that was spending for on compressed video in there and then on the
Ross side the reason for the rewrite was because the old one was difficult for fragmented MP4 not
ugly so they wrote a new one but the new one is good so they started to write more and it's
probably eventually going to replace and as you can see most of the list there are features that are
just to get back to parity or more over the engine so I can predict that eventually
this will take over and the sign of that the dash
max are basically as changes preference and prefer the the Ross implementation now
the reason it was not is because it was missing all these things which was very useful
not a work in D3D 12 API this year it really became the default not D3D 11 anymore so it has a
GST Libs helper for the shader which are the same it's been split in his own library so there's
helper library to do HLSL yeah new feature new capture it has the IPC source and sync like the
D3D 11 one there's some feedback when you do plug-in of your GPU I've never tried that in my life
something I should someone made it a me made it compile by removing feature on a Ming
DW so that's kind of nice it's integrated into convert frame that's something related to
on sync when you get the sample anyone to convert it to something yeah let's keep this one
map mapping support even though there has been some regression reported we'll have to check if we keep
that one as a default and more scale or more scaling and interlacing feature just to complete the
portrait some nice feature something that can be annoying with things in D3D but they added an option
that once you close the window it doesn't throw an error I think I think it's nice we should probably
generalize that one to do and yeah of course this is the same guy you did the new navigation event
in the navigation API so it's got implemented there so it's my it's more complete it has color balance
implementing the color balance API HDR capture more pixel formats what else oh until he
has him in the swap chain sync element there's a VPH encoder so our legacy is covered and
yeah it's just oh yeah the rain the rain got got turned into primary for most of these elements
so they become like the decoders are primary they're gonna be picked by gentle plugger
more window stuff web view too which is kind of the windows widget for web view is like the
the WPE of windows it's been implemented yeah the right QSV and other things I mean they there's a lot
of places where they actually just added interlacing to make sure that it can be used with D3D 12
and yeah and there was there has been some maintenance in the D3D 11 element too because that's
not going away and then we can move to a big block the codex domain which includes many things so
I've started splitting with the parser and the software codex one of the why change we did is that
initially we had a v1 and we didn't really clearly define what a v1 was and it's actually multiple
packing format organization of your AV1 streams so now we've pretty much explicitly set a TU alignment
on the element that didn't specify anything so it's it's very clear what you're actually streaming
TU is time unit for those that are not aware H6X4 pars now supports the AVCD codex
configuration we attempted to implement a backlog to fix some AU splitting issues was a failure we
reverted it hopefully we're going to have time to look at that again this year VVC parser with the
typefinder and the mime type has been merged so it's not complete yet there's going to be more work
but at least the basic is there you can parse it it exists you can find it so so it's start and you
can mix it into MPEGDS I'll see EVC that's kind of a supplemental layer on top of legacy codex
so it works with any codex it's a MPX standard and has been implemented software the
color and then color the hardware part is to do and the SVT GPExS software codex has been added
including interlay support and someone enable the m101 matrix on compressed video support
that is implemented in an FFMPEG and also we expose the color roomy three out of David decoder
and yeah a final one a bit special so we have a fake decoder now it's called a fake video deck
you can give it anything you want and it's going to decode it right and it's going to generate
you a snake pattern this is very nice for a unit test where you basically just need something that
moves you need the video but you don't need the real decoder. Vulcan VAM SDK so basically all the
accelerators so the Vulcan video support is progressing slowly we got an h265 decoder now
and there's more work pending the VA encoders is mostly feature complete so GPG VPA VP9
the rate controls are I've been enable render delay to optimize the performance
unfortunately we add to these able the i 965 encoders that are on maintained by Intel
and are just too crashy we would have to implement the legacy code to make them work losing the
M above and we decided not to do that we now have a VVC decoder and yeah that's it the past processor
has interpolation enabled but it's limited to iHD it's the only driver that implement that
and Intel folks did the same thing in msdk framework on the before I'll side both plugins now
implement the dma drm caps negotiations so you can negotiate dm above with a dm with a drm format
from the next drm not the right management thing and modifier so you can actually effectively
negotiate an opaque format enabling compression notably the dec 400 from VVNTE or fbc from arm
that's now possible there's still some refinement to do but it's a lot better so
126 to look forward there was some stuff in 124 now it's a lot more usable in complete in in the next version
we implemented what we called dynamic frame rate in g3 mer011 on the before l2 state full
and remade codec the native implementation has been added so it reduces the amount of interaction with java
engine making it faster and more profiles and stuff we remove the magically implementation
because they dropped it on their site so we dropped it they actually use the the Android and decay now
and on macOS traded output got enabled that took a while actually more rainfall
hvc alpha and hlg there that's me move on a bit faster so in our tp streaming
we have a bunch of new features i'm not going to name them all but i think it's good to mention
that synchronization has been improved quite a lot and bigger feature also the renegotiation
and web rtc been i think that that's kind of very useful and was missing it's probably also
helps in making it more compliant more web compliant implementation
more streaming rtmp2 which replaces the rtmp plugin basically they implemented ideas from live rtmp
so you can actually add commons i mean it's more like an IPC the rtmp protocol so you need to
be very flexible so it adds a bit flexibility and there's the line network authentication that
I did you plug in the rust side queen i think it's queen i don't know the pronounce it
which implemented quick protocol sourcing sync pretty nice and payg live source
which basically is a wrapper that wraps an MPKS source reads the PCR and generate a
clock from the PCR and that let you basically play back the thing on sync with the incoming string
the transmitter clock basically in that years hlsmulti variant that's basically the fourth
implementation of an hls sync but this one is it's specialized you can see by his name so
i'm not an expert i'm not gonna go too far but it's there it's new you can use it
oh i skipped the lyrics so you can you can actually ingest the lyrics from a Spotify
which is nice if you do a karaoke of course we got a rewrite ongoing that was announced
two years ago when hearing about the rtp stack and this is pretty active
the initial RTP been got merged there's an RTSP client that has been written
of course if you want to rewrite you need to write everything so payloaders are being rewritten
so a really big set of it got written and then we got to web RTC so there's
you need to kind of layer it up
a v1 in the raw has been added and yeah you can actually change your your resolution and
framerate dynamically and there's more support for different encoders which we
unfortunately have to wrap individually to support this may accelerator yeah
what should i mention is why i need to skip a little bit so you'll find dmb of support in gls
think then dmb of support in wailin some improvement on the qt where you can actually do the
color convert at the same time your render that's quite nice we should generalize this one too
gtk for the gtk for paintable as gain dmb of past true so you can actually get the dmb of
straight into gtk and gtk will try to use a hardware plane so it will actually internally
create a subsurface and do all this stuff for you making it probably one of the most flexible and
performance thing we have so far in the on the lean excite at least so watch that out it's really nice
and yeah and yeah that that was possible with the ability to pass true dmb of in the
gl sink wrapper video capture there was a bunch of issue in the qml six source now it's usable
i don't think it's perfect but it's usable and it actually uses the downstream pull that you provide to it
which has been a key in being able to do zero copy to the to encoder through that that sink
there's support for the adja capture cards capture and output cards decling ones got improved with
HDR you can now in image frame image freeze and life sink you can actually pass compressed data
so i frame only compressed images because there was no reason not to there's this new q which is called
god buffer which basically queues a minimum of one gap there's a tracer that i think is pretty nice
in rust that will output basically a point in your stream into a pickup and then you can
analyze the pickup with a tool like wire shark or other tool and yeah there's also a new elements
that let you modify the group IDs in the streams that's more of the issues because oh and
disrumrated in services i've gained a reverse playback this year
much time I got okay so in the close captioning i think it's just moving to rust quite quickly
last year there was like one one logo no it's only this and yes mostly c a seven o eight implementation
and with backward compatibility so i think it's quite a full feature and there's integration with
the transcriber so the transcriber bin is a bin that let you transform text into speech speech into text
and most of the implementation are of course using cloud-based AI system to do so but it doesn't
have to and it also transformed the text into close caption so it's a nice little adapter there well
it's not little um as i said there was a lot of changes in the rehaul of the the cross platform
build system that we use this is used mostly to produce like windows, android, macOS images where we need
to build the dependencies also in order to make those binary while on Linux we rely on distribution
to take care and yeah basically there was a lot of plug is not enabled for a long time or because
there's a new rust support was not everywhere so it's been added where possible and yeah small
feature lip prox is now a dependency that we build so that you can do prox see if there's still something
you do in 2025 and yeah last but not the least i wanted to mention the work we do on it is happening
on arc just in time compiler so it's a byte code language that get translated into s i md
and there's two developers that are pretty active on refactoring cleaning up, optimizing,
improving the cache usage and they have actually a bunch of ideas to make it better and one of
them important change which is one of the cause of disabling arc and making all the software stuff
really slow is that it can detect s elinex and no exact flag which would otherwise crash
as usual if you need help you want to reach to us you want to talk to us we got this course
which is the forum which has been very active i didn't extract any statistic on it but i think
would be nice in the future we're preactive on metrics we have dedicated channel for different
subjects there's channels for newcomers if you feel that is to kind of impressive you can file bugs
there in github if you find any issues just remember an issue is not a support ticket we got the
forum for that so it's because you think something is broken in github or it doesn't matter if
it's true or not that's our job to tell you just don't confuse the support from that and it
makes our life a little easier we're not as big group of maintainer but we have a quite big amount of
issues and issue reporting and major requests to deal with and that's it we should have about five
minutes for questions i believe and thank you
yes
so the question is if i can and elaborate on what is the synchronization improvement in
RTP been partially because i was not involved but it's related to the synchronization the
flux synchronization in RTPC feedback so best will be to go look at the yeah but i don't i don't
have the i didn't dig into the details for that one this year do you have anything a
timid that you would add right but part of the comments i've read there was like
cases where it will integer overflow which doesn't it's not a security issue but it's a
statistic problem yeah so there was bits of that and it would break other mechanisms
that depends on having the right the right value there and it's i think i think some of this
is quite visible now that people uses it in browser with static statistics graph something we never
made any effort on before except from pexip which do that in their own property system so we don't
necessarily see it if we're doing something wrong yeah other questions
yeah going I would call the RTP been in a work in progress mostly not i think the features
like that implemented are oh yeah sorry so the question is uh basically our usable is the new
RTP been in in the rough implementation um my belief for what i've seen and what i've got a
report i haven't used it myself mostly a hardware accelerator guy but what has been implemented
has been done as well as possible so it's new code but it should be fairly usable but it's missing
feature and when we speak about RTP we speak about a lot of feature we speak about hundreds and
hundreds of pages of spec for all these these features so this is going to take some time i think the
transition to to and sorry a question about when we'd be the transition will be when we can actually
put that back into a browser and the test actually shows us that it's good enough for a browser
i think that that would be the right answer i'm making up maybe some developer won't agree with
me so just take it with a grain as well of salt that's my belief the main point of the new
RTP been is also to address some of the performance issues when you do multiple sessions actually
concrete yeah that's a good question because okay so for the GPEG access from SVT implementation
traditionally these are made for our PC server the actually specifically made for Intel server
so that's something that i would be curious to check myself too so thanks for the question
i don't have an answer for you but it's a good question really really yeah so do i know
what explicitly changed for the NVMM cooler work basically the caps negotiation for the NVMM
caps feature has been completed and is usable now so basically in two caps you can you can open
the parenthesis and have memory column NVMM and that comes with a bunch of APIs that you need
to use specific to Nvidia and these these are being I have been added there's a big benefit
when you use actually the the downstream basically stack from from in the
yeah so i'll repeat to measure i heard correctly so if we have an initiative to
be able to translate GSM into web assembly right i believe it's something that has been
attempted in the past but it's extremely difficult because GSM is the framework the real hard task
is done by dependencies by library so when we speak about web assembly of GSM you don't just have
the GSM code base that would be really easy you actually need to pull all the dependencies and this is
a hundreds of library which are built differently with different languages so i think it's a big
feat you can probably make it with a subset of plugins that that you want to use but i think it's
gonna be really difficult how that will be really nice so Tim was saying that there's a project
that started and they they actually being made a website in post of the result and the difficulties
or they're good things they managed to do so something to search time is up thank you very much