I'm a maintainer of QtMotemedia, my name is Artham Domen, and this is my colleague.
I am an expressive scholar, I am a software engineer in Domenia team.
I've been working with QtMotemedia for three and five years.
Mostly I was focusing on QtMotemedia stuff.
In different subdomains and different platforms.
And it's also doing nice contribution in many QtMotemedia subdomains.
And now we are going to elaborate on what QtMotemedia, what it explores,
which benefits we have, which problems we have, and probably engage in the audience
to any discussions, let's see what we can end up with.
There's a full of what's QtMotemedia.
It's a model of the QtMotemedia framework.
It provides various audio and audio functionality.
We're back, recording, working with audio devices, abstractions and so on.
QtMotemedia represents like a part of general Qt philosophy.
It means that it complies with Qt design.
It complies with Qt infrastructure and so on.
We will talk about it a bit later.
And it's permissive what's it in LGPL and any sort of part of related stuff like meat and so on.
Okay, let's talk a bit about strengths of QtMotemedia.
First of all, we are striving to support both sides of QtMotemedia.
Definitely, it's not kind of the same.
It's a surplus plus 1K energy of more advanced functionality.
And we expose more advanced level API.
But with QML, it's definitely just high level stuff.
And let's talk about this a bit later when we are discussing any challenges and so on.
Then we do support on video support and variety of platforms.
First of all, it's desktop, but also we are highly focusing on embedded stuff and also it's mobile.
Then it's a part of the Qt approach that we expose high level abstractions.
So we first of all focus on something that can achieve the exact result.
Like media player, media recorder, camera abstraction and so on.
Then as I said before, we are highly interrelated with common Qt infrastructure like QtMotemedia.
We are seeing loss loads and everything is related.
IO device mostly is related to audio stuff like inputs, outputs and also it's related to media recording.
And also what we are striving to do, we want our sub-module work out of the box.
It means that we are delivering our dependency.
First and foremost, it's a firm pad out of the box.
It means that if you install QtMotemedia, then everything is supposed to work.
Okay, nice.
This multimedia effect.
This one corpus.
It was a plan.
Okay, let's go on.
The QtMotemedia API, you can be represented as a set of different inputs and outputs.
This inputs and outputs can be purely logical elements or be an abstraction around any physical device.
I mean, rather high level abstraction than low level abstraction.
So here on this diagram we can see kind of main functionality.
It's media player, it's inputs and outputs.
So we can output to widgets and QML, like presenting immediately.
Or we can also expose just video frames as this.
So that users can access to some low level video data.
And also let's take a look at the media capture session.
It also has a set of video inputs, audio inputs and outputs.
And for the users, the most common case of using capture session is recorder.
And users can create audio and video files by attaching any inputs to capture capture session.
So it can be camera, screen on window capture, audio input, like a microphone.
Or user can send their own frames, audio or video frames.
And in the scope of formats it could support.
Yeah, and the same, the most popular case is recorder, but also it can be directed to other stuff.
Let's not focus on any geeky stuff now.
Let's talk a bit about the contents or plugins that we support.
In Q5, probably many of you know.
We did support native backgrounds, first of all.
And we do support them now, but only partially.
We have switched on the FMP, it's a primary background now.
And we have time to utilize each on many platforms, but we can apply it on embedded platforms.
Because of the limitations.
And on embedded platforms, just in your now is primary stuff.
Actually, we want to get rid of this, but with the current limitations, the lead cannot.
Because FMP requires implementation for specific works.
And it's a specific API for each vendor.
And that's why it kind of blocks us to switch to FMP stuff.
But it would be nice.
Also, I can mention that this became functionality is mostly related to encoding and decoding.
Other stuff like cameras like audio is implemented through native machinery anyway.
So to put some of this into context, I have prepared just a few examples.
First off, we're going to showcase playback and to do so.
We're first going to be doing this in C++ with the cute widgets.
If I run this code as it is, it's very simple.
It's layout with some text labels, very trivial stuff.
In order to get some video output on this in this application, we need the three parts.
We need the video source, which is either a stream or a file.
I have prepared a file ahead of time.
We also need a media player object, which can load this source and play it.
And we also need an output, which is a UI element that will just display the frames.
So, I do not have two hands available, but I have prepared a cheat sheet.
If we take a look at this updated code.
So the only new parts right now are here.
You will see we have created a media player, we have selected our source.
We have told it to play immediately, once it's loaded.
And finally, we have added a video item, a video widget to our UI,
and we're going to specify it to be the output of our media player.
And in addition, I have added a simple audio output as well.
And the default behavior of this audio output is to choose the default audio device on your system.
So, if I run this, oh, damn it.
Yeah, I come to the wrong file.
If we run this, we will have a video playback.
So, it's a very simple process. In this case, we added 7 lines of code, 8 lines of code.
Right. So, that shows us how easy it can be to add multimedia functionality to an existing cute application.
I have one other example that I want to show off, which is recording oriented.
Yes.
This example is written in QML.
QML is the more recent UI framework within Qt, which is more targeted to mobile and embedded.
But it works on desktop as well.
It's designed to be less oriented around C++.
And the easier to understand from a front end perspective.
What I want to show off here is that we have this code in particular, which is a column.
We have a button, a label, and a video output.
If I run this, it will look like I'm actually not going to run this because there are some issues when doing screen recording with this application.
But we have an app that looks like this.
Now, I have also added some screen capturing functionality with some output in this code.
You will see it here, the relevant parts.
This is not correct.
Okay. So, it's set up for camera right now.
But no, it's first screen capture.
Okay, so the point that I'm trying to convey is that we are able to add the easy interactions to record.
You need a capture session, you need a media recorder, and you need a source.
Now, I just showed off some screen capturing, but if I run this code specifically, we will be able to see that it's also able to use camera.
And it's basically two or three lines called changed in order to change the source.
And that concludes the parts I wanted to show.
Let's go back to the slides.
What these examples are striving to emphasize is that we expose some high level API.
And actually, this API doesn't need to know any specific multimedia advance stuff.
Okay, let's go through the challenges that we have.
It can be interested for one who develops any multimedia applications.
Everything is revolving around the modern maintenance.
And the points below just describe what it's this about.
First of all, the asynchronous nature of the backends sometimes doesn't align directly with the cute multimedia design.
And it's not always easy to fix.
You can elaborate more on camera and the capture functionality.
Right, so in traditional cute API design philosophy, everything is preferably synchronous.
And there is not much error handling.
However, in multimedia we are finding that there are lots of asynchronous tasks, including stocking a camera or stocking a stream or setting up the screen capture.
There's a lot of operations that should not be blocking because otherwise we would be blocking the UI thread in some cases.
It's not ideal and there's also a lot of change operations and things can fail at each step.
And this is a bit hard to align with the usual cute design philosophy.
Okay, thanks, yes.
Okay, then we are starting to support features in KML because it's also a priority.
And KML is not compatible with handling low level data.
That's a known issue.
And that's why users need to integrate some C++ code on top of KML examples on top of their KML solutions so that they can achieve more.
And for us, it's also a challenge to have an API that expose the things that are needed for users on their other hand.
On the other hand, keep the API consistent with our high level concepts.
Yes, and then different behavior and API designs across platforms and buttons.
It's also a known issue.
And the most case is a firm pack, backend and distributed backend.
For instance, the distributed media backend has highly asynchronous nature.
A lot of asynchronous processes.
Also, it has a lot of some press setups.
It has limitations with changing the pipeline on the flight.
And it doesn't comply with the QT multimedia design in some cases.
So we are still fighting with it.
And the things that probably everybody encounters who use QT multimedia is complexity is auto test.
First of all, it's a challenge of covering hardware related stuff.
Specifically, if it's a camera or if it's audio device,
we need some visual stuff or we need to implement some loopbacks or whatever.
And it definitely complicates the CI and it complicates auto test infrastructure.
And definitely we got flakiness.
Jim is here, he knows about flakiness more, but let's not stop on this now.
And then the last light about our plans.
First of all, we are focusing on maintenance and we are going to do it.
We are going to proceed with it.
This year, we are cooking a proof of concept with some AI integrations.
Let's see what we can do.
First of all, it's a embedded area, just a mere oriented solutions.
Let's see.
Not only our team is working on this, but we are striving to come up with some solution.
It might be a streaming support.
I mean, service side streaming, who knows.
It might be video processing API,
so that users can combine the frames, blend or whatever.
And the most controversial topic is review multimedia philosophy.
And instead of using some backends under the hood,
make multimedia able to work with these surf patches just on the interface level.
It might bring us more opportunities, but it requires significant redesign.
It's definitely not for Q6, maybe for Q7, but it's to be discussed.
It's just a very brainstorming idea when also how it goes.
Pay attention to your attention, guys.
If you have any questions, any proposals or any feedback,
if you are using a good multimedia, welcome.
Are there any questions?
So in the meantime, the next speaker, can you please connect?
Can you unpack in your laptop?
Why?
Yeah, or at least make yourself ready.
So who are that again?
Hi there.
Thanks for doing this talk.
You talked about error handling and making this system kind of easy to use.
What do you use to see if any of this fails?
How do you wrap up the error handling, essentially?
That's my question.
It was so nice here.
You're asking about error handling and how it works.
It's good multimedia, right?
Multimedia.
I'm interested in what happens.
You're streaming a video streaming camera.
You mentioned the device fails.
There's lots of areas of failure in this.
How do you wrap that up?
How do you encapsulate all that in this API?
The API is quite straightforward.
The example is quite simple.
What happens when something fails?
What does it use to see it, essentially?
What we can do is handling this loss connection on the fly.
Then propagate this signal to the application thread,
specifically to the camera thread.
I made this signal from the interface that error healed.
I installed this and stopped the playback.
Maybe switch the status to an inactive or internalization.
We can represent any abstractions around the error.
But we don't access, don't have access to the error.
I mean, because error is usually a platform specific stuff, right?
But in YouTube, we don't have platform specific.
We expose some common functionality.
We also can come up with any common error types.
We also can expose this error types to the interface.
The reason for the error, if it's a system error,
if it's in a logical error or whatever.
Thank you.
Thank you for the talk.
You said the FMPEG is acting as the back end.
Am I right in saying that's what's interacting with cell Linux with pipe wire
and the actual lower level of the video in audio stream when you're streaming on your device?
Our media backend is mostly related to encoding and decoding.
What do you mention?
By fire, it's released by fire for audio and screen capture and implementation on Linux.
And it's kind of parallel for FMPEG.
So for accessing actual devices like audio, screen or window applications,
then we implement the machinery manually by passing any FMPEG functionality.
So we don't use the leap of the device,
such as a module of FMPEG, that is working somehow with the devices.
Because it doesn't represent, doesn't expose all the functionality that we need to work with multimedia.
Thank you.
Is there another question?
No?
Okay.
I was wondering if you just tell us a little bit more about the history of the project in the background
because I was curious.
I'm guessing it's running across all OSs or any limitations there too.
The history of project is the following.
Probably I don't know all the details because I joined when it was Q6.
And I didn't participate in any Q5 development.
But it was interesting that in Q5, we used only native buckets under the hood.
And it has been developed for a long time.
And actually, it exposed more functionality than we do now.
Why so?
Because step by step, it was added.
But without any thoughts in mind how it will be supported during the long time.
And at the end, we ended up that Q5 is the whole variety of features.
It's just not supported.
It's not supportable.
And it's buggy.
It's really hard to maintain it.
And that's why it was decided to show in the functionality in Q6.
And expose things that we really can support.
And then we've been adding some features that were requested by users.
Someone of users requested, oh, we need some functionality from Q5.
Why it's not in Q6.
And we added some stuff on top.
So this is kind of a short history here.
Who?
Did you answer that news?
No?
OK.
Can you please scroll back to your first slide?
This one, or this one, maybe.
Can we have a picture of you, too?
In front of your slides?
OK.
Just there.
And he's taking a picture.
Thank you.
Thanks a lot for being here.
Yes.
Thank you.