So, good morning, it's a pipe wire talk, I'm been tired months, I'm been working on pipe
wire for 10 years now, sponsored by Red Hat, so what is pipe wire for you, don't know, it's
a low level layer for sharing multimedia on your desktop.
So first of all, sharing of multimedia, which means audio, video, and all of that, but then
also later processing, audio processing and other stuff.
So it started basically as a way of sharing video from Wayland, because that wasn't supported
natively.
So then a scheme was developed to transfer video frames from one application, the window manager,
to other applications, the consumers via IPC, and then it was invented that you could also
use these four webcams, because currently webcams on that desktop, most browser go directly
to the kernel with IOCTLs, it doesn't work so well for newer cameras that deal a little
bit more setup and so on.
So these things were retot 10 years ago, and that's how it started later on, audio was added
and it became big.
So a little overview of what's currently happening with pipe wire, so it's sitting there
is basically just passing around multimedia between a whole bunch of applications and
services that there are.
So you have video apps, you have audio apps, there's also professional audio apps that
used to go through a different audio server back in the day.
You have window managers, all the things on top, media players, VLC, G-Streamer, that all
needs some way of accessing cameras, Bluetooth devices, also devices and so on.
Underneath that for the kernel, you have a couple of services and libraries that are used
like also, lip camera is a new library for accessing cameras, specifically for more complex
cameras that have processing units and so on.
So you can't really go directly anymore to the kernel to get your camera things and
expect to work these days.
Then there's also the Bluetooth services, which traditionally on Linux, used Bluetooth,
which is basically a debug service.
So all of that needs to be made available to applications in some sort of API, so traditionally
we have pulse audio doing nothing and the whole separate thing for pro apps which jack.
So popular is basically the common denominator of all these things to pass around this
data.
So basically an application of sorts would make a note in a draft, so it's all graph based.
Data flows from one component through a couple of links to another component of two
whole bunch of things.
These components can be applications or devices or enemies you want.
So it's built a generalization of what pulse audio did.
It's very similar to what jack did, but not also extended for video and audio.
So yeah, 24 was 1.2 release.
So last year I talked about 1.0 release, which was basically a BI stable, all of the features
baseline features implemented.
So in 1.2 we just continued doing that.
So couple of things that are new in that release.
So in the 1.0 version you could have that if one of the processing blocks didn't respond
in a graph everything would stall because basically you give it data, pipeline will wait until
it says I'm finished and then we'll go on processing the next block.
So if you block one part in a graph the whole thing doesn't finish to completion.
So you can have this situation where you can control v and up pause it and things just
don't work anymore.
The only way to fix that is to add delay latency.
So you start the processing of the app, but you use data from the previous cycles that you
schedule it.
If you didn't do anything then it's just silence, but you give yourself enough time to be
able to handle the fact that it's not doing anything.
So asynchronous processing I don't know we don't really activate it yet because it's
I mean it's a real problem, but it's not very I mean don't really get to it very often.
But theoretically we could do it for flat-park apps so that they don't block audio, but
it's really like a very theoretical case.
Apps that don't work people just usually don't run them.
Most apps they work fine.
So I was going to show this but then I realized that the only thing to actually show it very
well is with a jack client, but the jack simple client doesn't really do what I need
to do and then I'll make it a virtual device but that doesn't work either.
But anyway you could run for example two jack clients if they come back to the right sync
which they don't buy default, they can stall things and you can manually set the property
to true with some properties and then it will be in asynchronous but it will have a delay
but you won't really notice it normally but at least it won't block everything.
So jack used to have something like that but system wide so all of the things would be delayed.
In part where you can select one specific app that is yep I have a question.
And because you will block in client, block everything that is somehow connected to it
or will it only block something that's where data flows through it.
So the question is if a client doesn't process in the graph will it block everything
or only the things it's connected to? So only the things it's connected to is the answer.
So if you have two media players they go to a sync and one is blocking the other one will still go
through. But if you have for example two media players going to a filter going to a sync
then one will block the filter and it will also block the data transfer for the other.
So our components in the middle there will be blocked. So that's what this could fix.
So this for example is by default for things like pulse audio. So all the pulse audio apps
that currently use the pulse audio API they are already a sync. It's slightly different
but they will never block anything in in part where. So you need to have jack apps or native apps
that can do these bad things. So it's already a very small part and the same thing
if you have the jack app that was means behaving it will also not work with jack for the same reasons.
Yeah, I'm not a trailer execution. The 1.0 or single threaded. But now it's multi-treaded.
Also I'm not activated by default because we don't really know exactly if it's interesting to do this.
But I know some people you can manually configure it and then some people they wanted to
separate the threats for audio and video and give different priorities and have different kinds of
things and pin threats to CPUs and all of that. You can do that but it's not really activated by default on desktop.
I'll wait a bit until more people suffer.
So these things, these features were added because some of the
asks for them not exactly because we're going to activate them for everybody but if you want features
you can suggest them. There's always a possibility that they are easy or that they make sense
to implement and then they will be implemented. So the goal of starting is stopping things in the
graph has been refined a bit so that it causes less glitches things like that. So it's more like
a continuation of what we've been doing, making things a bit better. So for example,
one of the things that are still ongoing is flappack security context. So one of the ways for doing
security currently in Pyquires by using a portal. So if you I'll show you that later.
If you for example go to a web browser and access the camera, you will see some pop-ups
asking you for permission to access the camera. This also sounds very trivial but this is a new thing in Linux.
It's using a portal to get access to the permissions of the browser and so on and then make
a connection and make a secure context where the user can only access the cameras that have been
allowed and so on. And there's another way of doing things with Whalen which is called the security
context where a flappack would be given a Whalen socket or in this case a Pyquires socket
which restricted permissions. So that would make applications work a bit more transparent because
they just use Pyquires to access things instead of asking a portal to do things. But these things
are in in the works. Don't actually use them yet. So also an ongoing thing that is
getting there and I think we finally have an idea of what should happen which is the video
processing pipeline. So the audio processing is pretty much figured out. The video part
not so much like one of the things for example is I did this here. So if you for example have a
normal, a newer distro and newer browsers all over the camera stuff will basically go to
Pyquires. So there's a little tool you can use. PW top that should show you and it doesn't.
Maybe I did something wrong. Maybe it's going to the fall back. I should have checked that. Anyway
it's probably going to the fall back here. Maybe I can restart it. Oh this is chrome yeah I shouldn't do
this chrome. And I have some extra time. See if I can get this
yet some very happy. So well there's a thing where you get first an access thing from
the browser itself which camera you want to allow. I probably already said yes to this from the portal.
There we are. So you'll see Firefox is now accessing the camera through Pyquires
and as you can see here from this format things is that it's using mjpec. So this is the thing
with the cameras. You can use higher resolutions if you ask for compressed format because of
bound with issues. The problem then is of course if you want to share the same screen or stream
with another application that doesn't understand mjpec then things don't work automatically.
So you need some way of converting the formats into one comonical format that everybody can consume.
So the question is do I add decolors to all applications or do I decode before sending it into
Pyquires and see if I do I have this up here or do I have help them?
Those are haven't seen this yet. So you can also make a graph of whatever is running
in Pyquires and if you use something that is not jack you will also see videos stuff.
You can actually drag video connections to other video apps but it's all a little bit wonky still
because in this case for example it would be mjpec that would be shared and you need an update also
to understand some jpec. So the current plan is to actually add the decoder to the camera part.
So before it goes into Pyquires decode everything so you can still get the high frame rates
because it's captured as on jpec and everybody can consume it because it's already decoded date.
So that is something that we're still moving to. So for that we would like to use
accelerated stuff to do that especially for format conversions. So usually if you do a jpec depression
compression is software you end up with some yuv format if you want converter to RGB
you would like to do that with welcome instead of also software. So how do we get to those pieces in
there it's ongoing. So another thing we're working on is explicit think. What is that?
Well it's something Nvidia specific if I'm not mistaken. So basically when the composer
renders the video frame it sends all of the commands to render the thing to the graphics card.
What you usually then do as a composer is wait until everything is rendered and then tell the
client okay here's a video thing that are captured. We'd explicit think what you can do is send it
to the graphic card and set the checkpoints or a timer and then you send everything to the client
wait for this timer to go off and then you can read the frame. So then you can already bump a whole
bunch of things to the client the client can put all of these processing that it will do on the
frame in the heart we're already synchronized on that sync point and then things get a little bit
smoother. So in pipeline that involves something over a couple more of these and a couple more
method data to get this working. Yeah so more Bluetooth codex so for Bluetooth for example
interesting bits and for the next version there's some more coming up but there's also an
opus codec that you can use or Bluetooth and there are actually two versions of them. So in one point
zero pipeline we implemented our own version the time Google came out with another spec so now we have
two two versions of the spec Google Opus and pipe wire Opus. So if you have if you have a Bluetooth
device that understands Opus which I don't know you do probably not then you can use this but
it's more interesting of course because you can you can make pipe wire act as a Bluetooth consumer
or produce you can link two pipe wire desks up together and send Bluetooth between them. So then
you can use any of the codex of course. Yeah something else that we're working on lazy scheduling.
So what is that? Some people use the compositor to just fetch frames and then display them
locally on their machine so you basically need to fetch frames at the refresh rate of where you're
going to display them, not where they are generated. So there's a way in pipe wire for a client to
fetch more frames and not fetch needlessly but only when there's actually something new on the server.
So there's a communication way between server inclined or there's a new frame client can start
the graph, pull the frame and all of that. There's some inversion of logic of who becomes the
master and who becomes the driver of the graph and all of that but this is going to be used in these
scenarios. So more things now where we are. So cameras are moving to pipe wire as well.
So in firefox it's already done. Don't exactly implement this in firefox but in web
party C and that piece of code should be shared with a Chrome. So eventually it will find its way
in Chrome as well. Yeah that's the answer. Not an L by default yet but it will at some point
be I guess. So this is good because then we can move everything of the camera handling to lip camera.
We can also start sharing the cameras. We can also add permissions for new in the access control
a little bit more fine grain but most important things newer Intel ships. You can't just
get the frames easily. You first need to configure it to media control the whole pipeline of
the camera and the processing units before it actually wants to do something. So firefox by default
wouldn't work on that. So by switching to lip camera and pipe wire you make this work.
Matter. Yeah so 1.4 is almost done. I made a pre release a couple of weeks ago.
Yep so what else is there to do? Not so much but we have media too.
So media is very old. It's used for synthesizers to send the notes that you do on the
synth over a wire to a computer but these days it's mostly also used for triggering samples and
doing interactive stuff. It's pretty good as an event mechanism and there's a new spec out
which adds more resolution to the pressure of the keys and you can use
non-standard tunings and all of these things. So there's also a new media format in the kernel and it
was added and those are people they'll all work to make that work. So nobody was using it so
finally we have a use for this. So basically we have everything in pipe wire now doing with
media is now implemented as the media 2 standard which is a packet form of called UMP universal
media protocol or universal media packet I think it is and you can just package standard media
or the newer version of media in it. So it's not too hard difficult to change the alls iAPIs to
use this. But we do a conversion between older and newer things it's quite trivial with these things
but it needs some time to figure out who converts what and all the possibilities.
What do you have as a concrete benefit with this don't know because the media 2 devices themselves
they don't really exist or they are very expensive. We'll see. So together with this there is also
an extension for the jack API. So most of the media tools that you can use on Linux today use the jack
API. So there's an extension for them to also use media 2. So hopefully this will become a thing.
So more of the video conversion that I talked about there's some experimental stuff we have
come back to convert stuff especially for the coding of jpec frames and all of that we use that
it works. Not enable by default but I think one of these next three is we will do that and then
you get the better use case for using videos currently it's not very flexible
you know this is I can actually try this let's see what happens to show you what i'd still
running here the camera are great. I think i can can i i have to stop the slides
so usually if you do for example something like this there's an example program
where am i gonna do this here. There's an example program here to take a frame or video source
maybe i should enlarge this i
called video play you can run it's unfortunately
oh i don't see anything of course so unfortunately the tool can only display a compressed
formats so because the camera is now using use already with by Firefox doing mjpec this thing just
fails to connect or do anything so with this thing here putting a video converter there
i'm pretty sure that i probably need to do something else here
i probably don't have this compiling i should have tried this anyway what you could have done there
is insert the video converter and then have the client consuming the thing
converting it to something it does understand this is not exactly what we want to do in the end
we want to have only uncompressed video in the end like we do with audio
okay so some of the else that we've been doing so one of the questions or use cases that people
have is okay i want to add an equalizer to all of my output or some sort of processing or for
example for my camera or microphone i want to add noise suppression something global
the way you should do that right now and we have been doing a whole bunch of things
to make that work is by inserting inside the graph in front of the the things
or so after the sources processing note
the problem is you need to have everything go through that both processing no buy pass it
because then you lose the advantage of the filter and if it's something that is mandatory like
limiting the volumes so that you don't blow the speakers it's not so very safe if you can buy pass it
so another idea that that was brewing was adding arbitrary processing in all of the notes
you need to understand a bit of things work for that so if you have pipe wire the basic IPC
for sharing audio you have something that produces the audio or the audio in this case and
something that consumes like for example and also sync and you have an app that produces audio
the component of the pipeline provides called a stream and internally it has a little pipeline
of stuff that needs to happen like format conversion because most apps and i don't know
16 bit of something we need to convert that to something that is understandable for
the graph it needs to be an example and all of these things if necessary or maybe even up or
done mixed so we had already a little graph pipeline of things that needed to happen so now
there's another thing you can add there which is a filter graph which is an arbitrary processing
you can perform so you can do this almost for itself or directly in the sync so there's a good place
to upload for example equalizers or noise suppression and your guarantee to always have these
active they're not bypassable by making links in a graph currently not used yet but
i'll can show a little demo for what that could be so a filter graph you currently have to use
something like this to specify what happens but you can say a couple of for example lots pop plugins
for peak detection and a limiter for example you can define those notes you can link them together
the output of this into that and then the inputs and outputs of the whole graph
and the inputs of that and the outputs of that so it's a little too component
filter and you can upload that anywhere and it will apply these effects
so what can you do with these things we have we have this already in a module but now it's more
generalized you can also use it in all places so there's a whole bunch of things you can do
except for the lots pop plugins or the other two plugins there's some built in things too
so for example you can upload this like this i'm going to try to show that as well
so for any of the notes you can set the property with this with the filter graph you can
up up to nine of them with the definition this is a very stupid one with just one filter
and it's complicated twice for stereo and I see if I can make this work here
so I'll see if I can make this bigger as well
come on
so a little bit of demonstration here what you can see so if you have this thing
usually I have one note here the player this is the the little player thing it
it goes to a default sync which is now set to this virtual sync here a loopback device
so this has an input and also an output they are not very linked together here in this graph
but here is the output and that's an eventually goes into the hardware which is this headphone
things so you have a bunch of notes and these things the sync here and the loopback things and
all of these things are all streams so you could you could have a look after that for example
the tools we use for this it's a PW top so here's the player you can now inspect a little bit
what form of cities using natively those are these filters the loopback filters and that's the
output device they have a number as well an idea
so you could for example now say let's say this one which has 61 was a number
I have this in my history
I don't know if you're gonna hear this but now there's a flanking effect on it
you can remove the filter again of course
oh this is off a spec
still wrong 61 and now I'm still with my bracket for
oh no
So this is something I don't know.
It doesn't change the graph, so I think it's better.
But you can, for example, also do other things.
Like you can insert the filter in the player itself.
It doesn't have to be the sink.
It doesn't work for video yet.
So it would be nice if all of these things would, of course, work for everything.
Yeah.
The pool.
So what's next?
What's up coming?
Why doesn't this go away?
Go away.
No.
Does it want to escape?
Oh well.
So what's up coming still for Bluetooth?
So there's an hearing aid support finally.
There are some standards for that.
One of them is Asha, which is supported by Google.
You have a supported hearing aid.
You can connect to it's private protocol.
What private?
It's a specific protocol.
There's also something more standard to upcoming for Bluetooth, L.E.
Oh, thumbs up.
Yeah.
We fix bugs and things will work better.
Quick questions.
Four minutes for questions.
Yes.
Do we have an external party with sampler or do we have our own?
Yes.
How can we use a private protocol?
Yes.
How do you make part why I put two multiple headsets?
Well, the easiest thing is to draw a line.
You know, from your output application, you drag your line to another sync.
And then it will play on two at the same time.
You can really do it because I want to have one.
You can also make a more automatic by using a filter that will automatically do that for you.
Yeah.
If you use a loop package device, that's like an introduced latency.
No.
So all of the notes normally in the graph produce no latency.
So there is the start of the also device that says our 1,000 samples.
1,000 samples are gotten from the player.
1,000 samples given to loop back they come out.
And then they go to the sync.
And then the cycle stops.
So it's loop.
One go.
Okay.
How does pepper approach audio interfaces that have more than two channels?
Because I've made the experience that the first two channels are taken.
Because maybe stereo is expected.
And the rest will need more.
You have to configure it.
Yeah.
So how do you work with devices with multiple channels?
So all of the configuration of the also devices is done with the old code from Pulse Audio,
which sort of does auto detection of stereo and multi channel things.
But it has its limitations and it doesn't find everything.
In Papua there's also a pro audio mode, which basically just opens the also device and gives you everything.
But it doesn't try to do anything fancy with volumes or mixers.
It's just a raw thing.
So if you talk about that and if you configure it in pro audio mode,
we only assume the first two channels are stereo and the rest you don't know.
So like for example, the session manager wire plumber will not root anything to the other ports,
because it doesn't know how deep it is.
Just assumes the first two are stereo.
So if you want to give that a meaning,
you have to place a sink in front of it, a stereo sink,
and then route it manually to the other ports.
Yeah.
There is some work being done to automate that,
because also has a configuration file where you can set these things,
and then we can automatically generate something more sensible.
But the config file has to be there,
and it needs testing.
Yeah.
Thank you very much.