So, hi everyone. Welcome. Thanks for being here. This is my first presentation. I hope
you don't want to mess it up. The reason I wanted to give it is because I, sorry.
So, the reason I wanted to give this presentation is that, first of all, I wanted to present
mirror hall, which is this application to use other Linux devices as kind of peer-to-peer
extended monitors over the network. So, you like use them as virtual monitors. And the reason
I wanted to present it was because I think basically how it was built this kind of
layer and it's faster to go a bit deeper into it and we explain how to basically
remake your own virtual display sharing solution at home. So, I just wanted to start with
a bit of history saying that when in the 90s, personal computers became more popular and people
started promoting mobile devices and mobile development. One of the core ideas that
works, which was the research institute, then Apple got the ideas from that long story.
It was to get those devices to seamless interact with each other. So, basically, you
had a portable device that could be connected to a larger one to use the larger one as a
monitor and all that kind of stuff. Why did that never happen? Why are devices still so hard
to interface with each other? That is because in the state never favored standards for peer-to-peer
communication, for open communication between proprietary platforms, but everyone went for their own
personal implementation in a way, so convergence and peer-to-peer sharing and wireless
desktop sharing never really became a unique thing. So, to be fair, we have a lot of solutions
that work really well for simple wireless desktop mirroring or Linux, such as Moonlight or
Sunshine, which are mostly meant for games. I never used them, I just know they're good.
No network displays, which I did use, and it's pretty good for like Miracast Chromecast. But
we don't really have anything for more interesting kinds of mirroring. So, first of all, I think
we're familiar with how mirroring works on the other hood, so it's basically, I have my
screen here. This is a display buffer. We record that screen. We reuse the same buffer.
We encode it, we transmit it over the network, we play somewhere else. But what if we instead
wanted to not mirror the same screen, but mirror another display buffer, then in that case,
we basically have to interact with the backend, and to collaborate with the backend, which
everything is to create somehow a virtual display, a virtual buffer, and then that is the one
we aim to record, the one we aim to stream. And that is something that is not clearly
standard. So, one solution that kind of does that a proprietary solution is a put side
car, which also never used, but it's kind of you extend your max screen, you use your
iPad as a second screen, some iPads, because you need expensive ones. But, during this,
only nooks using existing solution would have worked, because for example, all those protocols
are proprietary, so they are not easy to implement, and they also pretty, you also have pretty
high latency, because most of them use TCP. Most of the implementations are also under receiving
and, like, on your TV, are like software encourage, so there are a lot of latency, because it's
more important to get, like, accurate video normally than fast video, and what if I wanted to
kind of re-opuse it. And also, it's really easy to stream from Linux onwards, you record a screen
you stream it, but it's really hard to then become a receiver for that stream currently,
because for example, to do it on my podcast, you need very strange, like patches, to when you need
to, like, stop Wi-Fi, to set a paddock, very, very complicated. So, starting from the virtual
mirroring we saw before, I was wondering, like, what happens when you scale it up, like, what happens
when you have different monitors, different devices? Can you still do some kind of virtual virtual
screen buffers that really depends? So, if we have available video buffers, if we have available
hardware encoding buffers, because for example, when you use your hardware encoder for video,
usually you can't really put more than one stream at once, so you have to do, like, software
encoding for some and hardware encoding for one. And what if we're going on for a time when to make it,
like, bidirectional, so that we can turn every device into both a receiver and a streamer,
so that I can just decide. And that is basically what I wanted to do for a long while, so in 2020,
known for it, introduced a really cool, headless native back end, and virtual monitors API, which is
basically you ask matter to create a virtual monitor on my land. It does it without any kind of
packs and you can have stuff. I made my first prototype in 2022 using those APIs and the bunch
of Python stuff, and that was, like, I was using my Librum 5 as our wireless screen for my,
like, our wireless extension to just have it on my keep track of stuff. And then I made the first
setup in November 2020, 3 and the first release just a month ago. So, basically, the mirror hole
looks like this right now. I have to be really honest with you, I break my test like my test
tablet this morning, so I have an alternative one, which I hope works as well. But basically,
this is how it looks. I open it here as well. It uses MDNS internally, the same protocol as, like,
the LNA, I think, Chromecast for sure, like, so right now, I basically turn this device into a mirror,
it's kind of not very practical to be honest, but oh god, I hope this is setup correctly, because I'm
not, no, there's no network. Never mind, so this device apparently doesn't, it doesn't
have a working network, but it had 30 minutes ago when I flashed it, so anyway, let's do a very
boring thing then. Let's spawn up another instance from must, I hope this works a bit better.
And in theory, if I do this, we should be able to get something out of it.
Okay, this is not really clear. What's happening? Because that's the same device.
I'm really sorry, okay, this was not really, but basically what we have here is that we can kind of,
kind of, yeah, okay, you see what's happening, it's like, it's literally mirroring itself
and look back at mode, which is not very meaningful, I want it to do it better, but I'm sorry,
I'm also trying to predict it. So you didn't succeed. Thank you, matter, those are very,
very experimental APIs for a reason. Can I, can I, can I mirror, oh god, it is a mirror mode, but it's
not really, yeah, okay, this is the only way I can do it, apparently. So you didn't see anything,
basically, you didn't see, okay, yeah, this was, this was the idea, like you can basically switch
between it and now it went somewhere, somewhere in the, what, what the, what the actual, okay,
well, like this is not, what the fuck? Like this, this is right, okay, I never tested this
presentation using multiple screens and doing something that this is the multiple screen API to
test it, maybe it wasn't, okay, anyway. So the way this works is basically that we have
a mirror hall, which is this player and streamer app, which is just dedicated, you just
and below we have three small libraries that are lip mirror that detects the desktop environment
that you're using and uses the bus to communicate with matter, it was to negotiate a virtual
monitor to create it and then gets a pipe wire stream back. Then lip cast basically has a tiny
database of pipelines so that it can use hardware acceleration you're device to generate a fast
including pipeline and we use all UDP packets, not RTSP, not any, like kind of wrapper protocol
because it's way faster I know and then it basically takes the stream that you just created and
feeds it to it, okay, yeah, yeah, okay, you can see it. And then we have lip network, which instead
does stuff like MDNS, so the thing that you can see the device in the list, when you in theory,
when you plug it in you also sit there, but the network interface is down, basically that would be the
idea. So that's yeah, basically lip, a debust to know, to matter, get back the stream through
pipe wire, then you encode the stream using a pipeline and you transmit it there and the receiver
part is a bit simpler because it's basically a video player, it's basically a network video player,
you can also use RTSP as a fallback, which you will see in a second, thank you. And yeah,
because we also thought if you really don't care about latency, if two seconds of latency is enough for
you, then RTSP can also work and it was like a CLI so that you can basically type of very long
G-streamer launch command without having your whole installed and use a device as a receiver
versus H without installing anything. Okay, so now for the more technical part,
like I wanted to go a bit deeper into how this worked, I didn't want to talk about NDNS that
kind of stuff, but just about the fact first all that is only support smuster, unfortunately,
because as far as I know, there were no other standardized API except maybe in Swade,
and KD6 apparently someone told me has introduced something similar or is in planning too,
so but in theory, like once the RAPIs, his library is really easy to extend because it's basically
like creating more debust calls, detecting your configuration and creating more debust calls.
So the way this works and yeah, matter, I think we have a tiny bit of time, but you don't see
my screen anymore, which is not great. It's basically here we would have this because I can't see my screen.
Can we make this larger? Okay, we have a matter screen cast API and here there should be
screen cast, so what we basically do first of all is we create a session.
It will probably want some parameter, but I think it's it was an empty array here because it's
like the array of options you're passing in. So here we receive back the name of the object that
we're using and then we kind of go back to that object that we just received, which in this
application is a bit obvious, we have to scan cast again and you see this newly made object and
here under session we can do a record virtual call that will create basically a string object
and then we could already start the string. The thing is if we do we wouldn't intercept the event
that tells us where where the pipe wires, when the pipe wires turn appears. So to do that we
basically have to intercept using the bus monitor and in theory now if we go back to the
session that we created and we should receive a pipe wire stream here and you also see this thing
this yellow thing which means basically that we are casting the screen. Okay you see now we
received an event here and if we use a pipe wire SRC which is the one of the one component in
it. So we saw just now that the event was what was it 115 something like that 105? In theory
here's our virtual display. So now you kind of see the loopback thingy again and yeah that's
basically the interesting part so it is not too hard to get a virtual display out of matter and when you
then also like yeah that's that's basically it. Then once we have it basically we create this
system of pipeline that we saw which if we if we divide it in components just we have the source
which picks up the element then we have Qs for buffering then we use in this case X264 but again
it's chosen by the like X264 on Intel right now on this specific build but the
basically if you are using arm it will try to pick up better one and then we after we
package it after we encode it into a format we package it into with the payload encoder and when
then we send it in this case using UDP sync and yeah that's instead the pipeline that we just use
further demo is just this simplified one that just uses thank you video convert and auto sync because
that's easier and the other way around is also very simple so you can also do it from the CLI and you
basically see the command to also replicate it outside the app so it's just basically the same
components the opposite direction UDPs and auto source and so on so yeah this is something I honestly
just kind of talked about so maybe it's not super important but we yeah do the harder configurations
we use there should be a priority for like better over worse encoders and decoders which we don't
do right now and right now we're also capping for example the same quality because we can't really
negotiate it in real time like a really nice video protocol but we're just kind of guessing the
best configuration and we are using some additional configuration parameters to minimize the amount
of computation effort that has to be done to encoding each stream so it's maybe slightly larger
but also like or if sorry slightly less precise but also faster if you have many terms at once
and also right now are very big limitation that is I think kind of annoying to most people
is that you have to manually patch your firewall to use it because you need to authorize a bunch of
UDP ports that is something that the flat pack currently doesn't do and yeah another very big
very interesting limitation is that as soon as I release the 0.1.0 there was a regression in
known because nobody has used this API yet that made me a whole crash matter which was quite interesting
like I was like okay finally I have a very reason that was like one week after the yeah so I had
what that very scary warning and I don't really yeah recommend to use it in production stuff until
we're sure it's stable we're sure the API is very tested and so on so for the next steps I think
what's important will be to kind of first of all I think there are many things at once and we it
would be a bit more manageable if we split it up because maybe many people don't recover the UI
but care about only the heavy gun end point that basically has a helper to create a virtual display
on each platform so we ideally want to split that up we want to add encryption that's very important
because right now this term is not encrypted so it's fine if I connect it via USB but if I use
MDNS over the network it's a bit more I don't know I wouldn't do it and also we would like
apparently there is a lot of the headaches a lot of the heavy lifting of punching holes in UDP
trying to find the right route on the network and so on would be solved by using a pretty good
stack in Rust that is I I don't know how you pronounce it which is basically yeah
peer to peer on UDP with automated whole punching and so on and also to be really nice to
input methods in some way proxy the input method from device to another but that is very I
really like I looked it up and for now it will take a lot of time so that's all yeah I want to help
to thank all people who help me so that's the scale up there there's Sony there's a few of the
known people there's people I chat it with who help me like understand the whole pipeline thing
and if someone is interested in this please let me know because I don't really have a lot of time
so to work on it so like I'm just very slowly developing but it's someone is like you know
it's in testing it on strange platforms and so on that'd be really nice just let me know if it
breaks white breaks so yeah we can also meet in Berlin if you want and that's like this
preferences and I have 30 seconds for questions no yeah 0 seconds okay if you want
yeah so basically we're using RTSP right now it's a can you repeat the question
I'm sorry I was like okay so the question was whether we can use Chromecast to repeat
to stream this like to use a fallback sync for Chromecast and the answer is yes you can do that
it would be very slow like probably one second this latency but you can do that do it yeah and that is
well one of the reasons why I wanted to split it like in the part that creates this plan the other
yeah so