So, this thing works, but just for the lack of time, I don't know if I can do it, but I can't do it.
So, this thing works, but just for the lack of time, I don't know if I can do it.
So, this thing works, but just for the lack of time, so I should speak loud.
I don't know if I can do it.
So, this thing works, but just for the lack of time, I don't know if I can do it.
So, hi everyone, welcome, thank you for being here.
This is my first presentation, I hope not to mess it up.
The reason I wanted to give it is because I'm sorry.
So, the reason I wanted to give this presentation is that first of all, I wanted to present mirror hall, which is this application to use other Linux devices as kind of peer to peer extended monitors over the network.
So, you like use them as virtual monitors.
And the reason I wanted to present it was because I think basically how it was built this kind of layer, and it's faster to go a bit deeper into it and explain how to basically remake your own virtual display sharing solution at home.
So, I just wanted to start with a bit of history saying that when it is the 90s, person computers became more popular and people started promoting mobile devices and mobile development.
One of the core ideas, the Xerox, which was the research institute, and then Apple got the ideas from kind of that long story, was to get those devices to seamlessly interact with each other.
Basically, you had a portable device that could be connected to a larger one to use the larger one as a monitor, all that kind of stuff.
Why did that never happen? Why are devices still so hard to interface with each other?
That is because industry never favored standards for peer-to-peer communication, for open communication between proprietary platforms, but everyone went for their own personal implementation in a way.
So, convergence and peer-to-peer sharing and wireless desktop sharing never really became a unique thing.
So, to be fair, we have a lot of solutions that work really well for simple wireless desktop mirroring or Linux, such as Moonlight or Sunshine, which are mostly meant for games.
I never used them, I just know they are good.
And no network displays, which I did use, and it is pretty good for like Miracass Chromecasts.
But we don't really have anything for more interesting kinds of mirroring.
So, first of all, I think we are familiar with how mirroring works on the hood, so it is basically...
I have my screen here, this is a display buffer. We record that screen, we reuse the same buffer.
We encode it, we transmit it over the network, we play somewhere else.
But what if we instead wanted to not mirror the same screen, but mirror another display buffer?
Then in that case, we basically have to interact with the backend, and to collaborate with the backend, which everything is to create somehow a virtual display, a virtual buffer,
and then that is the one we aim to record, the one we aim to stream.
And that is something that is not clearly standard.
So, one solution that kind of does that proprietary solution is Apple's sidecar, which also never used,
but it's kind of you extend your max screen, you use your iPad as a second screen, some iPads,
because you need expensive ones.
But during this only looks using existing solution, we wouldn't have worked, because for example, all those protocols are proprietary, so they are not easy to implement,
and they are also pretty high latency, because most of them use TCP, most of the implementations are also on their receiving end,
like on your TV or software encode, so there are a lot of latency, because it is more important to get accurate video,
normally than fast video, and what I wanted was kind of the opposite.
And also it's really easy to stream, formerly in some words, you record the screen, you stream it, but it's really hard to then become a receiver for that stream,
currently, because for example, to do it on my request, you need very strange like patches, you need to stop Wi-Fi, to set a paddock, very, very complicated.
So, starting from the virtual mirroring we saw before, I was wondering, like, what happens when you scale it up like what happens when you have different monitors, different devices?
Can you still do some kind of virtual virtual screen buffers that really depends?
So if we have available video buffers, if we have available harder encoding buffers, because for example, when you use your hardware encoder for video,
usually you can't really put more than one stream at once, so you have to do like software encoding for some, and harder encoding for one.
And what if we're going for another one to make it like bidirectional, so that we can turn every device into both a receiver and a streamer, so that I can just decide, and that is basically what I wanted to do for a long while.
So, in 2020, known for it introduced a really cool, headless native back end, and virtual monitors API, which is basically you ask matter to create a virtual monitor on my land.
It does it without any kind of hacks, any kind of stuff.
So I made my first prototype in 2022 using those APIs and a bunch of Python stuff, and that was back then, like I was using my Librum 5 as our wireless screen for my wireless extension to just have it on my keep track of stuff.
And then I made the first setup in November 2023, and the first release just a month ago.
So, basically, the mirror hall looks like this right now.
I have to be really honest with you, I break my test like my test tablet this morning, so I have an alternative one, which I hope works as well.
But basically, this is how it looks, I open it here as well. It uses MDNS internally, the same protocol as like the LNA, I think, Chromecast for sure, like, so right now, I basically turn this device into a mirror, it's kind of not very practical to be honest, but oh god, I hope this is setup correctly, because I'm not in turn, no, there's no network.
Okay, never mind, so this device apparently doesn't have a working network, but it had 30 minutes ago when I flashed it, so anyway, okay, let's do a very boring thing then.
Let's spawn up another instance from last, I hope this works a bit better.
And in theory, if I do this, we should be able to get something out of it.
Okay, this is not really clear what's happening, because that's the same device, I'm really sorry, okay, this was not really,
but basically what we have here is that we can kind of, kind of, yeah, okay, you see what's happening, it's literally mirroring itself and look back more, which is not very meaningful, I wanted to do it better, but I'm sorry.
So you didn't see shit, thank you, matter, those are very, very experimental APIs for a reason, can I, can I, can I mirror, oh god, it is a mirror mode, but it's not really, yeah, okay, this is the only way I can do it, apparently.
So you didn't see anything, basically, you didn't see, okay, yeah, this was the idea, like you can basically switch between it and now it went somewhere,
I mean, what, what the actual, okay, like this is not, what the fuck.
Like this is right, okay, I never tested this presentation using multiple screens and doing something that this is the multiple screen API to test it, maybe it wasn't, okay, anyway.
So the way this works is basically that we have a mirror hall, which is this player and standard app, which is just GTK that you just have.
And below we have three small libraries that are lip mirror that detects the desktop environment that you're using.
And uses the bus to communicate with matter, it was to negotiate a virtual monitor to create it and then get a pipe wires turn back.
Then a lip cast basically has a tiny database of pipelines so that it can use harder acceleration, you're a device to generate a fast, including pipeline and we use all UDP packets.
It's not RTSP, not any kind of wrapper protocol because it's way faster, I know.
And then it basically takes the stream that you just created and it feeds it to it, can I? Yeah, yeah, okay, I can see it.
And then we have lip network, which instead does stuff like MDS, so the thing that you can see the device in the list, when you infer, when you plug it in, you also see it there, but the network interface is down.
Basically, that would be the idea.
So that's, yeah, basically, lip, a debust to know to matter, get back the stream through pipe wire.
Then you encode the stream using a pipeline and you transmit it there.
And the receiver part is a bit simpler because it's basically a video player.
It's basically a network video player, you can also use RTSP as a fallback, which you will see in a second, thank you.
And yeah, because we also thought, if you really don't care about latency, if two seconds of latency is enough for you, then RTSP can also work.
And it was like a CLI, so that you can basically type a very long g-streamer launch command without having me or holding installed and use a device as a receiver versus H without installing anything.
Okay, so now for the more technical part, like, I wanted to go a bit deeper into how this worked.
I didn't want to talk about an DNS that kind of stuff, but just about the fact first of all, the disownly support smother, unfortunately.
Because as far as I know, there were no other standardized API, except maybe in sway.
And KD-6 apparently someone told me has introduced something similar or is in planning too, so, but in theory, like one star APIs, this library is really easy to extend, because it's basically like creating more device calls, detecting your configuration and creating more device calls.
So the way this works and yeah, I'm not sure if we have a tiny bit of time, but you don't see my screen anymore, which is not great.
It's basically here, we would have this, okay, because I can't see my screen.
Can we make this larger?
Okay, we have a matter screen cast API, and here there should be screen cast, so what we basically do first of all is we create a session.
It will probably want some parameter, but I think it's it was an empty array here, because it's like the array of options you're passing in.
So here we receive back the name of the object that we're using.
Then we kind of go back to that object that we just received, which in this application is a bit obvious, we have to scan cast again.
And you see this newly made object, and here under session we can do a record virtual call.
That will create basically a stream object.
And then we could already start the stream.
The thing is if we do, we wouldn't intercept the event that tells us where the pipe wires, when the pipe wires stream appears.
So to do that we basically have to.
Okay.
Intercept using the bus monitor.
And in theory now if we go back to the session that we created, and we empty it.
We should receive a pipe wire stream here, and you also see this thing, this yellow thing, which means basically that we're casting the screen.
Okay, you see now we received an event here.
And if we use a pipe wire, SRC, which is the one of the one component in.
So we saw just now that the event was, was it 115 something like that?
105.
In theory, here's our virtual display.
So now we kind of see the loop-play thingy again.
And yeah, that's basically the interesting part.
So it is not too hard to get a virtual display out of matter.
And when you then also like, yeah, that's basically it.
And once we have it, basically we create this system of pipeline that we saw, which if we divide it in components, just we have the source, which picks up the element.
And then we have cues for buffering.
Then we use in this case X264, but again it's chosen by the X264 on Intel.
Right now on this specific build, but basically if you are using arm, it will try to pick up better one.
And then after we package it, after we encode it in the format, we package it into, with the payload encoder.
And then we send it in this case using UDP sync.
And yeah, that's, instead the pipeline that we just used further demo is just simplified on the just uses thank you.
Video convert and auto sync, because that's easier.
And the other way around is also very simple.
So you can also do it from the CLI and you basically see the command to also replicate it outside the app, so it's just basically the same components, the opposite direction.
UDP is an all the source and so on.
So yeah, this is something I honestly just kind of talked about, so maybe it's not super important, but we do the harder configurations we use.
There should be a priority for like better over worse encoders and decoders, which we don't do right now.
And right now we're also capping for example the steering quality because we can't really negotiate it in real time, like a really nice video protocol, but we're just kind of guessing the best configuration.
And we are using some additional configuration parameters to minimize the amount of computation effort that has to be done to encoding each stream.
So it's maybe slightly larger, but also like, or if necessary, slightly less precise, but also faster if you have many streams at once.
And also right now, a very big limitation that is I think kind of annoying to most people is that you have to manually patch your firewall to use it,
because you need to authorize a bunch of UDP ports.
That is something that the flat pack currently doesn't do.
And another very big, very interesting limitation is that as soon as I release the 0.1.0, there was a regression in no, because nobody has uses this API yet that made me or whole crash matter, which was quite interesting.
Like I was like, okay, finally I have a very reason that was like one week after that.
So I had what that very scary warning and I don't really recommend to use it in production stuff until we're sure it's stable, we're sure the API is very tested and so on.
So for the next steps, I think what's important will be to kind of first of all, I think there are many things at once.
And we, it would be a bit more manageable if we split it up because maybe many people don't really care about the UI, but care about only the heavy gun end point that basically has a helper to create a virtual display on each platform.
So we would ideally want to split that up, we would want to add encryption, that's very important because right now this term is not encrypted, so it's fine if I connect it via USB.
But if I use MDNS over the network, it's a bit more, I don't know, I wouldn't do it.
And also we would, like apparently there is a lot of the headaches, a lot of the heavy lifting of punching holes in UDP, trying to find the right route on the network and so on.
I would be solved by using a pretty good stack in Rust that is I, I don't know how you pronounce it, which is basically yeah peer to peer on UDP with automated whole punching and so on.
And also to be really nice to add input methods in some way, proxy in the input method from device to another, but that is very, I, I really like I looked it up and for now it will take a lot of time.
So that's all, yeah, I want to help to thank all people who helped me, so that's the scale up there, there's Sony, there's a few of the known people, there's people I chat with who helped me, like in understand the whole pipeline thing.
And if someone is interested in this, please let me know, because I don't really have a lot of time, so to work on it, so like I'm just very slowly developing, but if someone is like in there is in testing it on strange platforms and so on, that would be really nice, just let me know if it breaks white breaks, so yeah we can also meet in Berlin if you want.
So that's like just preferences and I have 30 seconds for questions, no, yeah, zero seconds.
Okay, if you want to.
Yeah, so basically we're using RTSP right now as a, can you repeat the question?
I'm sorry, I was like, okay, so the question was whether we can use Chromecast to repeat to stream this like to use a fallback sync for Chromecast and the answer is yes, you can do that.
It would be very slow, like probably one second this latency, but you can do that, yeah, and that is one of those I wanted to split it like in the part that creates this plan the other way.
Thank you very much.
And other thing related to it, why super as a protocol is not private and how one can work around it.
And at the end also better about how open nature best works and very currently are with the project.
So the quick super TLDR, but what it is is it's an open source genesis assistance data server, which crowdsources, genesis frame data along with the ability to ideally better aggregated from other sources, including potentially official ones in the future, maybe.
The goals are to have implemented a super compatible server for a GPS use or a genesis use and to be able to access a data source either for other projects that may be interested in using this data or for people who just self host their own super service as well.
How it came to be is interesting out switch to a bunch of touch a while ago and then noticed that GPS was very slow.
I found lots of foreign posts talking about how the reason why it was very slow was that it's a GPS was not supported on a bunch of touch.
While the reasoning for while it didn't work is a bit complicated, there are a few core issues for people who use foster devices or foster open systems or just want a fully private fully open source system.
Previously being a main model that I'll go into a bit along with potential licensing issues if you have, for example, a highly unbased OS or running postmarker OS because super google.com, which is the main one that Andrew uses, his mind standing is licensing is a bit finicky if you don't have Android.
That also goes straight into the fact that the main super service in the Western world is added to China is at this point run by Google or local governments.
There's also no existing fossil implementation for this so basically have to start fully from scratch for this one.
How Genesis works, skipping over a bunch of complicated physics stuff, basically each satellite is a very precise clock, where the positioning of a phone or a handset of receiver is done via time of flight from signals from each satellite.
So each satellite knows where it is, it transmits that and the current time very precisely to a device and you can calculate that.
The data need for the femoral and the ammo knock data, which is basically data such as where the satellite is, the current correction parameters needed and other physics stuff.
Transmitting this is very slow, each satellite signals are designed for signal integrity and to put a punch through things, not for speed, which results in a frame speed of about 25 bits a second, for example, for GPS, specifically.
This is system development.
Nowadays people have made systems, it's system systems, which download the state of an internet, which is a lot faster.
There is a standardised protocol for doing this over the internet specifically, it is the secure user plane location protocol.
It has heritage from mobile data protocols from our OP and things like that.
This needs an approximate receiver position, so it needs to know which satellites are approximately in view, so normally sound towers are used for positioning in this case.
This is also where more cellular network character comes into this being case.
Also straight into the privacy issue number one for SIPL, which is that the protocol sends the iMSI number of the SIM card, which is unique for each SIM card.
So this just goes straight into the fact that it's not designed to be private.
It's designed to be not private specifically.
The iMSI number is often identified to a singular person.
While most super or for this particular instance, ROP connections, which is the name of the protocol, if being done over purely a cellular network, not over the internet, are done by the handsets of the phone.
It was for the network to initiate connections as well, which is what I'm used by emergency services to get a position of a specific receiver.
For example, if someone has called the police or fire ambience and doesn't say where they are, the police can then determine the location for it.
It also means that you have to trust the government too.
Always use this for good reasons.
Also, knowing people here, the fact that super the google.com is the main domain used probably is also not optimal.
Now, sadly, you have to learn to work around these things, because super is here to stay.
There is really to be speaking at least for devices that have closed source GPS drivers.
So anything that runs Halyam, for example, it is here to stay.
Because it is industry standard, basically everything supports only it, which means you have to support it as well.
The assistance requests are additionally sometimes made on the Android phones directly by the chipset,
which means that not even the OS that's running can always see the requests being made, which is also a quite scary thought.
There is also the fact that the chipset firmware itself is almost always going to respond to super requests,
that the network for legal reasons, so that the police can determine where the phone or other thing is at any time.
The workaround for this would be to have a fully open source super stack with the ability to self host as well,
so that for the most paranoid of people we could, for example, potentially you can do something sort of host the server,
the super server itself on your own phone, maybe or on your local network,
and have it to get data from somewhere else, so that you know exactly what source code is running,
and that nothing is being intercepted by anyone as well, which is especially good because at least some Chinese phones
have been known to strip with TLS verification off for super, which is cheating the amazing.
The way that OpenH GPS specifically works is that satellite data is crowdsourced for base stations.
We could also import or collect data from other projects, specifically trying to work with Gaomon,
which is an existing project made to track, specifically reaching Gaulet layer,
but now generally all GNSS satellites and orbit, and already has an existing network for this,
but person time is a bit very busy at the moment, so we haven't gotten that to work quite yet.
The base stations basically collect the GNSS frames.
They just sit there listening, the positioning they're doing isn't important more than the data that they're getting from each satellite is.
This would then be forwarded by a two a server, which would either be able to directly handle super requests
or via other project protocols that will make you slightly more modern,
be able to forward the raw satellite frames to something else to be able to, for example,
to use a data for their own users or host the service themselves.
So access the data source, basically.
At the moment, all of the code we've written is specifically designed to be stateless,
except for the super code, because super as a protocol is not stateless, sadly.
All of the base stations push requests are done over HTTPS.
Well, at the moment, the base station code itself doesn't specifically mandate TLS, we do intend on doing this.
It's just easier for development at home to have just fast API dev.
In addition, the fact that it's always wanted to say this means that containerizing things,
which would be important for if we have potentially have to figure out ways to put servers or
some of the servers around the world, so that people in America can get a reasonable time of acquisition.
In addition, we've also designed things for the idea that potentially you could make the service handling,
super data disclos.
The same thing, in that, for example, certain VPN companies have disclos servers,
but this isn't a present goal at the moment.
It just would be a nice, if you could manage that.
We haven't fully decided on what infrastructure we're using, but in case anyone's interested,
at the moment we are co-leasing MariaDB for storing various pieces of data.
The back end, which is at the moment just a codename power gun, is written in Python.
And the base station code is written in C++.
At the moment, we're using ESP32C3 to connect to a new blocks receiver, specifically.
The base station code at this point is basically complete.
It works reasonably well.
It's basically designed to be as cheap as possible.
If you want to find your cheapest things from AliExpress, you can build one of these things back 15 years if you want.
Approximately.
The back end isn't done yet.
It's not even close to it.
But it's work has begun.
It will be back end include the super server and basically the genus as data collection server.
We originally hoped to have a very basic super server ready for a demo.
Here, we were a bit too optimistic on that timeline.
Have, yeah, getting such a quick 30 time as an having university didn't have.
We still have a demo though.
It's just not as impressive as one, as I originally hoped.
So, I got after.
Basically, what I have here is the features of your code day.
And if I can get a GPS reception in here, I should be able to show the base station booting up and sending frames to my laptop, which is running the back end.
But I'm not sure I'm going to reception in here.
It doesn't seem like it, well, I'll try it.
Yeah, it's really not going to happen.
Well, seemingly not there, you don't know that it didn't work today.
We'll allow for going to questions I'd like to specifically thank the end of that foundation for giving this project a grant.
And from the NGO, maybe if you found, along with Leath Cloud, a Dutch hosting company, which is sponsored some server infrastructure for this.
Welcome to an also mentioned that we have a website at OpenHepest.net.
It's very much not complete.
And it's basically just a single blog post to the exists.
The source here is available on GitHub, under the OpenHepest group.
If you want to contact me, you can do so either on Matrix, on the lecturers, from Matrix.org.
This is called with a lecturer, or you can send me email address at lectured.gmail.com.
So, I'd like to thank all of you for your attention.
I'm open to any questions you might have.
Thank you.
Hi.
I run in very simple.
It's kind of what I spy across the West, owned, and everything's in Flash, and it's a very spade.
I'm in GNSS, literally.
I might be able to buy a little patch on a server name and launch it towards your server.
So, the question was, could you, it's really possible to binary patch a, basically,
a mobile handset that can not have configuration changes will be rooted to your users.
Sadly, probably not.
What you could potentially do, I don't know about Clio specifically.
It's made.
It's not secure goods, actually.
Yeah.
But what you might get to do is providing that, it actually supports HEPS, and it doesn't go through the chipset itself.
Which a lot of these really cheap chipset firmers do.
You could try making it manually redirect super the Google to come to a different server,
because a shocking amount of drivers media tech is especially good to have this,
and welcome to a lesser degree in Chinese phones.
They specifically turn TLS certificate verification off for some unknown reason.
Yeah, you could try doing that other than that if it's a media tech chipset.
There's an XML file, you can change, which has the configuration for all of this in it.
On welcome, I think it's just a text file, but don't poke me on that, I haven't checked.
Okay.
You are first.
Super interesting project.
Thanks.
How many crowds or base stations would be moved for both wide coverage?
To one percent light is a thousand.
How many would be moved?
Because it was how many base stations would one need for worldwide coverage?
That's really depends on how good each station's antenna is,
how good the location is in many ways, because we had stuff, and if it's indoors as well.
The best estimate I can probably give is on average, one every few hundred kilometres.
But in theory, if one had an absolute high quality big antenna,
you could get close to a hemisphere of reception in each position, but that is not realistically possible.
In addition, many especially cheaper receivers, such as the ones I'm using, have limits on how many satellites that you can acquire to once.
So I'm mostly going on a thing of trying to see if I can get as many people as possible to consider doing this in the future.
So, yeah.
Sorry, what?
Yes. The question was, would a Google or any supercell work me send them a fake armassilin?
How does it can tell if you don't have a SIM card in your phone, it would send zero zero zero zero zero zero zero zero zero.
So we could just send that all the time in the public work.
Sorry, what is that?
The question was, if I was planning to run this and I said, what would be the privacy implications?
Yes.
Basically, the privacy implications, the data that is received by the supercell, includes the IMSI,
and zero zero zero zero zero zero.
The list of cellular towers that have been within range of the request.
For more, there's also superv3, I think it is, which can technically support Wi-Fi access points I believe.
Don't croak me or not, but I haven't looked into that one as much because there's some weird pattern stuff on that one.
This is focusing on superv1, which is the bare minimum unit in superv2 to certain degree, which you need for Galileo satellites, which doesn't support that.
Risky speaking, if you're running a AGPS server and you're smart enough, you could probably figure out,
I did have the thought of what happens if the government asks us to, but that's just push that aside and hope it doesn't happen.
The question was, how big is the whole data set?
Basically, the data set you need is a list of every cellular tower close to it at least.
It isn't very big, this project's open cell ID and I think Neon's beacon DB, which maintains such lists.
There are a few gigabytes in size of CSV files, it's not actually a big.
The satellite data is probably substantially smaller, but gets updated a lot more frequently, would be my way of putting it probably.
Each satellite frame, as much as it takes up to 12 and a half minutes in theory to get all the frames from a GPS satellite, give them back to 25 to a second.
It's not that much physical data, it just changes frequently.
Yes.
The question was, how do you consider the possibility of an adversary spamming a malicious data?
Yes, I have, well, at the moment we don't have any service running to accept data at the moment,
we'll be making a blog post in Athens for when we want to try and find people to post basations.
At the moment our plan would be that we'd be getting an issue in tokens for each base station.
So each request is authenticated with a token and we'd be able to track that back to the user, probably in email address, at least, from who to send that.
And be able to, obviously, if you notice that something's wrong, we can remove the token and hopefully, but it's just stopped up.
So the question was, is there a baseline for what kind of computer you'd need to run this other?
The entire stack, not that much, the, it is surprising massive computation, but it should probably be doable if you're willing to, for a singular person, but just to think,
I don't have a good data track set on that, but I did, you were a bit of measuring how much a SQL request to get the position of a cellular tower,
but it's trying to think of several things in one SQL entry would take, or my personal computer, which is, but recently quick, it did take something like 100 milliseconds.
It's not that, it's surprising massive computation is a huge database, but it's not that much.
It was reply can probably do it, I asked myself, but I can't really give you a good answer on that one yet, sorry.
Thank you very much.