The next one is the state of open source XR,
and another million by Christoph Hack.
Yeah.
Thanks for being through.
You took my words for me.
Thank you.
Yeah.
As I was introduced, my talk is about the state of open source XR,
which is basically about Monado and what you can do with it.
My name is Christoph Hack, and I've been working at Colabora for Monado for a long time now.
So in this talk, we will go very, very quickly about the R hardware and tracking systems,
because this is important for the open source of their support,
which is basically the bigger half is exactly these tracking systems,
and these smaller half is basically everything else,
like how to communicate over your speed of these,
how to correct what the lenses do.
Then we will take a quick look at the state of Monado,
and then lastly, what you can actually do with it,
what how does the ecosystem look like.
hardware in the R starts 60 years ago,
but for this talk, this is basically uninteresting,
because these old VR heads are super,
like the super loud resolution, if they are affordable,
and only in 2012, then the Oculus Rift DK1 was released.
Smartphone displays, they are coming up,
and they are small, and allow latency and low persistence,
you then get ghosting, so it's also finally a good experience with those.
So this Oculus Rift DK1,
only then I am new.
This means this has that noise,
because VR gravity is because gravity is an acceleration,
and it can track your rotation,
but it doesn't know your position.
If you take a step to the left,
it has that does not know that it needs to kind of be involved to the right.
This tracking system system is not developed in commercial,
consumer heads, there are basically three systems for this.
One was LED tracking,
nicknamed constellation tracking.
The second one was light horse tracking,
and the third one is slim tracking.
We will take a very, very quick look at those.
LED tracking looks like this.
In the second iteration of their headset,
Oculus put LEDs on the headset,
and ship it with a camera,
but watchers these LEDs,
and because these software allows the configuration of the LED,
you can basically take what's in for how this headset,
very little is positioned in space,
if you have an image of it,
and you use computer vision for that.
Valve came out with the light horse tracking system,
you don't have a camera for that,
you have these boxy,
stations that sweep your room with lasers,
they don't send anything to your PC,
everything that is sent to the PC comes from the headset.
There are sensors on the headset that detect
when the lasers sweep of these stations hit each sensor,
and you do basically the same,
if it was infer how your headset has to be positioned
for this timing of each sensor being hit to be possible.
And the third one is slam tracking.
Microsoft started with a whole suit of vendors
who made headsets for this.
This one in the image is from ASA,
I have one from Neodyon here.
You don't see any camera or base station in the room here.
You only have camera as on the headset itself,
and you'll use a more complicated computer vision algorithm.
To detect features in your environment,
feature something like an edge or a corner,
and apparently it's possible to detect these from different angles.
So if you store all of these features that you have detected
and can read detect them, then you can also detect your position
relative to them.
If this was too fast, then don't worry,
this was just an overview.
And what you would take from it is that there are basically three tracking systems
that you need to implement if you want to cover all the health advantage out there.
Here's a list of a lot of these headsets.
I'm not going over all of them in the lighthouse column.
You have a lot of high end headsets,
and enthusiast class headsets.
On these slam tracking columns,
you have more affordable ones.
And LED tracking has been kind of all out of favor,
except for one case.
And that is even those mixed reality headsets
are slam tracking themselves.
But you can see that the other person has controllers in their hands,
and there are LEDs on the control of us.
And the cameras on the headsets will track these controllers.
So even if you have a slam track headset,
you still need LED tracking algorithm to track the controllers for this headset.
There are also standalone headsets.
Everything I mentioned so far is your plug-in and HDMI,
HDMI and this platform,
or USB to your PC,
and the PC does everything.
And the lower headsets, like the Metacrest picture here,
basically have an Android device inside them.
These are typically bootloader locked.
You can't loot them.
You can't let your own software on them.
So they are not very open source friendly.
But there are some exceptions.
The links are one for example.
Links is an European startup,
and they had a lot of issues with the supply chain
and shipping them.
It's unfortunately very hard to get one of those.
But if you get one of those,
here's a first market link.
You can install it.
It's not very useful yet.
But you can hack on it.
That's also the steam frame on this slide here.
Steam frame is what the release very soon.
We don't know exactly yet for the outgoing to ship.
Most likely they will not ship their slam tracking
or their controller tracking open source.
But probably they will have more open hardware platform
than most, and will be hackable.
So how does this software for all of this look like?
In 2013, it allows to be an announcement
of the Oculus Rift DK1.
Open H&B was started.
You can see a picture here that was taken at first in 2019.
I am behind the VLC guy of the traffic cone.
It was in the changing over there.
But open H&B is not very active anymore.
It's no basically unmeted.
Who has the R was another SDK that came before.
Kronos made a unified API.
That was an effort by our company called SENSIX,
which made the software and in cooperation with Razor,
who made a headset.
And that complete came up tracking solution
and complete open source driver.
But for one another reason they didn't make it in the market.
And they seized operations.
And OS for the RS basically unmeted to now.
I say basically because we have one of the original main trainers
of OSVR and Cullabora.
And she has the foot access.
And sometimes maintains it a little bit, but not much anymore.
In comes Monado.
Kronos in 2018 finally made the push to make
a unified API for XR.
And there is Mesa for OpenGL.
In full can he wanted to have an open source implementation
for OpenXR.
And that's how Monado was born.
And it's loosely based on the Mesa design.
For example, how it implements OpenXR
is we are a state tracker.
Just like these are implement APIs and state strikers.
There's a very long list of drivers that currently exist
for diverse hardware.
And of course, there's this incomplete.
You can help by expanding it by writing a driver.
And like I said, the bigger half of supporting VR hardware
is the persistent tracking system.
For LED tracking, we have a simple tracking framework
that is dotted for PSVR.
But I guess people thought it's not theory heavy enough.
The board from this thing approach is what community
then one named Faiton.
Young Schmidt has started in OpenHMB.
This is currently being plotted over to Monado in various branches.
It's not obscene yet.
But if anyone wants to help, then please come to our community
generous and talk to the people who are doing this.
For Lighthouse, there is an open source library.
It's almost usable, but it's usual people who are
who have this skill set are very busy and currently
lips the wife is minimally maintained by Colabora.
What most people actually use is a close source
and a wrapper for steam.
We are a close source lighthouse tracking
because that's basically usable for using for playing real games.
And for slim tracking, we chose an academic
tracking system called basalt.
We improved it, but it's still quite CPU heavy.
Clicking quality is not quite reaching,
commercial systems yet, but you can use it.
At this point, I can maybe shower.
Oh, this is actually looking light.
With this headset here.
It's very loud as a solution.
So like I said, this slim tracking system is detecting
stuff in the environment and can,
oh, it's very, very loud as this screen.
And it can determine its own position
relative to all the features that are detecting
in the environment.
And more for that later, if I have the time,
we also have a hand tracking implementation that I,
geez.
I don't know if you can actually see this in this lower screen here.
Let's skip this.
You just have to trust me on that.
We trained model on open and on synthetic data
that is included and usable with monado.
So with these great cameras can watch your hands
and make our hand model and detect very little joins are.
Let's skip a little bit ahead because I'm that one time.
One thing you can use whenado for is,
if you were actually very tricky,
or you could have seen an Android driver in the long list of drivers,
you can install monado on an Android smart phone.
And you get basically what Google had with Google
cardboard, which was like this cardboard,
little cardboard, smart phone holder.
You can use this with monado,
but I don't have an Android phone with me today.
I have a first market OS smart phone.
You can also run monado with an example application here.
It's probably very small,
but you can see that it's running.
This is completely open source,
stack here with 200 as the will can driver.
So if anyone wants to build us,
then allow a headset,
you can use this as a platform on Linux or on Android.
And if you want to start hacking on any of the drivers,
or any of the other software in monado,
then it's a very good idea to come to the community.
This code is,
where most of the people hang out,
iOS is unfortunate,
and he's not very much used.
But we do pledge all of our discord channels to matrix two,
so you don't have to actually sign up for this code.
So there's a lot to be done for the future.
The trend goes to mixed reality.
There are headset,
detect all sorts of stuff in your environment.
They can detect keyboard,
mouse, objects,
semantically labeled them,
construct a mesh from the environment.
Stuff like this is not in monado yet.
For a long time,
it seemed this would be very difficult to implement,
but in the last few years,
the commutation of AI,
looks very promising.
AI is a lot more than just LLMs or image generators.
A lot of AI is actually used for perception,
and for making a model of the real world.
That can be very helpful for open source,
because it's not.
You don't need our computer vision,
PhD department to use these.
So this could be also an interesting thing.
So what can you actually do with it?
One thing is that people want to use their desktop's,
while they are in VR.
You can imagine, if you put a headset like this on your head,
you don't actually see your PC,
you don't see your keyboard, you don't see your mouse,
and people want to put their desktop windows into VR.
So this is a demo video,
I shot about two years ago.
There's a Raspberry Pi connected to a Valve index headset,
and what's running on VR is WXOD,
which is a desktop solution that we at Colabora made.
There is Colaboyant,
and I'm splitting on it with a index controller.
I think it's a high controller.
But that is something you can do with it,
even though...
Even though we don't have a lot of time to maintain
our desktop at the moment,
there are alternative options,
where VR is made by the community.
Start-up, XOR is a lot more ambitious,
it's aiming to create a whole comprehensive display server
and support most of the open XOR application at the same time.
So there's a lot of movement in this space.
You can develop games,
and applications with a lot of frameworks,
go to open XORs,
and the biggest engine here,
the VR is on the lower engine,
OSXGR is used in flight gear.
The rear kit is another small framework,
maybe all of those run on Monado,
and are running on a complete open XORs pipeline.
The web ecosystem,
unfortunately, does not look as good.
Firefox is right through here.
It's just support some that VR.
It's support is open VR only,
and not open XORs,
so it doesn't run out of the box on Monado.
It only runs on Windows.
The graphics are support is not complete,
and they fired their XORs.
They develop us a few years ago.
So currently, they are not putting a lot of effort into this.
Chrome, Chrome,
looks kind of better.
It has an open XOR implementation,
but it appears not to be interested in Linux support,
and I'm basing this on having waited ten years
for them to support it on Linux and they have been so far.
Thought who knows,
with the release of Valve Steam Frame,
they might change their tune,
because Steam Frame ships with Steam OS.
Currently, what's out there is WebKit.
Egalya is working on it,
and there is partially working WebExour support.
I do have a demo for this.
If you start WebKit,
you get a browser here.
This is a WebExour test page.
This is the sponsor scene,
that many people will know,
that's a WebG application,
and there is a button, enter VR.
If you click it,
you get permission for up,
do you want to go into VR,
and if you do,
oh,
outside of the model right now,
but you can see it.
This is working WebExour,
it's tracking the head,
and you will see this on your headset,
if my HDMI cable is what the headset actually worked.
And I see my timers up.
Thank you.