All right, hello, my name is Robin, and I'm joined today by my co-workers, Timo and Valer,
we're from Elman's Voip team.
We work on Elman's call mainly, and today we're here to talk to you about Matrix RTC,
Excuto, a Battle Royale.
Yeah, so of course Matrix is an open standard for decentralized communication, and it
is divided into rooms, and in these rooms, users can send these persistent events to send
text messages or things of that sort.
But of course, this is just one kind of communication.
There's a whole world of real-time communication that we'd also like to be able to do on
Matrix, such as exchanging voice and video data, or maybe real-time data for a virtual
world, or a whiteboard, or something like that.
So yeah, what if we don't want this persistent event model?
Well, we've spent the last few years trying to come up with a concept for Matrix RTC,
which lets you augment your rooms with new features, so it lets you add new slots for
real-time communication to your room.
And so the possibilities here are that an admin can come in and add a slot for, I don't
know, your Saturday night group calls, or your 3D virtual world, or your multiplayer
crossword game, whatever you want, and so these all are things that get added to your
room.
And yeah, that is the idea behind a slot here, is that it is a combination of an application,
which specifies what kind of data participants are going to exchange over this slot, and
an identifying tag, just so you can have multiple parallel calls or something.
Yeah, and so m.call is the very first application that we are proposing to add to this
spec, and this is, of course, just the basic functionality that you need for any voice or
video call, but third party apps are completely possible too, so you could just come in
with a custom identifier here for your world simulation, and yeah, how this works is an
admin will come along and they will send a state event, and this is how slots are recorded
in matrix.
There is stored in room state, so yeah, the admin here would send a state event with
the application specifying the type, and there can also be some applications specific
data there, and the reason why we are using state events for this is because it allows
for matrix R to C slots to be moderated essentially, they can be opened and closed just
by the admins, you can decide exactly what can happen in your room, yeah, and that way
they are persistent and they are authorized, so yeah, the idea is that just like you join rooms
when Allison Bob wants to get together on a call, they will connect to the slot within
the room, and so how they do this is Alice is going to send a sticky event, and sticky events
are another new concept that we are proposing to add to matrix, the deal with these is they
are just like normal room events, except they have guaranteed reliable delivery for basically
up to an hour, so whenever Bob comes along and sinks his client, he is going to receive
Alice's sticky event even if it was like the scroll off the timeline, and yeah, so the nice
properties of these sticky events is that they are a femoral, which is appropriate for
these RTC session participants in a way that state isn't room state, it's like you can't
delete an entry, so it sticks around forever, and they can be encrypted even, and they're also
low privilege, you don't have to be an admin to send one of these because they're not a state event,
yeah, and the most important thing here is that Alice's event declares how you can re-ter
the RTC transports, yeah, and so how this works, well of course in reality Alice and Bob
aren't a little Lego many figures sitting in a room together, they are people with computers
sending data across in that work, and so the way that Alice and Bob end up exchanging their data
is they are going to choose a transport, and the transport that we've now implemented for
Ellen Call uses life kit media servers, but the idea is there could be more added to the
specification for sure, and these are always going to be either resources provided by your home
server, or maybe some peer-to-peer thing, so yeah each member will choose their own transport
that's given by their home server, and then we let them, yeah, just upload their media there locally,
they decide they're on transport because we want to maintain the federated model that we
know and matrix even for real-time communication, and so yeah each member here will then go and
also download the media from the others from MoT server, and so this seems maybe a little unusual
for a calling system because you have to connect in multiple things just to participate,
but the reason why I think we love this model and think it deserves to be in this
fact is that it handles large calls gracefully without forcing them to be centralized, so you
get a nice balance with scalability because the thing is with large calls, more often than not,
many of the users will actually end up converging on the same home servers, that's just how large
calls in like office environments end up working, there will be resources like a home server provided
by the company that most users are on, and so they'll end up sharing transports, and that means
that to connect to this larger crawl, well okay it's only of seven people, Bob only has to connect
to just two servers to download the media here, and yeah so the live kit servers are still handling
the fan out of all of everybody's media, which is the hard part normally for large calls,
if you have to upload your media to and other people that gets pretty pretty expensive for
phones that are on flaky mobile connections or someone on a DSL connection,
yeah and so those are all the basics of matrix RTC already, so right there are four concepts you
should remember, we have slots, this is how you set them up and moderate matrix RTC,
the admin sets up the slot in the room by sending a state event, and that state event specifies
an application, which is the type of data you're exchanging, and then to join, we connect by sending
our membership event and we're publishing our media over our chosen transport, which is described there,
so yeah slots, applications, memberships, and transports, now there are a number of other
things that I haven't been able to describe here today because I don't have that much time,
the entry is interesting stuff is the late events and notifications and encryption, but there are
a number of other talks from last use matrix conference, if you're interested in learning about
how matrix RTC works in detail, you can go check those out and yeah the MSC's for all of this are
well yeah they're all up on the matrix spec proposals repository and I think they're not quite
ready for this spec core team yet perhaps but yeah they're very close to being sent off, we're
making our final adjustments to them and yeah so they all need your feedback, please go take a look
at them and see what you think all right and we'll have to team up. Yeah thanks for the introduction
or like the main part of the talk and thanks for giving us a background on the matrix RTC bits
and now we at the point where we want to build something with it so with this like big changes
or not big changes but this evolution of the spec we also did some really big changes to our
code base to make it all compatible and have a proven implementation for the spec core team and why
we did this yeah refactoring basically and adaptation to the new spec we noticed that we will
eventually have a very clear cut on our code base which basically gives us something like a matrix
RTC SDK for free and that's what we will talk about now so how to use it and what can you build with it
so first this is like a very very rudimentary overview what our element code code base looks like
we're building on the matrix JS SDK and those are all the parts which are just purely matrix
related sticky events and state events and then we have the element code repository and this
thanks to this refactoring and adapting everything to the new spec is now also pretty clearly split
into two sections the bottom section is matrix RTC and the top section is that I'm really
element call so the bottom section is things like connecting to life kid connecting to multiple
SFUs because that's all not done on JS SDK and we noticed we can easily pull out this bottom
block and expose it as a new built target so how does that look like basically here the sline is just
in between the layer between the yellow and the green block and that is the API we decided for
for this matrix RTC SDK so it's like pretty small fairly simple you have things like joining
leaving tracking members as a observable so you see anybody any new joiner and lever
and then of course you have the super important functions like sending real-time data, sending
matrix data and inside the members you have the connections to them also yeah subscribe to the
data and we even have like a high level data observable for all the life data and now what we
are actually now going with the demo is that we use this SDK target and build like a very,
very simple HTML template application for the good game and so you can use all the matrix RTC bits
like connecting to life kid connecting to multiple SFUs handling delayed event handling sticky events
authenticating with matrix all those bits extracted away and you just call await create matrix RTC SDK
you get an SDK and then with the things we just showed you can say store the data observable
into something called data um observable and then you can use that in godo so godo has like a pretty nice
FFI basically for JavaScript so you use the JavaScript bridge get the window interface
and then do dot matrix RTC SDK what we just stored in the window and then with this SDK we
have access to our data observable and all the other things and but let's just look at the data
observable one and we can use this in a game engine to build games with it and that's where
pass over to Valera showing you hello so my name is Valera so I'm going to show you like the
good at part so the part that probably will fail so yeah because we wanted to do to show like
everything we do in matrix basically we build upon blocks building blocks but you can reuse to do
most more things so this SDK events you could like send event that is but additional delivery
guarantee super important to say I'm in the call we have the delay events so now you can do schedule
event I can send this event this event tomorrow at 10 a.m. you can do self-destruct event but we want
it to do a game so this is a full local setup here I'm going to do like that first we're going to try
live after but you can see already that it's a very federation so they're not running on the same home
server and so the first part is this is like this part is matrix communication so you can see the
other side stay join game so I have now a game so here but everything that happened was in the room
so I've got a aware of the membership now I'm going to do start but the start is going to send
events by the data lifetime layer and I'm going to send the jumps so I can only jump for the left
player so one of them will fall down so you should hopefully see me on the other side like you can
see that I'm actually so this is like using two different life picture transport and now
the matter of historic we created a public room I mean maybe a Timo and I would start a game so
the room is here it's flat p dash or tc that falls down so I have to try from desktop I don't think
you can from mobile it's now which it's support yet but if you can we will let this room
open a bit if you want to try a bit later let's let me first start the game some waiting for players
so that's the part where we see if we have good network
it's like you stuck so okay I have a demo I mean I've recorded something so it's like
when we started that I say how hard can it be to have a world-but world-back network code
because I wanted it to be like really real time I wanted the birds to fire bullets
things that make you do small jumps so I said what a world-but-net will hurt it's easy
but here it was with a bad network and you will see that the other side is like
world-back and catching up so teleporting but yeah that's how it looks like with two people you know
so let me go back and see oh it's three players right okay that's the time
four let's try it oh which one I mean okay okay so if there's too much like it's going to last
the players I'm dead
so yeah I've had basically the game so you can add that so I think I would push about
to get to other people want to try out in some way a room if you want to have a break
I don't know who he's top one
are those five players so they don't want to play or more so there's also another game I just want to
show you so it's a widget so basically I can go here next tension we have another game I won't be
able to type but because I'm holding the microphone but just to show you that now it's
we'll ask you for a wall of permission it's a bit scary so like the widget want to take control
of your life so it should be loading or maybe I'll already load it oh this is the wall okay
so there's a lot of legacy haven't here it's long so but something we need to work out
in the widget API to rescue a bit to group a bit to permission service and yeah is this
single last bug it's requesting that several times okay yeah you can yeah we do have a couple
players though I can I get rid of my team oh are you just have to press it as often as it
called it okay there we go so here the game is like you have to type or I have to do something
you have to type to prove or you have to start the game right I think it started yeah so everybody just
types if they if they are not sure what exactly they can look at the three-letter preview on the
button left corner and then we have multiple people on the track I think for the layer they
I don't know I'm a little I'm a computer I show as little good old icons self of course one
I've been at to fail but as soon as one person will will make the full rounds it will send
into the room how quick they run it looks a lot better on this one yeah should we just like
that's time we try to play I can try to reload it oh yeah I think I already I mean they
bit and been this one maybe it doesn't like it and they're out two of them I think I already clicked
in that oh I can see people now oh they are okay okay how much I have to click and
oh I'm not clicking oh yeah okay so yeah people are racing you race by typing the letters
I'm sure I'd you go you could have put some real words yeah so that's it so like this is
a bit on top of the matrix RTC just to show like all the things you can build with the SDK
so any questions
and when I only copy I just basically gave you a thing and having all just the
key features just for the fake one to make all the games all the time we just put
up we know and use this real time for back in that I have and can use it I guess I have to
use it roughly you have to do that sort of RDP the question so basically if you want to
do a game but using something else that good up yeah another language so you just have to
have that sort of bridge so as we say in God it's quite easy like I have a JavaScript bridge
I mean just always meet you get a
I feel people can see things but I'm just getting the bridge I'm getting the windows and I'm
getting the match-off season if you have that bridge then it's just Mario of listening to heaven
oh you can do the game in JavaScript and I think we'll just work
and we can use and we can use it you can't use any of the GS SDK you have to build some
income and some benefits and I think there's another dimension or yeah I think there's another
dimension to this question so basically the way we develop RTC applications currently is
always as a widget that has a big advantage that you don't need to give some random game
your full matrix account with all the encryption keys but instead element web or whatever other
trusted widget supporting client you have will do all the encryption and key back up and everything
and the widget can just talk but it really needs to talk with this client and as soon as you
do a widget you're forced to use an iPhone anyways so the alternative would be really building a full
go client with then also a go-based matrix RTC SDK so then you cannot really benefit
from this work it's really for most the beneficial for widget developers and yeah our
idea is also that this is basically the safest and most the way to do RTC applications because you
get all the security benefits from not leaking your full key back up to the RTC application
yeah oh well it's the first thing so it's right and it's understood the transport part is also
structured and it's the thing on the signaling and managing the slots you have to do so if I
were to use some other transport like that so I can protect it in a place to implement the project
yeah exactly so basically the SDK currently is very much focused on um real-time applications
not really voice-in video since you have multi like multi widget support per room the idea is also
that there will be dedicated m.col implementation that do your calling and if you want an experience
like talking to people while playing a game you would just run two RTC applications at the same
time and yeah to keep the SDK simple it really is just focus on the data transport
and then yeah if you have a different kind of transport it would all be abstracted away in the RTC SDK
and yeah you could use it okay yeah we just learned in the previous part that's a RTC case
so much better and the idea of mine to do from the RTC as the K to RAS or yeah yeah I can also
do a comment on this maybe yeah we also hear other comments so basically as I said before
most of matrix RTC currently is built for the widget target because that makes a lot of sense
for the current use cases and maybe also for a couple of future use cases and of course
the best way to use a matrix RTC application is to have a RAS SDK client exposing a widget API
over the RAS SDK and then just having your actual RTC application running as a JavaScript widget
otherwise you would need to build like or there's discussions for building modules of the RTC
logic in RAS and then run it as web assembly in the widget and so I think we really talk about
like a full-fledged RAS based matrix RTC SDK as soon as people start to build native implementations
of voice and video calling which are not eye-frame based maybe somebody else wants to add something
to it yeah I would just add that let's see here I would just add that the dependency that like
element call for example currently has on matrix JS SDK it really doesn't need to be using
JS SDK at all because yeah it is ultimately just making widget API requests rather than client
server API requests and so the only reason we've gone for JS SDK there is I don't know we've sort of
hacked it so that it redirects things over the widget API and just has a familiar sort of JavaScript
interface that lets us develop element call for widgets and this like standalone SPA mode at the same
time but I don't think we want to have the SPA indefinitely at least like once we get all the
features of the SPA working just within our element messenger clients so that you can even like
to yes access through them at that point we would probably be able to switch away from the JS SDK
and have something more custom made for widgets yeah anything else no okay thank you for coming to the
talk