Right, hello, so I'm Robin, this is Timo, and we have a half-shot joining us virtually,
we're all from Elements VoIP team, and we're here to tell you all a bit about matrix RTC
and how to build real-time applications on top of matrix using this.
So you all probably know what matrix is, but it's worth restating that today matrix is
primary use case is instant messaging, but matrix really has ambitions to cover all kinds
of exotic communications use cases, starting with voiceover IP, but then extending possibly
to virtual reality augmented reality, the Internet of Things.
So let's take a moment to talk about what matrix is good at today and what building
blocks it provides.
So today we have primitives that are good at enabling these mere real-time use cases like instant
messaging.
And so some of the primary things that it gives us are when you're in a room, you have
a room timeline which represents the sequence of events that have happened there and
due to decentralization, this isn't just a strict order, but you actually get an aciclic
graph and so you have to think about what happens when the order of events isn't clear.
But if you want to use this for really streaming data quickly in real-time, well you can
get subsequent latencies perhaps if you're exchanging data locally just on one home server
or in some cases over Federation, but often over Federation you're going to start seeing
latencies that go beyond the one-second margin and especially if you want to be like doing
things like streaming, voice and video, putting this into a room timeline event really
doesn't make sense because the room timeline is for things that are meant to persist indefinitely.
And the other important thing that matrix gives us is room state which is a summary of events
that have happened in the room as basically a key value store.
And so these are great for building instant messaging, but what's missing is if you want
to do things like voice like whiteboards, we're missing a way to actually communicate
in real-time with low latency, low jitter and exchange data that might be more a femoral
in nature.
And right, so let's take a brief moment to go over the ways that we've tried to do voice
over IP in matrix in the past for the voice and video call use case.
So ever since there's been the idea that we could just use matrix as a signaling layer
for WebRTC and establish peer-to-peer WebRTC connections for doing these one-on-one calls.
And what you do is you just take STP offers and you'd exchange them in the room timeline.
And this is great because it automatically takes advantage of matrix's encryption primitives.
And so you pretty easily get an encrypted call working that way.
But this just doesn't, it's not a group call.
And so when we thought about how are we going to make group calls where it can matrix the
first idea was maybe we can just use some off-the-shelf solution that already scales really
well to hundreds of participants such as free switch or jitzy.
And so jitzy in particular was integrated as an I-frame as a widget that would just be rendered
within your matrix client.
And then we thought about, well, what if we want to make this a more native experience,
actually integrate it in with matrix and try to get, try to bring back some of leveraging
matrix's encryption.
So that was why when we were working on the first iterations of matrix RTC, we thought
about maybe you can just take this simple one-on-one WebRTC setup and actually just generalize
that to connect to more peers.
And so you could have group calls where you're just routing your media to your peers
in a full mesh configuration, every peer connects to every other peer.
And so this works to get group calls and they're encrypted, you're communicating, you're
just communicating your STP over two device messages that are in private channels and they're
encrypted with home.
But it doesn't scale of course because most internet connections that users are on, they're
just not going to have the upload bandwidth to be able to send your local video 10, much
less, 100 times over to all the other participants.
So what we really needed is something that was as ready to ship as an off-the-shelf solution
like Jitzi and that could scale as well as having a conference server that can sit in
the data center and have a better internet connection.
But also a solution that we can still tightly integrate with matrix so that we can check
this encryption box as well.
So we're really looking for an STK and we kind of found exactly what we were needing in
the form of life kit.
So life kit is a company that's building open source solutions based on WebRTC for enabling
all kinds of real-time applications.
They develop and as a few a selective forwarding unit based on Pion, which is an open
source WebRTC framework written in Go, that was exactly what we were considering using
when we thought briefly about writing our own SFU that would be more native to matrix.
So it's like if we don't want to reinvent the wheel, we should just use life kit.
And so life kit provides the concept of rooms and participants and for matrix RTC, we
want to have the concept of sessions that you can establish and advertise in matrix and
then just session members, the ability to join these things.
And so kind of just match up the IDs and this is how we set about integrating these two.
Right, and then think about well, how are we going to advertise exactly in matrix that
you want to be joining a column life kit.
So how do you advertise that you're in a call?
Well, here it makes sense to just use the room state as opposed to in the past, we've
been using room timeline events for things like the one-on-one WebRTC calls.
It makes more sense to use room state because then it's very easy for clients to look
up.
They always have a copy of the most recent values of room state, right.
And then for if you want to ring someone else's device, well, there's another matrix
primitive, which just thing that matrix provides that makes a lot of sense here, which
is to use an event that contains mentions or contains intentional mentions.
And this will push a notification and then you have data such as just what session you
are joining in the first place as some repeated information about the member that is joining
in particular.
And then the active life kit focus that she's going to be joining out for.
And when she wants to send a notification out to perhaps ring other people's devices, this
is what that looks like.
She's just sending an MRTC notify event, which contains intentional mentions.
So just by setting this room true, this is automatically going to be pushed as a push notification
to everyone else in the room.
And then finally, when you want to leave the call, that's another simple state event update.
You just set your member object to an empty object.
Right.
So those were some of the fundamentals of matrix RTC with the exception of encryption, which
kind of deserves its own talk.
And so I'm not going to attempt to cover that here.
There's a great presentation from TMO at the Matrix Conference in 2024.
If you want to go look up the details of that.
But basically, we want Matrix RTC to be a fully featured call solution with all the bells
and whistles that you might need for creating a consumer grade video calling experience.
And so now we're going to see the video output going to work.
Perhaps I think this is not seated properly.
Okay.
We're going to have on the one.
Yes.
We can do here.
Okay.
Right.
So now we have half shot joining us live and he is going to demonstrate a little
bit about what it looks like for these raise hand and reaction features that we've
recently added to the product or to element call, which is a video calling application
implementing matrix RTC, what those things look like.
Right.
I suppose we don't have the slides up.
So you can ask him.
Can you, can you, can you share the slides?
Yes.
Who is the kind of person among the slides?
Okay.
Next slide.
Right.
So we have this really great contribution from Milton of Nordic earlier in 2024 of adding
the ability to raise your hand in element call and the way this works is really cool.
It's just using a reaction event in matrix.
So we're just reusing that same thing that matrix gives us and you relate it to your
call membership event and then we render it in the call interface.
And next slide.
This is what that event actually looks like.
We have the relation and the key, I suppose, yeah, it's just a hand and yeah, it's just
the standard reaction.
Next slide.
Right.
Then we also wanted to add emoji reactions to element call as well.
Now these are a little different from normal reactions in matrix because you might
want to send the same emoji over and over again, whereas a reactions you can kind of just
add and remove them from an event, you can't send the same reaction twice.
So we ended up creating a custom event type for this, which we will hopefully go back
and spec some time in the future.
They look like when they're rendered and they have sounds which is just not playing.
But yeah, next slide.
This is showing that they are completely normal.
These aren't the standard reaction events, but it's using completely normal relations
and that ticks.
So it's just referencing your membership event and then you send the emoji.
OK.
Thank you, half shot.
I think we're going to speed things along.
Let's see how quick this switch knows.
I think it is, right?
Is it doing something?
There we go.
OK.
The guy can just move it over.
Right.
It's this that needs to be moved over.
OK.
There we go.
And here.
Perfect.
Right.
So as I was saying, Matrix RTC has ambitions to be far more than just voice over IP.
That's why we refer to these as sessions rather than calls.
But I wanted to highlight a couple of the ways in which Matrix RTC aims to be extensible.
So we have this application field when you're joining a session that you can specify
exactly what kind of data you expect to be communicating over the RTC session in the first place.
You might use this to, you can chuck in any names by any names based identifier and use
this to build a whiteboard experience, a word processor, what have you.
And you also might want to consider like integrating or we might want to integrate a third
party call back into the future.
And so the way you might do this is just changing up the type of the active focus.
And this is how you might integrate or just signal that you're using HITZY focus rather
than a live kit focus.
Right.
So I take over now and then starting with saying that this is something we, like if you followed
along the Matrix RTC project, we've advertised since a long time.
Like this extending with different kind of applications they all RTC with the same architecture.
And now we are finally at a state where we can, where we are confident in our current current
SDKs and we can show a demo on how this actually done.
So what will come next is a demo where we build a Matrix RTC application with the current
SDKs which does all the specified things send a state of end into the room, connect to
the live kit as a few, but does something other than calling.
So I'm, again, in this, like, slightly odd situation where my screen is extended.
So I do my best, but the mouse might be, like, a little less coordinated than what would
be if it's on the same screen.
So what I've trapped here is, sorry, is a project that is basically just a matrix widget.
This is cool because then I don't need to care about encryption and, like, authentication.
So I'm, in one room, like, this M room with two accounts, it's actually the same account
in this case, but two different devices might also could have used different accounts.
We'd make any difference.
And in this widget, I'm now starting a matrix RTC session.
So I'm joining the RTC session, element web should detect this as, oh, there is an ongoing
session.
Now things, it's a call, but it's a different type.
And then it will automatically build up a RTC channel over life kit.
So this is all just matrix RTC infrastructure.
There's no back and fronting on my computer, and we'll have, like, a little demo of, like,
a real time experience.
So I start here first, it says it's connecting.
And now it's connected, and we can see this client is actually detecting.
There's something really time going aside.
It's a little annoying because it's doing both, like, syncing the coordinates.
And so we have the name in the background.
So yeah, you can really tell us just, like, a tiny demo, which was done in more hours
than days.
And then we have an maximize button.
So we really can do all the things, like, write text, put it into a box, and edit it,
and then move around the frame, if I would remember how to do this.
I don't apparently, but oh, yeah, there it is.
So you can do all those kind of things.
And they all synced in this, like, smaller frame.
And then if we're done, we can leave on this side.
And minimize here, okay, that's, like, a little broken.
So we would look better like this, minimize here, and leave on this side.
And it will detect that the RTC session is over.
I should, like, clean up this frame also a little bit, but that's basically our
minimal product, how we can do, or any of you could do with just a couple of lines,
a RTC app, or you don't need to care about any back-end implementation
you have an entry-end encrypted with an asterisk, or a little bit of work on this side
needed, and where you have an entry-end encrypted real-time experience for, yeah, anything.
So this will be on GitHub eventually.
I think I will just do a swim announcement.
So if you are interested in such a thing and want to build one yourself,
you should be possible in the next couple of weeks when everything is published.
Cool.
So then I give over to you again, right?
Yeah, there you have it.
And I try to get this nice again.
Custom real-time experience built on matrix.
OK, and now with whatever amount of time we have left, I would like to just go through
a few things that we have in mind as possible next steps for a matrix RTC.
So one question is like, where do you put metadata about a call,
like such as just which life-it-instance to connect to?
We previously were putting this in a state event that just describes information about
the whole call, but then because of the ownership issues, we switch to a situation
where everybody proposes their own life-kit instance and then clients are just running
a simple algorithm that is selecting whatever the oldest membership is
or converging on their life-kit instance.
But in order to align better with our desire for this to really be a federated solution,
we might want to enable clients to just publish their media on their home server's own
life-kit instance and then have everybody in the call connect to all of the life-kit instances
involved and fetch media from them so that you're really, you're really never converging
on a single instance, but if you're working across Federation, you're going to have multiple
servers involved.
Another thing that I've been spending a little bit of time thinking about is how to do call
recording in matrix RTC.
Something interesting is that there's no solution out there today that I'm aware of for
video calling that tries to do both into end encryption and also call recording without
just breaking that encryption.
The usual way is that the call will start encrypted, but then as soon as someone hits the record
button, the server is basically going to join a recording bot to your call, which is then of course
breaking the encryption and just exporting all the media to the server.
And this is kind of nice because you can get like AI transcripts and summaries this way,
but what if we actually had a way that was more private?
So of course the manual way of recording is just like ask your coworker like hey can you
record this call, grab a recording of the screen and then upload it somewhere.
But perhaps we could make this somewhat more of an automated process, just have a button
where you click it and then it sends some signal into the matrix room saying hey I'm recording
this call so that if other people try to record they'll know okay so somebody already has it covered
and then your client will automatically download all the streams and save them somewhere
and then upload it automatically back to the matrix server.
That could be a relatively simple way of getting recording, working with end-to-end encryption,
but there's perhaps an even more interesting way which is what if the server was still
responsible for the recording so it doesn't matter if like any client crashes or leaves the call
or gets disconnected except the server isn't actually joining the call and decrypting all the
streams but rather the servers like recording but is still just sitting on the outside receiving
the encrypted media and saving that to disks somewhere and then after the call participants can
come back if they have the encryption key, receive the encrypted data and yeah just decrypt it so
the server has never actually seen it still and here is how that might work within live kits
architecture so live kit not only provides the as a few component it also provides these
ingress and egress components so I tried briefly over a weekend to get this working to have a
little demo unfortunately they're ended up being a missing piece but the idea is you can take like
live kits egress service just get it to start streaming the encrypted as frame data out and save
it to disk using gstreamer and you have some like lightweight recording and playback service that
the user communicates with and that just orchestrates all of this and then for playback you
take the data on the disk and you'd stream it back in through the live kit ingress using
a combination of gstreamer and call whip it's this interesting thing and then that's going to
create a virtual participant which then you could use room state for whenever you need just lightweight
mutable fields with something close to like last right wind semantics but another thing that I
think will become important if you're trying to build more complicated applications is going
to be relations currently you can think of like when you're reacting to another event that the
reactions to this event are some kind of mutable set and so mutable sets I think end up
be quite useful for building more general applications and so in the example of like a word
processor of a word processor you might take a suitable CRTT for doing word processing and then
any images and diagrams that the users create while using this you store those in the media repo you
get references to them and then metadata about the document you could store in room state because
this is stuff that's not super critical if like two people are editing at the same time
one of their edits could just overwrite the other and that's kind of fine but then edits to the
word document you might aggregate these by relating them to some kind of startup document event
and this is nice because then clients have a way to go and ask the server give me all of the edits
that are related to this document and then you have a way to sort of aggregate them after the fact
without having to page through the entire timeline the entire history of that room
right and then finally one thing that we're thinking about is if we want to build more of these
applications that are just VoIP we really need we're really starting to be something like a matrix
RTC SDK and so things that an SDK could be well suited for is like simplifying the connection
to live kit this currently takes some boiler plate code to get working because you have to
fetch some tokens and whatever but then just also the ability to have a more abstracted data model
for how you exchange data in real time and then persist it could be nice and so having some
way to save and restore arbitrary documents from a matrix room using this sort of combination of
room state and relations could be really interesting and perhaps we could even integrate this with a
one size fits all CRDT such as auto merge I don't know but I think regardless the future is
right for building all sorts of real time experiences on top of the matrix so thank you all for tuning in
as there is no follow up talk there's maybe some time for questions are there some questions
right yeah so you're sure that they move with a picture or extension or that's not called
so I'm sure this is packed as a model so basically oh yeah that's what we do always so
the question was in the demo which I've shown um if the things which we're seeing in that demo
respect and I mean they're different iterations to respect so there is like have their proper
MSC's been written which means that people like properly wrote out what they are doing and don't
run them stuff in the background and then there is actually making it into the matrix back and
making it into the matrix back like none of the voice stuff is is there yet so that's like a longer
procedure but for like the format of the state event what clients have to provide to understand it
and for how to connect through the life kit as a fuse that is all written up in MSCs there are
a couple of things how we do the idea of a membership which is like currently in discussion so
there will be updates to the MSCs which is like a normal MSC process but yeah that's a current state
so like all this could be understood by reading through MSCs but it's not yet in the spec so
clients aren't enforced to support it um like you will have lots of clients which don't support it
but um element web there's almost the same thing then the MSCs accept for this like membership
identifier thing which is like a super recent change yeah and uh like so so you credit call nowadays
power by announcing your presence can inhibit any specific way I credit it doesn't mean there are
what ways they're to decline it for but so we call you is there a way to say no for it and then
kind of stop the reading actually yeah this is this is something that the question would be
right so the the question is um we can advertise you know starting a call but um do you know
how to decline a call say that you you're rejecting a call um right this is one of the things
that's currently still missing from our specification right now but there are ideas floating around
for this like the primary one would be so um the ringing comes from the the notify event
rather than the the state event where you're joining the call so one option would be the receiving
participant just takes uh a reaction or some other relation and yes as thumbs down or uh no or whatever
and uh attaches that to the notification event that could already work right um you were talking about them
and matrix RTC SDK and I was thinking one way to expose that would be having a matrix RTC
what which could join the call and not really going to the call so all the basic coming of me
the rest thing so that it's a full participant it's just not a human and thanks so that would be
one way to codify what we're thinking that's a lot right right so I suppose um this is more of a
comment yeah just about um rather than building an SDK around um creating clients that interact
with matrix RTC in a special way that it could be um interesting as well to have a way to create
bots that join just calls not necessarily other experiences and yeah I do new things in them
right I think this is something where it certainly interested in having at
some point in the future is it yeah oh wait right so um on the just how we're right right
the question how we're managing to encrypt that calls um so just on the the transport side
we're using the s-frame standard uh which I I don't know the full details have but um
basically we have what the things we get are um a small field in the trailer to each frame I believe
which indicates the index of a key that you're currently using um so clients are able to
potentially be switching between different keys um during the same call and still have this all be
decrypted seamlessly by the receiving clients since there's information about um when exactly the
participant switched and on the key management side uh this is a part that's also uh still subject
to some change um the solution we have today is just that when you're joining a call um each participant
in the room will go and uh send uh the key that they intend to be using uh into the room as a
timeline event and this is an encrypted timeline event and so we we don't get all the nice properties yet
like perfect forward secrecy uh post post compromise security yet um but we are intending to move
to a solution where keys are not shared just in the room timeline but rather directly uh one on one
with the other call participants by using matrix two device messages and this is the um the solution
where um you really should go and check out team as full talk on the subject to get more the details
about what we might be going with there yeah so basically that that talk then gives you
what the MSE will become so currently we I think there are like multiple sections in the MSE
it will so that's also point which isn't like as far as you want it to be um um one section is about
this like timeline thing and then we have the two device version which will be much more superior but
it's still in the making then I think one last question we had right
probably it may have more feature question but since you showed this mouse moving to the
mouse feature and they have a thought about the feature where when you share your screen
others can basically move their mouse because that's right that would be the perfect tool for
that type of timing that's always the thing that's interesting but I realized this is really hard
on like way length apparently uh so and they have a thought about it and so yeah so a question for
this stream um when we showed the mouse moving around is there also like the opposite direction where
the listener can control the mouse and my computer and actually take control over the computer
okay or just showing a pointer um so this is very much like beyond the matrix RTC
aspect because it's just a client needing to do it and yeah I think like the the question already
contained this is most the operating system security thing and the trend is very much in the
direction that this is less common so operating systems are strict and strict at how much
applications can do with your wind management and mouse control and that could be such a thing
at least drawing something on top but like the the example of the question was like
pep programming and I think they are almost a more interesting solution would be what editors
are currently heading towards where you have like a proper peer programming editor session and
that could of course be done over matrix RTC like they could be a visual studio code extension
joining an RTC session and then doing all the mouse movements and all the edits over RTC
and I think they're already like things that do this I mean like Microsoft ships their own thing
and one could maybe find a way to build a back end on matrix RTC so yeah definitely multiple
and lots of opportunities to use matrix RTC for this kind of secure pep programming session
not sure if like mouse control over screen sharing will be the final thing but super interesting
topic definitely oh yes yeah fast on so you just left it right the multiple projects moving to
as of now for example picker but I think that's how someone from the community
will you be possible to just use their platform are you in this room or do you have permission
to participate in this call there's something one could extend but not as of now and then you get
a bath token from the from the life kit as a few so yeah that would totally work like a
big blue button already installs a Docker container with life kit you could just additionally
at this DWT token service you just give the DWT token service a life kit master token or that
it's called differently but something along those lines and then you basically made your life
kit service for big blue button a matrix RTC back end so yeah that is totally feasible to
basically use the same the same back end solution on both so thank you very much Robin thank you
Timos thank you well