Right, sorry about that. Hey everyone, my name is Eric, I'm a South
Engineer at Element, I primarily work with the back
and team who help build synapse. So I realized a few months ago that we just haven't
talked about synapse for I think years and years and years, which is a
little shame because I think there's a bit of a perception at times that synapses
a bit boring, it kind of works kind of doesn't a bit old and crusty.
We're actually, we do do a lot of work in synapse and we just fail to talk
about it at all. So yeah, let's talk a little bit about what we've been
up to, where we're going. I'm also just a little bit about what we're trying to do
in synapse, what the goals are, what the aims are, and that sort of thing. So
first, what even in synapse, so synapses, it's just the home-seven implementation
and Python. So our home server is the thing that lives on the server, it's where
your account lives, it's what your client connects to to send and receive messages,
your home server then talks to other home servers, so that's what allows you or
make sure to or you have a account and make sure to talk to users and other
servers, either their personal servers or other companies and things like that.
There are a lot of other implementations of home servers out there nowadays, which
is really, really fun to see. They're trying different architectures and trying different
languages and the like. But synapse is still the oldest, it's been around there for about
over 10 years, like 11, I guess, this year. So it's the most
featureful out there, it is the most battle tested and certainly the most widely used in
the ecosystem. I think that's about 10,000 according to our phone home stats, which I
checked the other day, but that's just the ones that have actively opted in. So the phone
stats, so God only knows how many they were actually out there. But, you know, it's been
10 years, so it's gone through a little bit of an evolution. So it started out as a prototype
back in 2014, before we even knew what Matrix was. So we were just trying things out and I
was there, Keegan, where he's. And like we sort of built some things and figured out what
this Matrix protocol could look like at which point we launched and told everyone about
it and it was great. But, you know, it was no longer a prototype. And so we sort of
more saw it then at that point as a reference implementation. So the idea there is that
you'd have different spec. But if that was a bit confusing or you didn't understand how
that would work in practice, you could just go and have a look at the code. And it would
be nice and simple and easy to follow. You'd have this implementation which is kind of
no froze. But it would allow people to try it out and see how you built something on top
of it. Unfortunately, more of people started using Matrix and that requires more features
and more config options. And since we wanted this to be a great success, we kept saying
yes, obviously we'll build these features. And we kept building as the ecosystem grew as
a matrix grew. We kept adding more and more features on top of it. And it very quickly
stopped. Being a reference implementation at that point, and started being more of a
feature for batteries included style implementation. All of that happened very naturally
and organic clear of the time. It's not like we set down and go white with changing direction.
It was just the right thing to do and we did it. But it does mean that we haven't really
ever had a nice kind of pithy one-liner for what is the goal of synapse, what we trying
to do with it? Which is unfortunate because it's really nice to have something like that
so people understand what we're doing and why we're doing that sort of thing. So yeah,
over the past few months, that's kind of been a topic that we've talked about internally
a bit more on this part. The synapse team, like how can we explain to other people what
our goals are? But long story short, this is kind of what we've ended up with, is the
idea that we want to provide a great experience out of the box. So any more concretely,
what does that mean? Well, for end users, this means being reliable, like it sounds like
a really low bar to not drop messages as a messaging app. But it's quite important. You do
one of those wants, if it's wants in a blue moon and people just stop trusting you. And
like for instant messenger, you need your users to trust. Otherwise, they're just going
to go and do something else because if you can't rely on them, your message is being sent
through. You're just going to have to use something else. And it's not just the kind of
whether a message gets through. It's also whether the user perceives it as kind of working.
They can't see what's going on in the hood. So like that perceived robustness is also really
important. So if the app feels slow and janky and kind of a bit jittery, people are going
well, something's clearly going wrong. And so performance is also really important here,
like perceived performance. So that's important for us. And on top of that, obviously, having a
fast app is nice and that will make people want to use us. And then on top of all of that,
you need all the modern features that you expect from a messaging client. 10 years ago,
the messaging ecosystem was different. There are a lot less features. Whereas nowadays, you
have encryption everywhere. You want multi device support, you want reactions, you want threads.
And, you know, as time moves on, that's going to keep happening, like more and more features
will need to be added. And synapse will make sure it's as a whole, will kind of need to keep
pace. So that's important. But users are on the only people which want to target here,
server operators are also important. Like a lot of you, I imagine running synapse in one way or the other.
And it's really important that it's easy to install. At least initially, even if you have to do
some more comfort down the line, if you get bigger. Because no one's going to recommend running
synapse. No one's going to try running synapse. If it takes hours and hours and hours and a lot of
fast to get going. And similarly with maintainability, like if you're running this stuff,
you're running a server-sized thing, is always going to require a little bill effort. But if it's
taking hours of your time, kind of every week, just to keep it updated, again, it's going to feel like
a FAF and people are just not going to recommend synapse to others. They're not going to run
at their communities or the businesses. So operators are also really important to us. And so
we want to give them a great experience too. But this is, we can't be everything to everyone.
Like this is, you know, there's a lot of different use cases. In matrix, we see a lot of different
deployments out there, whether that's just tiny, like personal servers or the communities or
something like first-end or like those really big servers like make sure to or big companies or
small companies, tons of different use cases. And so we do need to have a little bit of focus
of who we're prioritizing and what sort of features and deployments we want to do at the
least, we're just going to run out of time. So this is kind of just what we're focusing on now.
And the first big one is the smaller medium-sized messaging deployments. So that's like the
vast majority of the community, certainly, are fit into that bill. And the idea there is that
we'll spend as much time as we can trying to make those better. Make, yeah, give a great experience
to those users and the people running those deployments. For things like the slightly larger deployments
where you start having to do a lot of stuff with workers, we're still going to support that. Obviously,
we're not, this is not about taking anything away. This is just, we're kind of acknowledging
the fact that if you're going to run those larger deployments, it's going to take more tinkering
in a little bit more time. It's not going to work straight out the box about any kind of extra
configuration. And we kind of see that as fine people will expect that. And also for non-messaging
use cases, we certainly want to support them and we do support them. But again, it might take
a little bit more tinkering just because we don't know what the messaging use case is kind of
the usage patterns and that sort of thing. And that can be radically different than a messaging
app. And so that just requires a little bit more performance tinkering and that sort of thing.
But we do still support it. Just not quite out the box. And then finally, you have some of these
really, really massive deployments. We're talking about like millions of users here. And that's
just something that synapses currently, really just doesn't particularly support out the box.
It tends to just immediately fall over. And part of that is just the architecture of synapse fundamentally
the way that replication works. And the like, element does need to do that. To actually do that,
you do need to have extra tooling and expertise and you do think a huge amount of time trying
to make that work. You can just about do it. Element has just done some of this and help people
run at that scale. It takes a lot again, a lot of custom tooling and stuff. So if you do want to do
come and talk to us, the fundamentally kind of out of the box, the first version just falls over
and there's not much we can do to not make that just because of the use cases that are very
different and kind of have to take it by a little bit more case-by-case. So this is where our focus
is. So I know when you're at the top that people start having issues with the larger deployments
will fix that. We'll continue with the documentation for how you run like that sort of deployments
as well. So that's a lot of talk about kind of goals and aims and kind of how does this actually
relate to what's actually happening around and the practical reality. So how are we doing today?
Like some of that was aspirational. I think things that we do do quite well is making easy to install
so like you only need a postgres and a front-end web server and then you can just grab the
Docker image or the devian packages or one of the many third-party package that people maintain
and you just generate the config and voila. I mean Matthew showed earlier just how easy that
can be. It takes 30 seconds. Sometimes it's been these sort of things up. So that's great. We also
try really hard to make it easy to keep updated as much as it personally horrifies me to see people
running so that's from years ago. It is quite satisfying when you do need to update. They bump
the version, get the new package and they just restart and it all just magically works. Like 30 seconds
of time taken to do that and a huge amount of work goes into that. Like we do database
gimmers changes all the time but we really really try to make that force compatible so you don't
have to do any manual steps there. We've also been continuing to just add new features that clients
need. We're kind of looking at for example adding some better threading APIs for the next or the
rest of it and reliability. We don't really drop messages ever nowadays. Again it feels like a really
low bar but it's an important one so you know I'm going to take the win. But obviously synapse
isn't perfect you know consistently the biggest complaints we get from people are around resource
consumption and usage so things like memory usage, database size, get CPU usage and that's sort of
thing. So that's a big one and closely related as performance you know especially
perceived performance from users so whether that's just it takes a long time to load in the app
or to send a message or something you know that's something that you really need improving.
For a lot of people it's actually fine and a lot of people do use synapse after the box and it's
fine but it's really easy for one of your users to just join makes you say it's cute for example
and then you watch synapse really struggle on those smaller boxes which is unfortunate because
it shouldn't take one user to do something dodgy for it so it goes slightly webly. So I mean a great
example of what we've been doing recently to make this better is doing things like the signing
sync API which is part of matrix 2.0. So the sync API is used to get your rooms and messages
from the server and so when you start up it's how you get all those rooms and it's how you get
new messages and it's last time and things like that but the problem is it's that it doesn't
scale very well with how all your counters the more large rooms you have in there the more data
it sends down at that initial launch and if you're someone like Matthew not trying to pick
a new but that can take 15 minutes if not longer now to 20 minutes keeps going up as the problem
it started off for a couple of minutes and now it's at 20 and so that's obviously just fundamentally
unacceptable like you can't run a messaging up like that so sliding sync is something that
kegum has been working on to just get away from that and change the API to be
scale to larger counts and that has taken the time to sync initially from 15 minutes to a second
which is a thousand x speed up which is great and it's a great example of how we can change the
API's just to make it more efficient and that is used on element x so you can try it yourself for
example and yeah I mean it's one example of what we've done and we're trying to do some more
of that sort of thing then sometimes you know a second time a second is still too long for this
sort of thing but at least when you're at a second it's a lot more plausible to start
optimizing that heavily and really bring that down just know we essentially take a second it's just
us being slightly slow in Python so to support that we've added a vossport into synapse we're not
rewriting synapse into us at all as much as I'd enjoy doing that it's just used as a tool in our
belt so when we do find one of these CPU in terms of tasks we hear a hard we can move that into
rust and hopefully that will go faster and things like that as a small sidebar I am
astounded how easy it was to integrate rust into the Python ecosystem pyro 3 in all of its
libraries just make it super easy as expecting at the very least during all the native wheels
and all the native packages would be a pain to set up the CI4 but they just have a CI build will
command you stick it up there and it just builds everything for you which is great and example of
how this has helped us these numbers are from a very old dodgy laptop that if you send a message
in a room with 5000 local users it takes a long time to calculate all the notifications and all the
push rules and that sort of thing previously because that's quite intensive algorithm because
everyone changes their notifications all the time there's notifications settings all the time
and that you take three seconds and that has come down to 500 milliseconds by pouring that bit of
code into rust and actually the rust part where it actually calculates the push rules takes a few
milliseconds here it's just quite a lot of shuffling around for other bits and pieces for 500
so that's only a 6x speed up there for that API but it's still not shabby and it's something
that we can slowly use to move other bits and pieces over to rust where it makes sense
obviously we've been doing a lot of other stuff so I have two is a big thing
if you know other reason it would be nice to have two face support and also everything else
authenticated media is a big one which protects yourself from being used by your users just as
a nice file which is really nice for your users but annoying when you're running a server
there's also been tons of smaller performance improvements which just very incrementally move
the nail but together really help and then obviously the bug fixes and security updates etc etc etc
so that's kind of what we've done over the past deal話 here and coming up
we're kind of not quite sure what we're doing yet but this is the sort of stuff
that we're looking at so continuing to support major features and that sort of thing
but really to help really to look into the API changes and changes to state res and those
kind of algorithmic changes to really chase the performance improvements state res is
as security also talking about just now is a huge contributor to performance problems
and resource usage because it tends to enlarge rooms you have to pull out all of the state
do this algorithm put it back but we can look at moving that to incremental things
and incremental algorithms and the like and then also better of us support what we have today
is very bare bones and you can't do a sink away ever it can't directly access the database
so it kind of limits our usage of it quite a lot a lot of our actual heavyweight algorithms
do need to talk today to buy so directly a lot but if it has to bounce back into Python and back
to Rust you kind of lose a lot of the benefit of writing it in Rust in the first place
so yeah I think that's basically what's on the roadmap and hopefully this gives a good summary
of kind of this capacity we can take to get to our own goal I mean we're going to continue
to do all the day to day maintenance as bug fixes and security updates and performance tweaks
but hopefully hopefully hopefully with a bit of time and a bit of effort this takes us back to the
idea of getting synapse being giving a great experience out of the box because that's what we want
and hopefully that would really really help metrics as the ecosystem as a whole and yeah
just want to quickly say thank you to the community as well it's been great to see you
help a lot especially each other with running synapse otherwise yeah thank you
I suppose is someone of you wants to ask the question
thank you what's the question when are we going to finish faster room joints
oh not I don't know Matthew when okay wait a second when I'll be going to finish faster room
joints so we've got that that's the idea of when you join a room over matrix over the federation
and if it's a really large room it takes ages to synchronize all the state
they make sure they execute what I can't remember it's like some obscene number like 100 megabytes
of Jason or something which just takes a while to first pull out the database and then shove
over the wire and then actually process on the other end we changed it so that you could do that's
a two-faced thing where you kind of do the join you get joined to the room and then you can
get messages and receive messages and send messages and in the background you're synchronizing
the state so you get a really fast kind of snappy join and then you can again incrementally put
things in unfortunately that's what we have today unfortunately it's not quite good enough
with the user experience because when you join you have to have that synchronized state before
you can get any history so you still end up staring at a blank screen and the spinner while
everything goes on in the background in terms of what's next I honestly can't remember how much
work remains I think it's at least a couple of a month or so of just trying to make
do some performance optimizations to make that faster and do better kind of thinking of the state
and allow especially allow backpagination to work so at least you have some history and
you don't scared of blank screen when you're joining so yeah
there was a head over there
sorry yeah okay so the question yeah so the question is are we going to make
starting to think the default across the ecosystem we really really want that to be the default
and certainly it's kind of going to be we're going to duplicate the old sync API's
I don't know how aggressively we want to push that I mean step one is to actually get that into
the spec it's commonly waiting on me to have the time to finish up the remaining concerns
it's almost there and then it can hit FCP and we can actually get more apps using it
but certainly we want to deprecate it it's a bit harder to deprecate these things because
there's a lot of use cases especially with bots and things that do quite like having
getting everything at once and they don't care that it's so slow and they don't care that effects
serve a performance so much but that's certainly a conversation to have around when can we actually
delete that endpoint from the spec and how much work to support like the bot use cases
how does that make the methods that all and all the scale of the user in the context of the
free scale by talking up so I'll have to make sure to all kind of on the scale in terms of the
priorities we listed a lot of that is me with a stick poking it which is kind of what I
meant by takes a bunch of expertise to run we element also does have a bit of prior to retooling
on the side help with the bigger parts of the scaling problems that's specific to make sure it's
to org but yeah generally we just we try really hard to just keep an eye on it I'm
almost surprised that my pet phone hasn't page me about it recently but yeah I mean that's very much
approaching the scale when you do have to have a lot of the custom tooling and just for the auto
scaling and for some of the custom bits for the federation state media and stuff so
there's about 170 workers yeah 170 workers I think across two beefy boxes there's a bunch of
VPSs so yeah yeah yeah I think about five yeah yeah I mean we have yeah about that number
so for the microphone how many people are directly working on synapse probably the the
wider back in team is just a bunch of other work as well as probably about six seven
actively on synapse any given point at the moment is realistically closer to two or three
and yeah depends whether you also include make sure both indication service and that which is the
move over to I want to so you two questions you and you so I have a question about the
mass of the authentication so you wait in the synapse and you just went and so the question was
are we going to integrate mass into synapse at some point where masses the thing that speaks
and the hope and desire to get at least a small part of mass into synapse so that when you do
spin synapse up you have something basic that works with like username and password and stuff
where it was actually the moment mostly focused on just getting mass onto make sure it's to org
so we haven't quite gone to the stage of looking at when we want to do that but it's certainly
desire to have something there probably not the full blown mass but at least their bones basic
thing which means if you want to do have more features and use a more comfortable thing you can
spin out a full blown mass and the nice thing about masses is written and rust which we can
integrate more easily into synapse so you can just use it as a library is the plan and then you
just need to spend some time hooking up all the database API calls and the thing like
probably was saying the standalone mass is going to continue to be the primary thing so it's
not going to be about sliding thing where there was a throwaway proxy that got killed and integrated
instead the standalone mass will be the main way that people use it so if you're wondering
should I wait to deploy mass now so there's no just going to it now it will continue to be
even trying to grab it you do it's wondering what we have and and to end in crypto and
charge it's when you're calling with what about files and if there's some way to press it on so
that you know man in the middle can make sense of and so is the question like how do we
deal with file attachments with all of the encryption so all of that is already handled so the
media APIs they are just all the same and what happens is on the client side you use the same
encryption scheme that you use for all of the events to actually basically encrypt the attachment
before you upload it and then you share the keys for decryption with everyone in the room
and then the server just sees as opaque it encrypt a blog it can access it but obviously it's
encrypted to there's not much point and then anyone else can download it but unless you have
the keys again that's that's pointless it's already done yep
not about implementing some advanced crypto things to you and privacy like things that
signal does for example product context recovery or is it sender things
we'll have some of that question so I'm doing some more advanced stuff around the privacy
perfect for serving protocols like signal I think some people have been looking at it that's
watch more in terms of crypto team land I don't think there's any active what going on
but there's MLS we're looking at I mean there's MSE's MLS going around and that's the thing
but we as a back-end team I'm really looking at anything at the moment thank you very much