I'm trying to see what I'm going to tell us more about it, so please stay away.
Yeah, thank you.
Good morning.
Thank you for making the early talk.
I'm Travis.
I'm trust in safety tech lead for the matrix organization.
We're going to talk a little bit about community moderation.
But first, we need to talk about how matrix works.
So this is a slightly different diagram than when you might be familiar with.
If you've been to a matrix talk before, we have servers in the middle here.
In clients on the outside.
We transit events.
We're also going to call the messages at the same time.
This is a full mesh distribution.
So any time that say a client on server A wants to send a message.
It goes through their server to the other servers and then through to their clients as well.
This, yeah, is just the full mesh that we have today in matrix.
We have a little bit of a problem though when somebody wants to send something a little bit.
Let's say not great for the room or for the community in general.
So we have moderation bots.
Just like drop here, mail in there and molnir.
That can reside on another server.
So in this case, I'm using the little hammer to represent that moderation bot.
It's on server B in this case.
And the message is trying to originate from server D somewhere.
So that message goes out just like any other message.
The moderation bot will see it and try and clean it up.
So this leads to a lot of arrows.
This is what we call reactive tooling.
So that moderation bot sends out a reduction to remove that event from the room.
But unfortunately, everybody's already kind of seen it.
So this is possible.
There's lots of events flying around.
This has worked for most communities for quite a while.
The molnir is existed for five plus years at this point.
A lot of the other bots have also existed for quite a while.
Most communities use this model today.
Where if you receive an event.
Your moderation bot will check it.
Remove it if needed.
Bend people.
That sort of stuff.
There's also some server and capabilities in there.
If we're looking for a more proactive approach though,
we can kind of switch to policy servers.
So the diagram has changed a little bit.
Server C, we've just gotten rid of its clients.
It's just a whole policy server now.
And server D, we're now calling it the questionable message.
We're not sure if it's good or bad or if it needs to be moderated in some capacity.
And we're not even sure if we want to see it in the room.
So the foundation maintains a policy server implementation called policy serve.
It's very creatively worded or named there.
So when the server D wants to send that message,
it first checks with the policy server.
Is this a message that is worth for the room or not?
So the policy server will respond.
In this case it's saying it's great.
It's fine.
Sure.
Whatever post it.
And you get your normal full message after that.
That gets an additional signature attached to the event.
So that way other servers and other clients potentially can see that the policy server has,
in fact, checked that event has signed off on it.
It's doing great.
Send it to everybody.
It's going well.
The policy server, however, says don't.
That's that's not under them that I'm okay with for this room.
Well, that's that's where the diagram stops.
It doesn't ever reach the room at least hopefully.
But in current room versions.
Because matrix has room versions to sort of segment this functionality.
There may be a case where that server just sends it anyways.
And in that case, you go back to your layering of tooling.
So you might still have a moderation bought.
You might have a policy server at the same time.
Your moderation bought will catch that event that server D has just sent anyways.
The policy server said don't bother.
And there it goes.
The moderation bought will take care of it.
So that is the rough architecture of how matrix can work.
At least in a public sort of room.
But there's other tooling that's available for these types of communities.
So in public rooms, we consider, or at least public communities.
We consider communities to be a collection of rooms in a individual room.
It could be a whole server.
There's no specific technical necessary definition of what is public.
Obviously, like rooms that say you can just join today,
or probably a little bit better for that.
But you can use all sorts of different things.
Like if you're in a room-based community, moderation bought's obviously policy servers.
Surgery seals.
That's where you can prevent other servers from posting or joining the room.
You might also have like non-technical things like code of conduct.
Community guidelines.
That sort of stuff to sort of guide people into a more social kind of moderation there.
And obviously bands, production, silences.
That sort of stuff.
On the server side, we are looking at something called search readirection.
So when somebody is looking for a room, they might be using keywords that aren't necessarily useful or productive for that particular server-based community.
So you can just simply say, there are no results and redirect them elsewhere.
This is primarily helpful for CSAM style readirection.
So we're trying to prevent searches for CSAM and offering help for people, say, to go to StopItNowin.uk.
Hasharmaster actioner, or HMA, is from Mayda.
It's an open source thing that allows you to basically just dump a bunch of content hashes into a giant server somewhere.
And you can upload more hashes to see if it matches.
And then you can take action against them, hence the name.
We are currently in the process of deploying this on Matrix.org.
Not necessarily for the whole server, but at least for our community-based rooms.
So whenever an image is posted, we can actually start to see, like, it doesn't match certain types of content.
Do we want that content in our room? Probably not.
And we can kind of, where we can just prevent it from happening.
Either through policy servers or moderation bots, or both.
And again, you can continue going through a lot of the sort of social,
a moderation style technique.
You have terms of service, that sort of stuff.
You also have your reporting tools, which might lead into more of the rules tooling,
which I believe they have a talk after me on that.
So I will leave them to talk about, I believe, Osprey and Coup.
And so, yeah, Coup is their investigations to Osprey as a rules-based engine.
Other way around, I'm being told, I'll update the slide.
But yeah, the rules-based engine is very similar to policy servers in Matrix.
If your community is not quite public, but pretty public, we might be called as a near public community.
So this is where you might have the same sort of tooling from before,
but you might also consider something like a join gate, or join rules,
where the idea is, as people are trying to join your room,
you are simply telling them, hey, you need to, like, agree to the community guidelines.
You need to check a capture or something before you can join.
And that can limit a little bit of the sort of join flooding that can happen in some of these communities.
You know, I have power levels, this is your moderators, your admins, those sorts of things
that allow you to expand upon your actual capabilities in that room.
Obviously, topics, pendivents, that sort of stuff are where you can describe a lot of those sorts of rules within your community.
And again, that's more of that social moderation sort of stuff.
Invite links, more working on it, but that's the idea of, like, hey, you can't quite join yet.
You need to have a specific link to be able to join.
Obviously, on the server side, you have great limits, anti-spam web hooks,
that sort of stuff that can actually help bring more of that tooling in.
As we continued down the sort of four categories that we at least consider for moderation or communities types, we have near private.
This is where you start getting into those, like, more exclusive, not quite everybody can join,
but, like, somebody might be able to get in with an invite.
These might be encrypted, they might not be moderation bots come up again,
because moderation bots are just useful for managing your different communities,
because a lot of these bots support changing those power levels,
changing whose moderators, admins, sort of stuff.
Some of them allow them to synchronize that as well.
Join gates also continue to sort of make an appearance here, because, again,
you can use that to limit who is able to actually join that room.
Same thing on the server side, you start getting into registration tokens,
very specific invite links, just to be able to create an account on that server.
And, again, room stealing comes and starts to make that a little bit easier,
so you can start to manage that community, start to see what's kind of happening,
and making sure that it's as safe as it can be.
Obviously, we also have private as the sort of end of the sort of four taxonomies here.
It's very similar to any of the private you're going to want probably encryption,
the moderation bots, power levels, registration tokens, invite links, that sort of stuff.
ESS Pro is also where you're going to start to get some more of that admin functionality
to manage that sort of community, suspend accounts, that sort of stuff.
Of course, it would not be a false demo, unless we had a actual demo.
So, if you forgive me for using Windows, and I'm just going to start a bot here.
So, this is essentially what like a moderation view might look like in a just a standard sort of community here.
I'm just going to make sure that I'm mute, so that way element does not send notifications to the live stream.
So, you'll have a moderation internal room typically.
This is where you're tooling might sit, so an apologies if any of this breaks.
I am on the false demo Wi-Fi testing and production.
So, this is going great.
But yeah, deep, deep do.
Well, those refresh elements, and hope for the best.
All right, do I have at least a policy server?
Cool, I have a policy server.
Well, that's all the things that I really need.
So, we will hope that that is working, but yeah.
So, conveniently, there's some spam in my general chat here.
It looks like it's pretty common of what they're posting.
The accounts are pretty similar as well.
So, we will first, oh, there's my moderation bot.
All right, we're back.
So, yeah, we're first just going to see like, oh, I have a keyword currently in my policy server.
There's a bunch of different filters available.
Just to make sure I just see like there's a lot that you can do with a policy server.
This is our implementation.
This is not all of the things that you can enable different trusted sources.
You can disable media.
You can enable media for some people.
You can check message density, length, all that sort of stuff.
We're just doing the simple word.
We're just going to change the keywords to ban our particular type of spam.
That's going on here, especially how it's getting quite high in the numbers there.
It says it's going to take a minute.
I'm going to hope that it's not.
And if I switch conveniently over to my my little spam bot script here,
you can see that it's starting to actually get rejected when it's trying to send these messages
so earlier.
It was very successful.
And now it's just getting told, hey, that was rejected as probable spam.
That's, you know, that message has been denied by the policy server.
Which also means that in theory, if I go back to element here,
the messages have stopped.
We've been waiting about a minute here.
It's been about a minute since the last message.
The messages are done.
But I still have some number of messages to clean up.
This is where I might use my moderation bot to sort of get rid of those users.
So I'm going to ban those users.
I'm going to ban all of them.
And then I'm also going to use a feature of a lot of these moderation bots to both ban and
redact all of their events.
So if I use the the bad reason of spam here,
that should cause the the auto redaction to happen.
So again, we're just going to hope.
Oh, yeah, that's great.
Apparently I did not set the default list in my mole here, which is great prep.
So we want to just try that again.
There we go.
So yeah, it's banned a couple of users there.
If we peek through the spoilers, it looks like a gut both of the span accounts that happen
to be there.
And if I go back to the general chat, it's done the thing.
That's been removed.
And you know, element is just backfilling to a few days ago when I started setting this up.
So yeah, that is.
Thank you.
So that is moderation tooling and action very briefly.
We will continue on the very or the last couple slides here before going to questions.
So next up for the foundation is we're hoping to actually finish the FCP on
MSE 42 84 FCP is final comment period.
It's currently in review with the spec core team.
So hopefully in the next few weeks, we will see it actually get accepted into the spec.
And then it's a matter of actually converting that 700 line MSE into real spec for the sort of for everybody else to sort of implement and get it stable and all that fun stuff.
We're also hoping to actually implement that in that later room version, making those signatures required from the policy servers.
That way we don't necessarily need to layer the tooling as much.
And we can instead just require that, hey, if you don't have a signature on that event, it just can't be sent.
It's not accepted into the room.
It doesn't exist.
We're also going to continue exploring a lot of the open source tooling that's available primarily through Roost.
And where the foundation is one of the partners.
So yeah, we've currently got Osprey in most of the way into our staging environment.
We are working on trying to figure out how we can integrate coop with a lot of our stuff.
HMA is currently in progress.
We're hoping that it'll actually help us turn media back on in our rooms because we've disabled media for close to a year now.
And we would like to stop seeing screenshots showing up in our action log on the foundation side.
So we know it's important for just doing support and trying to get that support.
So yeah, we want to be able to turn that media back on.
We also want to support something that we're calling server-centric communities in policy serve.
So these are the ideas of putting joint gates in search read direction possibly through policy serve.
That's not necessarily to say that policy servers in general will support these features, but at least the foundations implementation will experiment with that and see how it's kind of going.
Finally, if you want to actually set up a policy server for your community on Matrix, you can follow these six lovely commands.
Essentially, you just invite the bot here to an encrypted room.
Set up your community, start to apply for some rooms.
That'll send some notifications to us on the Trust and Safety team.
We will most likely approve it assuming there's nothing terribly concerning in that application.
Once we approve it, it should be fine.
You can set up all of your filters on itself ahead of the approval as well.
If you have any questions or support or anything like that, just join our room on Matrix, and we will be happy to help.
Other than that, questions, concerns, complaints?
What can users do if their messages get blocked off?
No, they should be okay because you see that the algorithms on social media often filter out stuff that's okay, that's not bad.
Yeah, so the question is what can users do if they get something blocked that they maybe shouldn't have been blocked, is that correct?
Yeah, so a lot of these, like with policy server in particular, a lot of the filters are static analysis, so it might falsely flag an event.
And so there are some things that, like, particularly moderators and community admins can do to adjust those filters to allow those events.
But unfortunately, like, if the policy server says like that event can't be sent, it does just get rejected.
Same thing with the moderation block, if a moderation block reduces an event, it is redacted.
You can try resending the event in our foundation rooms, if you try to send an event that was rejected by the policy server.
It puts you typically on a few minutes silence just to make sure that any sort of follow on events don't go through.
So you can try again in a few minutes, try adjusting the content in particular, just to get around some of those other, I guess, more strict filters.
Mem.
Other questions?
I'll go over there.
So maybe the elephant's a room here, haven't we now introduced a single point to failure in our lovely federated network?
Are you using policy server?
Yeah, so question is elephant in the room, is there a single point failure?
Yes and no.
The policy server is not able to, or when you configure it in a room, it is completely optional.
You did not have to use it. A lot of communities might find that they don't need to use it.
And the policy server itself is not allowed to scan its own configuration in that room.
So if the policy server goes down and your community finds that it can't communicate anymore, you can just remove the policy server from the room and then your communication is restored.
That's also important when you have layer tooling.
So that way your moderation box can still take care of things. It might be a little bit more manual, it might take a little bit more time.
But you can still use the existing tooling to ensure that your community has kept safe there.
It does put a little bit of strain on the policy server itself, because obviously if you have a large room,
say matrix hq with a couple thousand servers, then all of those servers will have to go to your policy server to see if there is an event that needs to be, or if there is to see if there events can be posted to the room.
But again, policy servers are intentionally designed to support that large amount of traffic.
I believe there were other questions over here.
Matthew.
Yeah, I have some funders or anyone's two questions. Firstly, can you have multiple policy servers?
Okay.
And then second question, what happens if I email a piece of matrix to who is it in my call or not?
Fair enough. So yeah, first question is, can you have multiple policy servers currently know there's some complexities around trying to figure out what that means?
If you have multiple policy servers or multiple layers of tooling, then you might have basically both the least and most restricted configuration wins.
So when your policy server, let's say you have one policy server that allows basically anything, and then you have another policy server that allows basically nothing.
Which one should win? It's kind of the question. So we're kind of deferring that to a later MSC at the point or at this point.
But it is something that we are thinking about.
I believe the second question was, what happens when you email abuse at matrix to org, which is our trust and safety email address?
Is it a black hole? Hopefully not.
We do investigate as many reports as we can. We don't necessarily reply to all of those reports.
But if you do get the auto responder email, then it is in our inbox. We are seeing it. It has a ticket number.
We will take action against it if necessary.
It just might take us a little bit of time. We receive a lot of reports every week, and we have a whole team that is responsible for ensuring that those reports are taking care of
and go through the various stages and that sort of stuff.
You might also not see that the action taken on your report, or you might not see the action taken against the subject of your report,
because we might suspend an account, block it, deactivated that sort of stuff.
We might also remove rooms. We might remove media, and we might not necessarily advertise that we have done that to the outside world.
Any other questions?
Nuxie?
Yes. Is there any time you know that you lose the availability of the room when the reports are stuck with your information?
So, yeah, look again. The question was, is it intentional that you lose availability of the room and the policy server goes down?
So, again, like when the policy server is not allowed to scan its own configuration in the room, so you can just remove the policy server from the room.
There are two ways of doing that. You can just change the configuration to not have a policy server, or you can remove its users from the room.
Yeah.
I'm not asking whether you can get out of the situation.
Could it be whether it's intentional, like, bail safe and not come again?
Yeah.
So, the question is, I guess, whether or not it's intentional, it's fail safe.
Currently, yes. We do intend that the expected behavior is when a policy server goes down, that all messages are effectively rejected,
because you do not have those signatures. You do not have a way to get those signatures from that policy server.
So, policy servers, in general, are expected to have a high availability component to them.
On the make sure serve foundation, we have a significantly high availability set up for our instance of policy serve,
just to make sure that it can support a lot of those communities in those sorts of environments.
How will they notice that a policy server is down?
I don't know, just noticing that they're almost too required.
So, how will a, I guess, a community understand if the policy server is down?
Hopefully there are status pages, or ideally, they just have a high up time.
But aside from that, a lot of users might report that, like, hey, I can't send messages,
or policy server does support webbooks. I didn't demonstrate it here.
But it can send webbooks back to your community to say, like, hey, this message was rejected under these filters for this reason.
It was sent at this time, all that sort of stuff.
So, if community might notice that they're getting a lot of notifications all of a sudden, they're just like, hey, like,
sending completely benign messages of just like, hey, it's not quite working,
or like, that might be an indication of a filter failure, that sort of stuff.
So, you might notice a lot more activity on the sort of failure front.
And particularly, like, your community admins might no longer be able to post,
and that would be a very clear sign of, like, hey, wait a minute, it's not working.
So, that would be the general expectation.
Are there any questions in the chat?
There's one from BGT level who asks, in what I may have a script that was sending spam messages.
After it was getting bought, you said those boxes took the policy server,
is blocking your event as a possible spam.
How is that communicated? Is it reduction reason sent to the client?
So, the question is, I guess, from the chat, like, is the, or in the script,
is the reduction reason being sent to the client?
And it looks like it's, you know, it's been running this whole time,
and has been being told it's either well, it's not in the room anymore.
With the policy server, when it is getting, or when it is rejecting an event,
the client in this case, like it happens to be a script, so it just receives the error message directly,
but there is no reduction reason, because the event has not made it to the room.
It has not joined the day, it has not been accepted into the room.
So, the client just receives an error, currently on element,
that error is near message failed to send.
We are working on improving that.
We want to expose more of what the types of harm that we think of as a policy server
has found in the message, so we can better communicate how you can correct that behavior.
But again, it's not actually sent to the room.
Yeah.
Since, so when did you consider not telling the standard that we've had been,
that these messages have been rejected to think it's been successful?
Question being, did we consider not telling the standard that the message has been rejected?
Yes, we did consider it.
We have found that ineffective in some of our other experiments,
particularly shadow banning at registration, where most people,
well, most farmers will figure it out pretty quickly that their messages are being sent,
and just not actually making it to two other users.
There is also some complexities in how that might work at the day level,
because what we're also trying to prevent here is the room itself,
becoming larger and larger and larger, filled with spam,
and then you're having to persist that.
And so it's ideally a situation where you can just prevent it from joining,
or being accepted into the room at all, and then it becomes just,
you just don't have to store the events, particularly during large volumes,
spam waves, and that's sort of stuff.
Then, questions?
Yeah.
What about media's only text to check if I've provided some services,
but also much media, so it doesn't have explicit content on the way.
Question is, does it also check media on the policy server?
Yes, they can.
So our policy server implementation,
does have the capability of detecting media types,
and standing that media with HMA currently,
and then, yeah, you can do all sorts of fun stuff with that.
Our current configuration in the Foundation's public rooms is to just detect the media type,
and just say no, it doesn't scan it, it doesn't do anything,
it just, it just outright blocks it,
but we're hoping to relax that restriction through various trusted sources,
so we can see that, like, oh, this event is probably from a trusted source.
We're going to scan it with HMA anyways,
just check it against some known content banks, and just see, like, oh, it's,
it's not there great, it's probably okay to sort of let it go into the room,
but we will see, or yeah, we won't rely on that.
I believe I have 30 seconds.
We're going to just call it there.
If you have further questions, I'll probably be hanging around the Matrix stand,
or just drop by our rooms in the next Foundation side,
and we will aim to answer them as quickly as we can.
Thank you.