Hi there. I'm Jade. I would be presenting with Nexus today. Unfortunately, they can't
be here. A quick disclaimer before I start this talk. I'm not a cybersecurity professional.
This is not a tutorial. This is how we dealt with a real situation. I'd about continuity and
continuity is a matrix chart client. Is that stuck? Oh no. Okay. There we go. continuity is a matrix
chart client. Matrix is a distributed chart protocol if you haven't heard of it. As a project,
we run continuity.org, which is where we keep all of our official accounts. Continuity is a part
of an ecosystem of other servers all built from the same original code base. The currently active
names right now are conjugate, grapevine and tile. Now, just set the scene to this talk. It's for
him and I can't sleep. So I open up my computer to download some books for my Kindle and I see this.
This is quite bad. If you can't quite see what's going on there, my project account has just
sent a message. I've been query raw itter, user ID, device token and that commands just printed
out all of the session tokens for my account. There are a few of things going on with that message
but we'll get to that later. Now I saw this message 23 seconds after it was sent. That was
incredibly lucky because that meant we had time to react. If I hadn't seen it, the things would
probably be a lot worse. But what's the first thing you do when you suspect your account is compromised?
The first thing you do is you alert your team. The second thing you do is panic, right?
And then you go through and you try to limit the damage. You disable the account's admin access.
We suspended the account so it couldn't log in. We reset the password in case that was how they
broken. We disabled all of the login tokens that just got leaked. And then we started putting out
the plugs on all of the servers that we had access to. Unfortunately, the person who manages
the continuity or cone-suffer was asleep so that's one plug that we couldn't pull out.
So we're safe for now. But how did the attacker do that? Have we stopped them from doing it again?
It's time to investigate. Identifying the attacker's server is pretty easy. There I count
joined the room that that message was sent in, like just under an hour earlier. And we go and we
look at the event. And looking from other servers, everything seems to check out. There's nothing
suspicious from that point of view. But I have a hunch. And so I get the contents of the event
from the continuity or cone-suffer. And we can see there are the all events I explained what
those are later. And you can see the body of the message where that admin command is.
There's some extra data there that marks it as part of a thread. I think it's a server notice for
some reason. The event ID. And then there's some more data there, like what room it was in,
the sender, and some signatures. But something is missing, I realize. And that is the client
transaction ID. Whenever a matrix client sends a message, it adds a unique identifier to the message.
So it knows when it gets it back. But this one doesn't have it. So the event must have come through
a different route. It must have come over the federation API, which came from a different server.
But other servers don't have this account, so it has to be forged. Now, in the meantime, whilst
we're investigating this, the attacker is causing as much damage as possible. We've looked
them out of my account. But they're going and exploiting the vulnerability in each of our public rooms.
One by one to ban every server from the room, so you can't communicate with people.
And we're already feeling the time pressure. It's now 5 a.m. it's been an hour since we were attacked.
And every member of the team who was a waked at point is following their own little path of
investigation. And I have a hot tip for you. If you are being attacked by a malicious actor,
you don't talk to them. They will mislead you. Next, I identified the attacker and started
a conversation with them. And from them, they were using a modified version of our software
to perform the attack. But they also sent next down a whole side track where I have been a cryptography
attack with signatures. It wasn't. And that was a waste of a couple of hours. The next real
insight that we had was, once we got access to logs from other servers, and we could see failed attack
attempts. And based on these failed attack events, we could work out. You know, the attacker
required knowledge of the room. They required author events. Now, author events are previous messages
in the room that control who has permission to do what. So when the room is first created,
when missions are set, when someone joined the room. And each of them has to have these
bodies, so that when the messages sent, if a server doesn't know about them, we are ready
if you can go and go and fetch them. And that just means that the attacker already had to have an
account in the room to perform the attack. So a private room we thought was safe. And the attack
was handcrafting events. So that explained all of the oddities with the first message. They hadn't
sent it through a normal client that manually typed out all the JSON. But it's half five now.
And one of our mentors has to go to sleep. And we want to call and now it's 10 past six.
And I have to go to sleep. So we call in external health, Olivia from the grapevine project.
So which is one of the other projects based on the same purpose. And we get some more useful
information. The attacker starts probing a server we have set up with the debug logging and instrumentation
on it. And we now know the last piece of the vulnerability. The vulnerability is one in a attacker,
or one at the server leaves a room. But at this point next expires, it's only Olivia left.
But at seven morning, so people should be waking up soon. Now, before we get to the vulnerability,
how do you leave a room in matrix? To leave a room, your server has to send a lever then. So
a message that's saying it's left the room. But to make that event, it requires knowledge of the
state of the room, who's in it, what the occupants are and so on. But your server, when it's
leaving a room, doesn't necessarily always have that information. For example, if you get invited to a
room and reject that invite, you've never joined the room that you're still leaving it. So
you have to get that information somehow. So the leaving server picks a server that it
announces in the room properly and has all that information and asks it to get all that information
for it and make a leave event. And then that server returns the event and the leaving server
signs that and sends it back for that server to distribute to everyone else in the room so they
know you've left. What could go wrong here? I wonder, I think there are a few steps in this thing now.
I mean, let's go through pointing into this code first. So for each, each server we know about,
we send that Federation request and we try if it's successful, we take that and we go to the next
step and pass that and get the JSON from it. We insert some metadata into that and we sign
that event and then we send it back. Yeah, there's still something missing there. Validation.
The attacking server could return anything in that response, not just a leave event, but a normal
message or a ban or anything like that. So it was time to devour the patch and leave here,
what was hard and makes the patch where we basically insert that Federation,
we check the event type, the sender, the affected user, the runer, the type of membership that
it is, whether it's a join or a leave or a cell and a cell on. But doesn't have some piece still.
How did they trigger that exploit? Because your client doesn't just leave rooms by itself.
You have to manually trigger that and at this point there are all sorts of wild theories going around.
Maybe it was a discord bridge that got broken or something like that. But I've opened up and
we now have access to the logs that continuity to the org. My own continuity to org, we run a little
box that has enhancements and this was the bot that was leading the rooms and it had a
plug-in on it that manages which means that it joins. But rather than just ignoring invites,
it would automatically reject any invites. So that was what was triggering the leaves.
So now we have a patch, you know exactly what happened and it's time to distribute the patch.
Now like I was saying, there are a part of an ecosystem that we've similar projects. So three other
projects are vulnerable and exactly the same way. The spec is also missing some validation,
not all of it. And some projects we have communication channels that we don't. And it takes
a few hours to ensure everyone's thought patch is prepared and everyone can hit the release button
at the same time. We released the security patch from the attack to the patch release. It was 13
hours and 45 minutes. We still have a couple of minutes. So we can talk about some of the ways
we're securing continuity against similar attacks in the future. We've added a lot of security
features. So we've added the ability to lock an account using an API so that you can immediately
prevent an account from performing any actions. We've added a command to force you look out
of all of the user sessions so that instead of taking ten minutes to do that, it takes us 30 seconds
next time. We've added configuration to find admins so that if the attack against access to the
server, they can't promote their permissions. And we've added hardening to make sure that commands
don't get executed outside of places by you'd expect them. We've also been working on projects
draw bridge over 100 commits, hardening the Federation APIs, which we haven't released yet,
but we're going to resume. Any questions? We've got one minute. So one question.
How difficult was it getting CDAs for vulnerabilities? Oh, that is actually shockingly easy,
but most of my maths gone. I can actually show you that right now.
Change. So, stop memory. On my screen right now, we have the GitHub security advisory.
Now GitHub provided a really handy feature which allows you to manage vulnerabilities, reports coming in,
and it assigns them IDs. To get a CDA, we just fill out the security advisory and hit a button
to request a CDA number and it takes maybe a week and they come back to with a CDA number.
And now, we can just click publish. CDA number.
Great. Any other questions?
Perfect. All right, then. Okay, I was on. Thank you.