Okay, hello, welcome to stop reinventing in isolation, bringing open source to trust and safety infrastructure.
First, I want to start out with a brief content warning.
We're not going to be talking like in detail about certain things, but we are going to be
talking abstractly about some of the sorts of harms that we encounter in trust and safety,
including things like child abuse, violent content, self-harm, unwanted content in other harms.
So I just want to give people a chance to understand that's what we're going to be, kind of,
I can abstractly talking about in case that is sensitive for you.
So this is a spoiler for what we're going to talk about, but first, who are we?
We are roost. We're a nonprofit building open source tools for online safety.
We are about one year old, right? We're informed about a year ago.
And yeah, each of those words I like to always point out, oh, it's not full screen anymore.
That's okay.
It's fine.
Each of those words is actually really important and really core to what we do.
So there's a reason that each of those is in our name.
It's more specifically who are we.
I'm Cassidy, I'm the community manager at Roost, and this is Ann, who is the CEO of Roost as well.
And now I'm going to hand this over to Ann for a trust and safety crash course.
All right. Excellent. We got this. We know that.
Somebody this morning said the best thing to do at Fazdom is improvised.
So I feel like we're, you know, we're doing well.
Yeah, so like Cassidy said, I think, you know, we've had some talks this morning about trust and safety.
How many people don't feel like they are familiar with what that generally entails?
Oh, and I mean, that's pretty decent, but there's a solid group of hands down.
So to give context for the rest of the demos and things we're going to show, I'm going to do a super speed run on what is it that we talk about and I'm referring to.
Because I think sometimes we use very abstract words to not actually directly mention the things.
And I know you all probably responsible for instances and servers and breams.
And so that's why we want to share a little bit more of the specifics of what this work entails and what we talk about.
So you can think, oh, I haven't thought about that as something that could happen on my platform or I haven't investigated enough of how might I prevent that.
So the super crash course, to frame things before we get into specific harms, you kind of just can imagine and Cassidy picked Cass.
So if you are a cat person, you can take it up with him, but.
So, you know, if you think about like, here's the thing that this is a harm, this is a thing we've decided.
This is a not good thing for our organization, our community.
You might want to think about, how am I going to identify that? How am I going to block that? How am I going to watch what might be going on?
To then decide, now it is at a point, it is a threshold that I need to block that.
Someone earlier was talking about the difference between reactive and proactive detection.
How are you going to find these cats in the first place?
And, you know, cats are crafty little adapters.
Look at how they went from big cheetahs to cute little minksy things.
You're going to also similarly follow the evolution of that as a harm type.
Our colleagues are incredibly brilliant and they had developed a framework around this to help you remember like, how does this workflow work?
What does this look like? They called it the dire framework a little cheesy because the situation is dire.
But it is a very helpful memory tool.
So you start at detection like we were just talking about, you know, I need to be able to identify what's going on.
I need to be able to investigate it and not just find here all the instances, but what's maybe the underlying pattern of, you know, the question behind the question of,
where are all these instances coming from? Why are they coming from the same place?
Or maybe the same users, the same IP addresses.
Things like that, you're then going to need to still review all those individual things and figure out how you're going to take action
and enforce that, including also things like someone asked a great question about what do appeals look like?
What does audits look like? How do we know that we can review as a group looking back?
We've made good decisions or actually this is a place that may be upon reflection.
We should have made a different choice, but now we have those audit logs to have that kind of self reflection.
The other thing to keep in mind as, you know,
regulation around digital platforms is rapidly changing and depending on what jurisdiction your community is in,
there might be laws where you are required to have things like appeals process and audit logs.
There might be laws around the types of material you are required to take down or to take certain action on.
So in particular, we're talking about things like child sexual abuse material.
If you're not familiar with that term,
Locally you might call it child pornography, but we like to use child sexual abuse material to distinguish between situations that have consent
and a child obviously does not have consent involved.
So harm type, CSAM, same thing with terrorism and violent extremism content and different jurisdictions.
There are certain things that are defined as that and you might be required to take action.
This is a rapidly changing space. It's a little overwhelming, but it is something to keep in mind.
The problem with all this is that everybody is trying to do essentially what is the same thing.
There's nothing about this that I mean from our belief should really be something like a product differentiator of,
you know, you come to my platform and the difference is you're not drowning in violent content and child abuse.
That is not the state of the world.
This is really if we think about our digital spaces as public goods, a very important part about what society means today.
That safety also needs to be equally available to everyone.
That's a really big thing behind resounding, but you can think about this in a couple different ways.
Of we're all trying to figure out what are the things we should be detecting, how are we going to take action?
What are these tools look like?
And in trust and safety for a long time, you know, people have been relying on interesting vendor tools, proprietary tools,
but what it means is if they're like, well, I can't afford that or I can't access that tool.
They're sitting there writing very basic review content dashboards over and over and over again, rather than saying,
I know that there's this particular group active on my platform.
I really just want to dive into taking care of that, but I got to start back with a very basic review panel.
It's not a great use of time.
So this is just a brief flash I want to put on the screen here, not to overwhelm anyone, but as you might think,
you might see something and say, oh, gosh, I'm not familiar with that.
I might want to jot that down and think about doing a little bit more learning and figuring out, is that something that could be prone to happening on my platform,
or if you go on to do further reading, these are just common terms that you might see come up.
So we wanted to share the acronyms, I think, in trust and safety.
There is, there's a lot of shorthand, there's a lot of acronyms, and we wanted to get folks just a quick moment to think about different things that might be happening.
And thinking about learning, the last little part of our crash course.
Rather than listing every single resource out there, because I think there are a ton of really wonderful and helpful things,
I figured I would list maybe a first stop shopping rather than if you're familiar with the phrase like one stop shopping.
A first stop of here are some two great free resources from the trust and safety professionals association that if you were to look them up,
you would probably find something and go, oh, I'm going to pull it that thread a little more.
I'm going to go research more about this, it'll take you down some great rabbit holes, but I figured that is a good first stop for folks.
So with that, I'm going to hand it back to Cassidy for exactly what we're doing.
So yeah, let me talk a little bit about our nonprofit and open source approach and why we think that's the right way to do it for trust and safety.
So yeah, the status quo is kind of talked a little bit about in the crash course there is that trust and safety is kind of broken in the industry right now.
Smaller platforms or newer protocols might not know that they need these tools.
There's not a lot of like open resources that are easy to find and just intuitively know that you need these tools.
And oftentimes, like answer that industry tools can be unaffordable or inaccessible depending on what, you know, how big your organization is or what type of platform you are.
And even when you can get access to those proprietary tools, a lot of times there's lengthy application approval processes that are just really hard to follow through with when you're trying to actively, you know, combat abuse on your platform.
And again, we talked about reinventing basic functionality instead of focusing on specific harms.
But we're also seeing a generative AI and LLMs accelerating the production and spread of harms and that's a very real thing we're hearing from platforms that like we're already outnumbered by a spam bots or moderators are already outnumbered by spam bots and now they're like these people who are spammy are being given these tools that can just rackably generate more and more content.
And then in the trust and safety industry openness is kind of bent taboo historically.
There's a sort of sense that if we talk about the trust and safety, then bad actors will figure out how to get around our protections.
But as I hope we all know, security through obscurity is not really a valid thing, but it is that mentality is still common in trust and safety.
So what do we believe?
We believe that the missing piece in all of this is really tangible tools.
The actual tools code that can be used by platforms, not more proprietary platforms that can be used but just like actually building tools that can be used.
And these tools can and actually must be open source so that they are accessible so that they are modifiable so that they're auditable.
And so that they're part of a public safety commons so that you can actually like we can actually all build them these things and make them together.
Which again audience, it falls in my feel like understands that, but these are the sort of things that oftentimes in the trust and safety world is kind of a novel concept.
So roost from the start, we're very open source.
We're all in an open source, both licensing like technically it's open source but also the development model of open source, the community driven development.
We've talked, if you've been here for a couple other talks this morning, there's been lots of mentions of HMA, which is probably not actually out of meta, but we've kind of taken on the community aspect of HMA, so we run the office hours and help kind of people get together and contribute and collaborate on HMA.
We also host something called the roost model community, which explores ways to use and improve open weight models in trust and safety, so it's using some of the best automation tooling.
And roost itself, we really strive to be open source in every way possible, so our things like our roadmap, our community documentation, our government documents, things like that are openly available and actually open for feedback from the community as well, because we really want to show that open source is the way to go and transparency is key.
As far as specific projects we're working on, there's a project called coupe, which is actually going to be releasing very soon, which is a review console that we acquired and are working to actively open source.
And then osprey, which is an automated roost engine that actually originated from discord, and that's what we're going to talk about a little bit more here, which is still me, yes.
So osprey, we like to, Julia kind of coined this term, or call you calling this term, you can't talk.
Automate the obvious and investigate the ambiguous, so kind of think about how that works with trust and safety and what osprey is.
It's an automated roost engine, again, it originated at discord, so this was actually discord built internally for their trust and safety tooling, and we open source to at roost.
In partnership with discord and in-dev, and it's built for incident response, so when you have an incident happening on your platform, it's how you can actually go in and respond to it and adapt your approach.
It's self-hosted, it was open source, it's self-hosted on your own infrastructure, it's not a service we're running, it's a tool you host, it's highly scalable with built-in load balancing and it ingests tons of metadata.
So events come in, not just the actual content of a post or of a message, but you can have image or video caches for hash matching, you can have the actual text content, but also like IP addresses and headers,
basically all of the metadata so that you can then perform investigations to actually dig in and find the patterns of attack.
So when I'm used, going back to, I guess, the cat analogy, let's say you have an emerging incident happening on your platform involving cats, which you've decided are evil and you're not going to include on your platform.
Maybe there's word of a cat manifesto being disseminated on your platform, maybe accounts are linking to a cat related site in their bio and there's spamming people to go, make them look at cats.
And, or maybe accounts are joining public channels and just spamming tons of photos of cats, which some of these might sound familiar if you're aware of things that are happening in the decentralized space.
So, Osprey helps in these incidents by helping determine the jurisdiction for the incident to aid in reporting, so let's say you have something being posted and you want to know what local authorities do we need to report that to.
Like, this is illegal content, we need to report it somewhere, where does that go, where is this event happening.
And also, you know, you can search for that cat manifesto, the title of it, or key phrases to create new labels and rules to automate things.
And you can also perform really deep technical analysis and investigation using all that metadata we've mentioned.
So we're going to do a demo, is it the fun part, which on fuzz and wifi, we'll see.
So, this should have been running in the background the whole time, so we have lots of spamming cats.
Oh, how are we going to display good, hmm, good drag it over, we're just going to be fun to drag it.
You want the mic for button to search.
Okay, so this is Osprey.
This is Osprey.
This is Osprey, it is, this is the dashboard for it.
So you can see there's tons and tons of metadata coming in.
The, how you kind of can do performing investigation is you will write a query.
And so on top left there's query there, this is using a special query syntax.
Some made up language, SML, great.
It's SQLish, yeah.
So we can have these different rules that are defined actually by your, the, by obviously running on your platform.
You can define the rules that you have implemented.
We're good.
Okay.
So we're going to search for, we're saying there's an attack with like scammers or spambers or something.
So we're going to search for a suspicious display name, which again, this is a rule that we've created on our platform that is implemented in the code.
And we're going to see if the query runs on the Wi-Fi.
It's good. Hey, something's happening.
And yeah, so we're performing just a really simple, like, hey, doesn't match this specific rule that we've created ahead of time.
And you can see, yes, there are, there are posts coming in that match this rule.
And here's the time chart of these, these different posts.
You can break it down by different, you know, by out day, hour, minute, et cetera.
You can then also, like, drill down.
So you say, okay, these were things that were flagged using this rule, but we want to drill down even further and see if, like, check out the display names that were actually flagged in this.
So we have, you know, users on our account with display names like free giveaway and DM me now and click here.
These were flagged by the rule, but we're coming in and kind of confirming and labeling them maybe more specifically to aid in other automations.
So do all of these have labels.
Yeah, here's one that got, no.
That one shouldn't have had the label, that's fine.
But you can go in and add a label.
Let's say you, we're doing an investigation and you wanted to say this one is, you know, a crypto scam or this one is a,
some of the kind of scam you can label and more specifically that way.
The ends are also spam.
Yeah.
Oh, yes.
And you can have, it supports expiration of these labels.
So these labels can support enforcing different rules on your platform.
And you can set an expiration.
So it can maybe they'll be shadow vander banned for a certain amount of time.
And then once that label expires, then they'll post would come through again.
Yeah.
And you can see the somewhat overwhelming amount of metadata here that we're pumping in the, the,
Oh, that opening in the new tab.
Optimizing.
Yeah, this tool originated at Discord.
So some of the loading messages are very fun.
And when you delete a chart, you eat the chart, which, yeah, you know, it's fun.
Or going to eat it not is open for a comment.
Yeah, eating, eating the, yeah.
It's an open source project.
We might change it.
So let's say you found a pattern here.
You said a lot of the, a lot of the content is coming from the same IP address.
So you could, you can see that and then you could drill down even further in
Osprey to filter it by that IP address.
So you could do the IP address equals.
Yeah, it's so hard to, this is not the great demo environment.
Oh, yes.
It's a string.
Yep.
And then string.
So you match that.
Again, this is using the, the SML markup.
I'm looking at our time on what we should accelerate the demo.
So we can do the rest of our content because we started a little later.
And we should answer questions.
Yeah.
So it's, it's a little complicated to demo because there's so much going on here.
But the idea is you can really drill down and investigate specific attacks and find patterns.
And then you can go back and you can save these queries so that you can, you know,
investigate and follow up on these investigations later.
Every, every action that you perform in here from labeling and any actions that you take
are added to an audit log as well.
So that helps in, you know, reporting and, um, yeah.
I think we switch back to the,
What's it?
We're having a demo more one-on-one if you can find us after the talk as well,
to like, go through a, uh, of more kind of specific scenario, too.
Okay.
And we're going to find this slide back here.
Great.
That's a backup video.
See, I'll just click the next one.
Yes.
All right.
So something important to mention to us,
Osprey is production ready.
It's actually being used in production today across both discord and blue sky.
Um, at blue sky, they have reported that they're processing over 45 million events per day with Osprey,
um, with over 100,000 daily enforcement actions that are being taken from just their first set of rules,
like the kind of the first pass of like,
let's set up these rules and see how we can actually, uh, implement Osprey.
So it's being used in production there.
And at discord, again, this tool originated discord before it was open source.
And now they've reintegrated the open source version at discord.
And they, they handle orders of magnitude more events than that, um,
every day being processed.
So it's, it's truly, truly scalable and used in production and production ready.
Um, it's also being used by matrix.org right now.
Um, they're using, I think, in staging right now,
but they're bringing, you know, these trust and safety tools to, um,
the open source decentralized network.
Um, and importantly, it's open source.
So you can get involved.
Um, I want to hammer that point home because that's kind of why we're all here, right?
Um, we can't do this alone.
Uh, we can, we can do a lot, but we, we need everybody's help that, that we can get.
Um, come talk to us, come find us.
You can find all the links on our website.
Uh, the code is all up on GitHub.
Um, we have a discord channel.
We know we're in the decentralized communications room.
You can come yell at us about that on our discord.
Uh, you can, uh, try using offspring or any of the other open source projects.
We work on our support like H&MAT or R&C models.
Um, and share your feedback with us.
Feedback is a really super valuable way to get involved actually.
Just like, hey, I have a platform as doing this.
Um, we tried implementing it.
Here's what we learned.
That's super valuable for us to know.
Um, what to focus on, what to build next.
Um, you can also join our office hours.
We have things like, uh, every couple weeks.
We have office hours for different open source projects.
And so you can come in and chat with us there and and get help implementing things.
Um, and also you can help us by building integrations.
Like, uh, Matrix.org foundation has been doing with, with Matrix, which is super valuable.
And that is all of our talk, but we'd love to take a couple questions if there's time.
That's a great question.
The question was, uh, specifically for Matrix, would this be a policy server implementation,
which is probably a good question, actually, for Travis or, yep.
Um, but yes, probably.
It would be an, it would be implemented at the policy server level.
Um, and we have a lot of questions.
Um, and we have a lot of questions that there's time.
That's a great question.
Uh, specifically for Matrix, would this be a policy server implementation, which is probably a good
level.
Yeah.
Um, does next sense to use it also for less events, like, maybe, so for some more projects?
So questions would it make sense to use Osprey for fewer events than, like, the scale of having millions of events?
Um, yeah, it's, uh, I don't with Osprey we've said.
I mean, you can take that.
Yeah.
I mean, I think it would, that would greatly depend on, um, your teams, you know, comfort level with
the tool like that and ability.
I think it's really helpful for if you're sort of a, have any background in data analysis or
investigations.
It's a very helpful kind of automation tool in that way.
Um, if your vet load is very small and your team is very new to that.
And might, I could, I could say being a little overwhelming.
But I think if you get comfortable with the query language, it's really helpful.
All right.
I know we like to say, like, Osprey is one of those tools that you don't know you need it until
you need it.
Um, it oftentimes feels like way overkill and you don't.
It's like, oh, yeah, yeah.
We'll, we'll handle that when we have it.
And then your platform scales to a point where you get a big attack.
And then you, it would be really good to have had something like Osprey.
So it is kind of a balance of, of that.
Any questions?
Yes.
Okay.
Do you guys have any account levels filled with questions?
Do we have any account level fraud tooling?
Um, yeah.
Yeah. So I know the examples we gave were things like, um, see Sam and T back.
But a lot of folks are thinking about Osprey also for fraud signals.
Because you might look at, um, and you can define this in Osprey.
What time was the account created and how quick was it to a first post?
Is this account, you know, links should IP address that has created 10 other accounts within a specific amount of time?
These are all very user defined.
So I think there is a, yeah, definitely for fraud creation is.
Yeah.
It's important to note that like all of the, all of the metadata that's coming into Osprey is like that's up to the platform to define.
So your platform if IP addresses don't make sense for your platform, like then you wouldn't be feeding IP addresses into it.
So all of the metadata that's coming in is actually like up to the platform to implement the defined.
Yeah.
Oh, go ahead.
What are your self posting requirements like, and how are your deployments right now?
What are your deployments like?
What are the deployments like?
That's a good question.
Are you pulling up?
I don't remember where that is.
Yeah.
I think Haley has a good.
Okay.
So yeah.
It's part of it is, you know, with this with the, a horizontal scaling that Osprey can do it depends on how many events you're processing.
That said, the demo we were running, you can run it on a laptop.
So it's not like this massive, massive thing.
You need a huge server for necessarily, but it really does scale with the number of events here.
Yeah.
Do you pull it up?
I didn't find it, but here's my, my promise.
So blue sky has actually created a very wonderful helpful architecture diagram that they posted of how they got it set up.
So you can see, you know, general line of all different requirements.
I will find it and put it on our, my blue sky.
And our time is up.