Next up, we have David Ratzel from the Master of Project, who's going to be talking about
the fetus-covory work they've been doing with Master of the Project.
Thank you very much for having me.
Welcome, everyone.
My name is David.
I work as a web developer at Master of the GGMBH, which is the non-profit organization
that oversees the development of the Master of the Software.
And actually, I'm a longtime fostering fan.
It's my first fostering for 10 years due to different reasons.
But before that, I came here all the time, as evidenced by the stack of t-shirts I had
to go through a couple of days ago.
I never imagined that I would one day stand here in front of all of you, so this is really
amazing, but also very nervous, so please bear with me.
I'd like to talk about a search and discovery on the fetiverse, and with this topic, there
is one huge problem that we have, and this problem can easily be illustrated.
Let's say you are technically inclined, and you would like to set up a Master of the
Server for our friends and family.
By the way, I'm using Master of the GGMBH as an example here, just because that's what
I am familiar with, but similar things will happen with all the different activity
pub-based software projects.
So you have this new Master of the Server, and the first time you look in, you will
probably see something like this.
This is an empty timeline.
This is not nice, but maybe, I mean, you are technically inclined.
Maybe this is not totally unexpected.
So the first thing you will probably try to do is find follow.
And you might have heard about this, this garg-run guy.
He seems to be very popular on MasterDone, so you want to follow him.
And the obvious thing to do is just enter the name in the search bar, and if you do that,
you will see, this here, no joy.
But honestly, you don't care so much.
You only came to MasterDone because you are promised cute cat follows.
So the obvious thing to do is search for cat follows.
And if you do that, you will see this here, again, no joy, sorry.
This example is extreme, of course, it's an extreme example, but it's one that everyone
on small and even mid-sized instances will be very familiar with.
And even on MasterDone.Social, which is the largest instance of MasterDone, there is, it's
the one we operate.
We regularly get reports of people leaving the platform because they couldn't find the
people and content they were looking for.
Even though it's clearly there.
So the reason this happens is pretty clear, with all the February servers, happily
federate with each other using the shared protocol, activity pub, when it comes to search
and discovery, every server is on its own.
Before I introduce you to our idea, what could improve this situation, and you take a
little detour, and talk about service providers for a minute.
Many February servers today already use one or more external service providers.
The most obvious example being storage providers, something like S3, or compatible services
to store media files.
But keep that idea in mind.
We have external service providers helping out with search and discovery.
Once we have that, a February server could use one of those external service providers.
And it might, some service might use a single one, service could also use more than one
external search provider to help with search and discovery.
And while every single server only has a very narrow view of the full Fediverse, as soon
as two or more separate servers use the same search provider, this search provider has a chance
to get a much broader view of the large Fediverse.
And this is an idea that we are working towards.
We called our project, had this discovery, and we have a couple of goals for this project.
First of all, we want to try out this provider idea, the idea of having external service
providers solving problems for the Fediverse.
This is our first proof of concept of this idea.
We don't want to build a project that's only useful for MasterDone.
On the contrary, this project is only successful if it will be useful to others as well.
To that end, we are not just writing software.
We are writing specifications first, open specifications that anyone can implement.
So everyone will be able to write their own provider, and also every developer of Fediverse
software of existing Fediverse software projects will be able to integrate this into their
project.
And to make this all work, we need to work together with other projects.
We've tried to reach out to other projects, but we probably didn't do a very good job of this.
So if you are an implementer of an activity-pop-based Fediverse software, and you think
this idea might be useful to us, to you, please come talk to us.
We will also build an open source reference implementation of this.
And I said it before.
We also have this open specification so everyone can build their own.
We are very lucky to secure funding for this project, the NGI search organization, helped
us out with that.
If you have a grant from some of these organizations, you know, that you are working on
a timeline.
We have some very tight deadlines to meet.
The very next one is actually at the end of this month.
So I said please reach out to us, but I want to apologize in advance if I'm not as responsive
as I should be in the coming four weeks, because that's the next deadline.
The end of this project is scheduled for June.
But the important thing here is we won't stop working on this just because this grant
has ended.
We are in this for the long run, and we hope to build something useful together with others.
I said we have specifications, and we put them on GitHub, or rather the first drafts
are on GitHub.
This repository here, it's this very long name again.
So you can't miss the repository if you look for it.
In this repository, you will find a couple of drafts, and the first one is concerned with
general interaction between providers and favor service, things like registration authentication
and so on.
These can be reused for other purposes.
Then we have data sharing, which I will talk about in a minute, and we have an open pull
request for the first user-facing specification, which is Trends.
Very soon you will find a draft for account search as well, and until the end of June
we will also have account recommendations.
Because not in this NGI search project is the big one post search, but it's on our internal
roadmap anyways.
We needed to cut the scope of the NGI search project, so this was the one that we had
to leave out, but we will work on this after June.
So data sharing, this is probably the big one, because for the search provider to be able
to return results, it needs to index content from the Fediverse.
And this is of course a difficult topic.
There are certain privacy expectations on the Fediverse, and we are very well aware of them.
So the concept we came up with, and that we want to run with a little bit, is the following.
This master on server on top here, this is not using a search provider, but it is using
activity paths, so it's federating with other servers.
This generic activity paths server on the right, this is a server that is using a search
provider.
And once new content arrives, and it knows of a new post made on this master on server,
it will notify its search provider of that new content.
But it will not send the actual content.
It will only send the URI, the idea of the activity path object.
So the search provider is then responsible for actually fetching that content.
And it will do so using a signed request.
If the master on server on top wants to check the signature, the search provider will actually
pose as an activity path actor, so the signatures can be very tight.
We take privacy and content very, very serious.
And I'd like to point out, first of all, that we will only ever index public content.
And if you know a bit about master on, we actually have different levels of public content.
We have something called quiet public.
It's used to be called unlisted, where you say, hey, you want to publish something on
the web, but you don't want to announce it in any way.
And we will not index those posts.
This is only for public public content.
And the Fenver server is responsible for only sending public public content, but also the
search provider needs to double check that this is really the case.
The same goes for consent.
If you want to check if an author of content has opted in to being discovered and being
indexed, we already have that in master on actually.
We have these two flags on the actor called this coverable and indexable.
And they exist today.
And I know of other Fediver software projects that have implemented them as well.
This coverable means you as a person or an actor have opted into being discovered on the
Fediver's.
While indexable means you have opted into your content to be indexed, so it can be found.
And this is there today, and we will of course respect that.
And again, the Fediver server that shares URIs, we need to make sure that these both are
set true and the receiving search provider will have to double check this again.
You might remember that I said a search provider would do signed requests.
And this means that Fediver's service that support this can check the signature, check
the actor, and that means those requests can be blocked.
You can either have a server level block or even an individual user could decide to block
search providers.
So we have different layers of security built into this concept.
We hope that this is enough, but if you can think of any ways we could improve that,
please let us know.
The first use of facing capability of a search provider that we are currently implementing
will be trends.
So we define an API for Fediver's service to ask a provider what is currently trending
on the Fediver's, which posts, which hashtags, which links are currently trending.
This is still an early draft, but you saw the timeline we will probably have to implement
what's there today, which doesn't mean this will be finished at the end of the month.
We will continue to improve this, but don't be surprised if we merge this open PR without
much further discussion.
And similar thing goes for a counter search, the specification is not quite ready yet,
but we expect the first draft to be quite simple, a full text search of all the public
information on actors that describe them.
Again, feedback is very welcome, even if we will probably have a first implementation
of this draft, we will continue improving that in the future.
In order to be able to implement this, and we are working on a reference implementation,
I decided to pull out a couple of things.
You may know, Macedon is a Ruby shop, we are using Ruby on Rails, and so our reference
implementation will also be based on Ruby on Rails, and if you are familiar with Rails,
you might know that it has a plugin system, and we extracted two plugins from our reference
implementation, so everyone can start their own provider project.
So if you don't want to bother with all the authentication stuff and registration stuff,
we built into this, you can simply use these ready-made plugins if you use Ruby on Rails.
I just opened up this repository two days ago, it's still a little sparse on documentation,
but if you're feeling adventurous, please give it a try.
The reference implementation is not there yet, it will be at this repository, and well,
you saw the timeline, it will be there soon.
Before I finish, I would like to address some questions that we got over the past months.
And the first question, I think I answered this already, but it's the most common one,
is this only for Macedon, and I said it, no, of course not.
On the contrary, this project will only be successful if it will be useful to others as well.
Another common concern is it doesn't just lead to centralization, and when I first heard this,
I was actually offended by this, because we are going out of our way to write specifications
to help others implement competing implementations of this.
So we hope to have several different implementations of this and many different installations of this.
But I said it myself, it will start becoming useful once more than one,
server uses a single provider.
So yes, there is some kind of centralizing force, but I, personally, I am not worried.
I know a lot of Macedon admins over the past couple of months, and they are well aware of this problem.
So I don't expect a single player to barge in and say, hey, everyone, please use my central search provider.
I don't see this happening anytime soon.
And another concern is of course privacy, but I think I addressed this.
Just to recap, we will index public content, only public public content to be precise.
We respect consent, and it will be blockable.
The last concern I heard a couple of times is, isn't this just bespoke APIs?
Yes, it is. This is not an extension to the activity part protocol.
And I think some people may be sad that it isn't, but I know others will be very happy that it isn't.
For the moment, this is just a set of APIs we define that we hope will be useful for many different projects.
It might evolve into something else at some point, but for the moment, this is what we're doing.
And I said it a couple of times now, we need others to help us out, to work with us.
And if you would like to contribute, please go to this specification, GitHub repository.
The one with a very long name, you cannot miss it.
And come talk to us directly. If you are here at Phospham, the best way to do this is visit us at our stand.
It's on the ground floor of the age building over there.
So I think I'm still in time. Thank you very much.
APPLAUSE
All right, we have some time for questions.
You want to go first here in my background.
Andy?
Are there two of the sort of single-central search providers problem?
Are there any plans to allow the providers to federate that information between themselves?
So you end up with a distributed index as well?
Not yet, but the idea is super interesting.
Sout of scope for now, because we had to cut the scope a lot, but certainly interesting, yes.
Yes?
All right.
Before by sputtering instances, question mark, I think I addressed this.
We are not going to crawl or spider the web or the featherers on a contrary week.
I explicitly do not do this.
I talked about these different levels of security we built into this concept.
And actually, there's one of the things that will be resources intensive, crawling would be easier,
and we would get a lot more content.
But we will not do that.
Yes, so question, is a website owner?
I'm not very popular on a mobile phone, I have four followers.
My wife and follow me yesterday.
Don't collect it, anyway.
If I post something on a mobile phone, if I post something on a mobile phone, I get
data by, I think, the well-known data's effect.
All the different instances, effects my website for the little thumbnail thingy, and my website
goes gloriously down, it's a heavy server, I'm not very popular, I think this is a big issue.
Anyway, this may help if the kind of metadata is also shared, and all the different instances
do not need to refetch data.
I think out of scope, but will I need to worry about not only being data by hundreds of
multiple instances, but now also with a few search providers?
There will always be a lot less search providers and a method on instances, so I wouldn't
worry about this just yet.
This will not solve this problem, but we are well aware of the problem, and we will
need to solve this soon.
Okay, thank you.
We're also working on that.
So if I understand this right, the order of magnitude of the data that you have to send
to search provider is sort of in a similar ballpark as what the data is that's being posted
to a particular server.
If you aggregate a whole bunch of them, this becomes a very significant additional resource
you can get a consumption in terms of the cost of operating a message in instance, and then
if you have a search provider that aggregates 100 different servers and it has to have lots
of them, otherwise it is not particularly useful, that becomes quite a research-intense
thing.
On the other hand, that research-intense thing is not something that's use-a-visible,
so you can't just go to the end users and say, hey, I have this cool master instance
for the XYZ community, you know, help me support it.
It's sort of something that's not really seen, so it's harder to get donations, so other
kinds of funding for it.
What are you thoughts around this?
I don't have many thoughts on this yet.
I've said it before.
We don't know exactly how resource-intensive this will be yet, and we will learn a lot about
this in the coming months, so we will have a much better idea, a couple of months' time.
On the other hand, I know a couple of method-on-appments.
Some of them have a lot of resources at their hands, so for some of them spinning up
more servers isn't really an issue, so I would expect several community servers to step up and offer
their servers.
We as Macedon might do the same, we don't have any concrete plans on this yet, but of course,
we may do this, I don't know, but yeah, I think we will see if this works, if this
works at all, and we will see a couple of servers.
It's probably not something that a single user instance admin would run.
Do you have something like a developer room in the matrix for discussion?
Partner didn't get that.
Do you have something like developer room in matrix for discussions where the place where
you communicate?
No, we don't have that at the moment.
We're going to take maybe one or two more questions for note.
Do you know?
Okay.
Come to this time that the user's question is coming so smoothly.
Oh, that great.
So if you have questions about this implementation or about Macedon,
Macedon does have the stand, so please come ask questions.
Please visit us.
Maybe one or two more.
Image is going to be indexed by cash also.
I'm sorry.
Image is going to be hashed in searchable, like.
Image is.
Yeah.
No, this is totally out of scope for now.
Okay.