Yeah, let's figure out how we do to enter the matrix.
I'm also very interested by saying that, and thanks for coming here.
And in this big round of people, thank you very much.
And a big applause.
Thank you very much.
My name is Mikhail.
Here we have Gregory, Nils, and Melon.
And we are part of the Ubuntu Matrix Council.
And Melon is part of the Ubuntu Community Council.
Excellent.
Yeah, first let's have a short call.
Yeah, I did.
First let's have a short call to see how many people here are using matrix.
How many are using R.C. and how many of you are using both.
Okay.
We're going to use our hands.
Yeah.
No, no, no, no.
Yeah.
We wanted to do it on matrix, but then the results might be a bit skewed.
Yeah.
Sorry.
Yeah, sure.
It comes to slow me.
Flip part you to see.
Maybe we can go to the next slide.
There's going to be an out.
Yeah.
The QR code is here again.
So a bit of history.
Why did we need an alternative to the R.R.C. and other chatting platforms?
First of all, we are using all kinds of different platforms.
For example, canonical is using metamose.
Other communities are using this code.
Telegrams like rocket chats, etc.
There's a little bit of this joint communication between groups,
teams and communities.
So we figured that it was time to find something that we can all commonly use.
And that we can all commonly collect.
Some of you might say, why not R.C.
It works.
It has worked for decades.
The problem is that it might work for all the folks or more traditional folks,
but as you can see, even within open to communities,
the message count for IRC support rooms has been steadily declining.
So why is IRC declining?
We figured that it was mostly because of these three causes.
Like of modern features, most people actually like sharing images, links,
widgets and occasional gifts.
It is kind of obscure to get started for newer contributors and newer members of the community.
Getting push notifications, getting AFK messages.
It's not ideal.
Let's say that way.
And there are no pretty much no IRC users apart from Linux nerds.
And we might want to embrace other people than us.
So we started our journey.
I think that you recognize this expertise in the comic strip.
It's a little bit modified.
So we are currently somewhere around here.
We've got almost everybody on Matrix.
There are some people that will probably stay on IRC until the heat death of the universe.
So we can do anything about that, but we can still maintain a bridge to the IRC.
So at least they can be somewhat happy.
So why Matrix?
Matrix is a modern chat platform.
Pretty modern.
Where you can use a single account to talk with multiple communities.
I think that you probably all know that from the past few thoughts.
And of course it's open.
It's open source.
So that's a big plus.
Other reasons it's popular in Linux communities.
A lot of other communities such as open-source federa.
They all use Matrix.
So we can already talk to them.
It's a similarly open to IRC.
And there is encryption possible.
We don't really recommend encryption in group chats,
but that's for another slide.
And there is adoption by some large players.
So I think this pretty much summarizes our experience
and most people's experience with Matrix.
It is a wondrous land, but here beat dragons.
One of the issues and so the rest of this talk basically we're going to talk about some of our issues
and how we solve them or are trying to solve them within our community.
And the first issue is that Matrix is an open federation.
This is an advantage, but this is also a disadvantage because a lot of people
come from different home servers from different communities.
They come join our rooms and they don't actually know our customs.
Because of this, we are creating a culture inside of our own rooms.
And we use defense and moderation for that.
We basically all typical moderator actions.
We basically divide them in two types of actions.
The first type is defense.
This is specifically about combating spam.
The idea of defense is that this will happen very quickly.
If you're spamming multiple images or not say fork images,
you will get immediately banned.
Most of this happens automatically or automatically using a whole bunch of volunteers in the background.
This is without mercy.
You don't get a lot of ways to appeal such a ban because you've been spamming.
And this is also without context.
Which means that we have a team of defenders who defend the entire home server.
Some of these people actually are from other communities.
And they simply, it's very easy.
If it's spammed and it gets blocked and it gets blocked across the entire federation.
Even if the user, for example, would be a user that has been a long time participant in a room.
Even if that participation initially used to be positive,
they will just get blocked immediately.
But this is only for spam.
And the second type is moderation.
Moderation is basically coaching users to communicate better.
We and the Ubuntu community, we have a very strict code of conduct.
We have very high standards in terms of communication.
Standards that we have applied to our own discourse forum,
for example, for a very long time.
And that we are now also applying to matrix.
But this often requires actually coaching people,
actually giving people feedback, allowing them to learn from their mistakes.
And ensuring that people who want to improve, who want to learn that they can improve.
So this is a process that is very slow.
If you are a participant in our room, and sometimes you use two file language,
or sometimes you are too abrasive,
we will contact you.
We will work with you.
We will coach you to become better part of our community.
This is also very manual process.
Basically, each room has a moderator or is should have a moderator.
That will just contact people when their behavior is not in line with our code of conduct.
Failure is a teaching moment here.
And this is always done by people familiar with the room tone.
So some rooms will have much trick tone and other rooms.
Other rooms people can maybe, maybe banter a little bit,
depending on the room tone.
And so this is always done with context,
with the context of which room is this user communicating with.
The second challenge of the open federation is that the internet is full of
gadgets who like to troll.
Basically, Matrix has a very big issue of spam waves.
Spam waves across the effort.
Across the federation.
The single one project that helped us enormously with combating this is the community moderation effort.
Also known as CME.
CME is perpetually working on a website to better explain what CME is.
But this website is still not online.
So I will explain it.
I will try to explain it myself a little bit.
CME is just a group of Matrix room admins.
Mostly of notable open source projects like Ubuntu,
Fedora, Mozilla, privacy guides and some people who are contributing to the Matrix standard
and are who are creating moderation tools.
They have a shared ban list to fight spam.
Only very, very clear spam.
But when spam is detected.
Anywhere on the network of all the communities who are connected via CME,
they will be banned across the entire network immediately.
And then there's some very, very strong bands ACL bands.
If an entire home server is deemed to be incredibly toxic,
too toxic, too just banned individuals from that home server.
CME is also a community of all these admins who coordinate of the fence who coordinate the fence during spam waves.
And so we also coordinate on how to improve moderation tools, building moderation tools and things like that.
And the idea is that only trusted members can actually update this ban list
because this gives you power to ban someone across the entire network.
Miolener and Draugner, basically specifically Draugner,
is one of the tools developed by some of the people of CME.
That's me.
Well, the automatically spam response is mostly done by CME.
And it works a long way for the spam that Miolener described right now.
But further of that, we had to develop some own tools for the community that worked for us.
They may work for others.
The codes and links are here.
Can get through some of them, maybe.
The first one is actually Ubotu is more or less like a conversion of our old RC board with some added functionality over the time.
So that's most of our public rooms.
The bot is there.
That's stuff.
But also, that's some spam detection now, like if you spam multiple images in a short time.
And it also does a not-see-for-work check.
Then those images get redacted and you can spam any more for a few minutes.
It's configurable.
Yeah.
Sign up.
In my checker is actually a module for signups that more or less improve the limitation that people can invite you to other rooms.
Because part of the spam was detected.
Hey, we can spam images in me more.
But we can just invite people randomly to our spam rooms.
And that's annoying, too, of course.
So this kind of combines the CME list with our own list that we can add.
Home service to that are blocked from inviting people.
That only helps with people from our own home service server.
So every home server would have to implement that.
Yeah.
The next one is the not-see-for-work board that's basically in addition to my do not spam.
There's the images stuff that's happened a lot actually.
And if you want to get more into those malware,
the framework to write easy metrics boards, we did at the last.
We did a workshop about that.
That also goes into detail how to set up a board and do some basic steps.
We didn't have time to go to the whole workshop here.
So if you're interested, please go there.
Yeah.
The last challenge that everybody knows, I think if you use metrics in any way,
that I think there's, I don't think there's too many clients.
That's, I don't think that's even true.
They're just not enough clients that actually can do everything or most of it.
They're spawning new clients all the time.
But then I don't have spaces.
I don't have this.
I don't have that.
So nothing of that is complete.
And yeah, varying quality and features, right.
And to get rid of that, or at least a little bit,
we decided to do documentation right from the start.
Right there, we have a nice getting started guide online that for new users,
you just get on there, get a nice guide, how to get onto the metrics.
Because of those, well, problems with the clients,
we still recommend using element just because it is the future complete one,
basically, at the moment.
Yeah.
So we can, if you don't use element, our guides may not be so useful for you,
because we have exact documentation, how to use that client for our home server.
Yeah, please come join our home server.
We should have put a link on there, but maybe it's Ubuntu.com is our home server.
So, think you can find us.
Yeah.
We have a lot of tips in our documentation, basically.
Well, link is on the side here too.
And, like, do not use encryption in public rooms,
because it just slows stuff down.
For us, it makes no sense.
You enable it there at all.
So, just do it in the public room.
And you get instructions on how to set up your private rooms,
your public rooms, how you do moderation.
Be careful with elements that.
So, all the little things you should, yeah, we remind you of.
Oh, that's all you.
Thanks.
So, the documentation of Ubuntu and most of the canonical open-source project
relies on discourse.
So, it's pretty easy to document, because you create post-using
and mark down on this course.
And that post is then available as a documentation
on the documentation.com.
But it easy is a lot the work we need to do.
And it facilitates contributions, reviews, commands,
and we can iterate over the documentation
and implement good documentation, thanks to it.
So, the previous page was the markdown,
and this is the handout markdown discourse post
into our documentation.
So, most of the discourse, most of the canonical open source
documentation is done on this course and handles that way.
So, the third challenge we faced is operational challenge,
maintaining updating, scaling, sign-ups, the right way.
And for that, canonical uses a tool called Juju.
Juju is an operator framework, so it allows us to create
automation around operation of applications.
Typically, I use the database example.
You want to operate the database, scaling us,
scaling down back up rest or disaster recovery.
The knowledge is spreading to people, usually pass strips.
Juju allows you to cut that in Python and do the automation.
And we did that work for sign-ups.
And that's how sign-ups is run into,
for the Ubuntu community.
So, Juju is an orchestration engine.
That allows you to create software operators,
and it allows the integration and the flow
of the lifecycle of your application.
At scale on any infrastructure,
Kate's machine cloud on premises.
And Juju relies on charms.
So, charm is an operator for an application.
And it has all the business logic to operate properly
the application, and follow the lifecycle
back up rest or scale up scale down,
and automate all that for the operator.
So, simple common lines, perhaps still need to be improved.
But if you start from Kate's cluster,
with NGNX ingress integrator,
an NGNX ingress deployed on it,
you can use Juju to add your kid's cluster to Juju.
It's a bootstrap controller.
The controller allows you to manage the terms on your cloud.
And then Kate's a model.
So, a model is where you will put your terms,
and assemble them together
in order for them to communicate and configure properly.
Once you've done that, you can deploy your applications.
So, you deploy the database with three units.
So, you have redundancy.
And you can deploy sign-ups with three units,
also, or five, or ten, or whatever,
depends on your resources and your need.
So, the way the operator works,
it will spawn a main process,
generic workers, and the Federation sender worker.
You can also deploy, enable,
be only, and such kind of features.
Then you need to have your ingress.
So, you deploy an ingress indicator.
It's a term that will just communicate with
Kate's to manage the ingress.
You deploy a Lego provider,
that allows you to get a TLS certificate.
To let them keep and using your DNS service.
Could be any DNS service.
They go as a lot of interactions.
And then you deploy reddish for all your workers
to communicate with each other.
And this can be also scheduled for three units,
or five units or whatever.
And you have a bunch of conflict to do.
You need to configure your sign-ups,
telling it what's your own server.
Few options, the base URL.
You need to configure your ingress.
Also, to specify the ingress class,
the URL for the TLS certificate.
And you need to create some secrets,
which was, and that that secrets mechanism.
So you create your secrets to pass the token
for your DNS provider,
so that the DNS validation is done automatically.
There is a granting, you need to load the secret
to be read by the Lego provider.
You configure the Lego provider,
which is your address in AL and the plugin you want.
I'm using of each.
And then you pass the secret ID,
so that Lego gets the access to the secret.
And final step is,
allowing all these components to communicate together.
So you integrate sign-up with Sposkers,
but we'll happen behind the terms,
we'll create a user, create a schema,
create the password, send it to sign-ups
without you manually doing it.
So sign-ups will reconfigure.
Similarly, sign-ups to reddish.
And then the last one is sign-ups to the denress.
And once you've done that,
your application is available,
the sign-ups is running.
So the way we manage and we ease the operation of metric
from our side.
So a few more features that we support.
We have an A3 integrator,
where you can pass some buckets,
so that you can do that.
So we have an A3 integrator,
where you can pass some buckets,
credentials,
and then relate your S3 integrator to sign-ups
for media to centralize the media.
Also, backups.
We also have backup and restore actions on the terms,
so that you can backup, schedule backups,
and get your private key backed up,
and your media backed up,
and the backup is encrypted,
also using an encryption key.
So they are this safe.
SMTP integrator to get emails,
so the integrator allows you to specify credentials
for your email account,
that your SMTP relay that sign-ups
will use to send emails.
We also have mailbox feature,
that allows you to deploy mailbox,
and all the authentication application
to authentication configuration for metrics
will be done automatically by the term.
So it requires some configuration on theumserver
and a mailbox.
This is done automatically,
and we've worked also on the IFC Bridge,
as mentioned earlier,
in order to look ISE users to get into metrics
or vice versa.
Huge challenges.
We get issues with the non-HAS aspect of sign-ups.
Once one worker is down,
one ends of your events and messages will be down,
and your user will fill it.
Big rooms is a challenge.
It can kill your own server.
It ups all of your resources.
We've got a few more times
due to people joining the metrics,
the metrics.room.
You can forbid some complex room.
There is an option in the term that tells you
if the room is to complex,
you won't be able to join it.
It's a challenge to know how
to operate properly matrix.
The community helps a lot,
but it's a lot of learning,
effort, and back-end force,
and trying to get it,
and fine training for the caching.
Canonical is a corporate proxy.
Federation and corporate co-opsie,
the proxy don't come along together.
You can be federated with any servers.
The proxy doesn't allow you
or doing or incoming things.
We had to find a bit,
basically have a dedicated proxy for sign-ups.
And then we also have as a community
and one Canonical employee lives.
Some challenges in managing the room
and means when they live.
Only the admin himself can demo itself.
So if the user is not doing that,
you need to use some API impersonation
and demo them.
If it's the last admin of the room,
we developed some scripts
and I had some challenges here
in order to manage properly,
levers and admin in the rooms.
That was the last slide for that.
In summary, we are slowly moving to matrix.
The entire Ubuntu community is slowly moving to matrix.
Many teams have already moved or are planning to move.
We just heard that something
sneakily moved a few days ago without even informing us,
which is great, which is how it should be.
But some critical IRC channels might be bridged initially,
partly to ensure that there is still something
to talk to other people with
when we have one of the robustness issues with our home server.
If you go through our Ubuntu community space,
it already has so many communities here.
Ubuntu chilling, Ubuntu budget,
Ubuntu Mate, most flavors,
kernel discussion,
a support room with almost 2,000 members,
a very, very active support room.
However, many challenges remain.
We need to fix these challenges in order to
have this as a reliable tool
that our community can depend on.
And then maybe in the future,
reliable tool that canonical itself as a company can depend on.
Because most of us here are volunteers.
The Ubuntu's move to matrix is a volunteer
offered of the Ubuntu project,
not of canonical itself.
Canonical itself is still using
matter most until they feel themselves
that matrix is stable enough in order to actually move.
There are still some open challenges.
Server stability is a big one that we're also working
a lot on Gregory,
is working a lot on that he is part of the canonical team
that does the operations for our synapse.
And then you have less helpful people
in our chat say,
why do you join the matrix.org room?
Everybody knows not to join that room
because that always brings down your home server.
And yes, but should it?
Should we really depend on our users
knowing that they shouldn't join certain rooms?
Because then we did those our own home server.
This isn't how it should work.
We have the issue that element is currently in maintenance mode.
We do recommend element to all our users
because we have noticed that that's the least painful way
to use matrix.
But element itself is in matrix mode with a whole bunch of bugs
that do not get fixed.
Element X is being developed,
but it isn't actually ready for us.
We need single synone.
We need support for threats.
We need an actual stable long-term support or release.
And then finally, moderation tools is incredibly limited.
Even though CME is helping us a lot,
CME is basically the reason why we are still on matrix.
They are basically unfounded or underfunded.
CME has no funding throughout near itself.
It's funded by an outlet,
which is losing its funding this year.
Thanks to stupid decisions by the European Commission.
That's the result.
Well, the poll results.
Well, it's obviously a good thing that we are not killing our C
because there are still five people using it.
But more than 85% are already on matrix.
So I think that it was a good decision to slowly start
moving towards that.
All right.
Thank you.
And I'm not sure if you have time for questions.