Okay, thank you, Ricardo, for warming up the stage.
So, okay, this talk is about a stalwart and whether it's open source software, it's ready
to run something like Gmail.
So for those that, you know, no stalwart, it's an email and collaboration server that supports
every single JMA protocol, they're except perhaps, yeah, a few things in that weapon,
oh, baby, whatever, it's pronounced, but all the core ones, mail and so on are supported.
And also support legacy protocols such as Carda, weather and double, okay, so the, so let's
see what based on, of course, we don't know exactly what JMA, what Gmail is using to run
out their scale, but we know a few things from research paper, research paper, so on.
For example, we kind of know that for a span classification, they use vectorization,
the molecule red pack, they use deep learning and ensemble models, then we know, do something
called big-type or spanner for metadata, then colossus for the blobs and search, we don't
know exactly how it's called, but yeah, we know it works really well.
So how can we replicate this with open source?
And here, let's talk a bit about the traditional email stack before store work.
So the program with the traditional IMA servers such as DupCot for example, has a rely
on mail there, mail is great for single server setups, but the program is that it does
not scale.
Some large mailbox provider can make it scale, but that leads to silos, for example, some
solutions that use like the directors, where they keep the mailboxes in a compress, in a
blob store, and when the user connects, the director, the one server becomes the owner of
that mailbox, downloads it, uncompresses, and then all requests go to that node, and that
node sends the update to the history store, which sounds very fragile.
So then we have the traditional MTA, such as post fix, of course you can make them scale,
but they have local cues, they're not, so you want certain node crashes, you need to
recover your cue somehow, or the message is stuck there.
Then we have the traditional span filters with the rely on bay base, I'm talking about
the machine learning part, not all the rules and so on.
So the single base has a decent accuracy, but you have to do a lookup per token, so that
translates into one ready query, or whatever you're storing your tokens.
And if you're using something like biograms or OSB, that multiplies by five, for example,
and then if you keep global models and per user models, that multiplies by twice, so it's
thousands of lookups per message, of course they can be cash, but still you have to do
those lookups and then update, in the history of the environment, you have to distribute
the updates.
So still has been built from day one to be distributed, actually the version 0.1 has the
rough consensus protocol built in, so it uses ROX-DB as a backend, but I was not so happy
about having rate replicas, I want to reach node to be like a primary, to take right as
well, that's why for this to be the environment that I decided to switch it to a foundation
DB.
So just like in a Gmail, Stowers separates a different amount of stores, this data store
for stuff that changes often, such as flags, calendar contacts, indexes and so on, blobs
are for static data such as raw messages and ship scripts and files from finals, then
there's a full-text search engine for sending your messages or calendar contacts and so on
and memory store for temporary data.
So in Stowers, all nodes are equal-built, peers, so they're not really replicas, they're
no leaders, no directors, users can connect to any node, no matter what protocol, I'm
up, JMAP, tab, so you can scale or is simply by adding nodes and you can out-to-scale
by running, running, throwing an orchestration platform.
So this is how we would look like the sample design, so you have your client connecting
to any node and then you have a different storage and a coordinator that we're going to discuss
briefly later.
So the data store, that's where the metadata store is recommended for very large environments
is foundation to be, because it's used by Apple, for example, where I cloud mail is used
in foundation to be.
But then of course, we are not running something in Gmail's calendar and you need distribution,
there is some posts around the internet, how can you say, yeah, I don't have to upgrade,
I use postgres instead, same thing, foundation to be great, I use postgres instead, postgres
also a great alternative to use, especially with read replicas.
So if you don't need that scalability, postgres is also fine.
In fact, Thunderbird is going to release a service called Thundermail using Stower and they're
using postgres, so we'll see how well it goes for that.
Then for the Blob store, the one without any staff, although this needs to be really tested
in detail, because it will be slow with high throughput.
But that is the thing that scales the most.
I know that for self-hosted, if you're running, you also use a three compatible API, but
it's important that they're not file, file sheet and dependencies, so they're no side
of like in mail there.
Then if the search, the full-text search, you can use, you can use the built-in search
changing with Foundation TV, or you can use elastic search open search is Java is not an issue
for Rust people, it is a bit sorry for the Java code.
And mainly search also for smaller deployments.
Then for the temporary data, you can use the radius, same cache, this is where the right
limiting another log and temporary information is stored.
Then in Stower, the MTAQs are totally distributed, so if one node crashes, the other one can
pick up the pending task from that Q. So you can keep also, if you have a lot of messages
pending for the delivery, you can start adding nodes to pick up nodes, to pick up
jobs from that Q.
Then for span classification, version 0.15, made a big improvement in performance and also
classifying accuracy, classification accuracy by switching to the follow-the-regularized
leader with optimal updates algorithm.
That is, it's from Google Research, it's used in other words, it's an, for other
words, click prediction, it's an online learning algorithm has also better accuracy that
base, because base has assumed feature in the pendants, so to make it work, you need to use
bi-gram or OSV, so FTRL is an optimizer function for a logistic regression model, so there
are no lookups with this algorithm, with this design, sorry, not only algorithm, because
is in the hash intrigue, the hash intrigue is another Yahoo paper, so this is U.P.R.O.E.
still used nowadays by Yahoo, which is compressing the features of your model into a fixed
size, and in order to reduce collisions, they use it, they change the sign with a second
hashing function, and their paper shows that they can fit in 8 May of its own memory around
2 to the power 20 features, the email for global and per user models for 430,000 users,
or 4 million emails, or 3.5 million emails, so the software by default is as to the power
20, but you can increase that to any number, and if you start having per and with collisions,
you can use another paper that is called the improves on a hashing trick, and uses a
kukuhashing, which is the kukuhashing algorithms, but updated, modifying the hashing trick,
so to raise the collisions.
So then in store where you have another component, which is a cluster coordination, so
what it does is when you, for example, there is a J map out there, it is in one node, and
the per hashing, the J map is written, or the user is connected with the 11th stream to
another node, so those are the things to be distributed, so for that another kind of
update, like failure, and so on are distributed with a coordination mechanism.
If you don't want another dependency, you can use a clips, a CNO, or if not something like
Apache Kafka or Red Panda, or something with a high throughput, then for a cluster
orchestration, the difference is that, so the orchestration is running the multiple instances
of stalwart in a cluster, and then when there is a, if you need to add a node and you
know automatically, the Jogasterity mechanism can do that for you, also when the load is
low, some nodes can be removed from the cluster, so the stalwart can run something like
Kubernetes, Apache Messers, and so on.
So I wanted to go, this separately, the cluster topologies, so let's say you have 1000, 1000
nodes, how you distribute them, you can make all those 1000 nodes, this is just for the
stalwart, not for the other service like Zeta, it's not so on.
So you could have all nodes doing everything, JMA, PC and DPA, IMA, Zav and so on, or if
not you can separate those devices, this, you really don't know the base on the, what
they will be doing by, by a protocol, for example, you could have some nodes for JMA,
but therefore IMA, PC and PC and so on, if not, you can distribute them, you can
know a weighted allocation based on how popular that protocol is and put more nodes on
that, on that sub cluster, say, and then another way to do it is pairing protocols, for
example, similar protocols, SETP, IMA, being one group, then the JMA, and DAB, protocol,
so HTTP, protocol in one side, and so on, and then you could also create a cluster based
on geographic distribution, you can have different stalwart clusters by, by region, accessing
the same stores, so to summarize, so usually you can start with unify and then move based
on on your traffic patterns, so how close are we to run something like JMA with open
source, so how close are all vertices and how close are the individual open source components.
So the closest we have to a JMA scale is Foundation DB, because it's super scalable and it's
used in production by Apple, of course Apple is not JMA scale, but it's also amazing millions
and millions of users, also could be shared that eventually sub clusters of Foundation DB possible.
So this is a, I mean, formal comparison based on public available info, so foundation with a
closer we have to a big table and spanner, then for blobs, safe, also a good alternative,
because of the, excuse myself, for example, by CR and HP and some other companies,
they claim they can scale to petalize to XOS of data, we need to make sure how fast it is only,
if you can handle high throughput, like a lot of rights, when you start receiving a large amount
of emails at the same time, if a safe can handle that low or if that will create a bottleneck.
So yet this is another informal comparison of the, of a safe and, of course, so the only issue we have
is, in my opinion, it's a search store, this is, because Gmail is excellent, a search in a scale,
we have, for example, a melee search which is also a rust, written in rust, but it's a single load
and the distributed one, the distributed version is commercial, and also it is not possible to
map every single jame up or I'm a query to a melee search query, because it's, it's time for
user-facing applications, so some queries won't work as your user's expect, then we have a elastic
search, open search, that the other program solutions, they also work well, but they want
the lever, the scale needed for running something like Gmail, right? So, how can we get there,
how can we get close to Gmail scale? So what we need is a truly distributed search engine,
so one thing, we're exploring is a sixth store which is a rustful text search
library, library that is faster than another library, than TV using rust, so a competition
who has a faster library and they like outperforming sea libraries and stuff like that,
and this one is quite promising, and they have plans to use, from addition to be as a
well, I propose that and they're open to the idea, so we'll see how that goes, because that will be
also, we'll make a search really scalable. So for the spam classifier, we're also, we're almost
there, for performance, the current model used by so always, it's quite similar to what Yahoo uses,
the algorithm FtRL is used by Google, but for something else, for other words,
the problem or spending things to solve is accuracy being as accurate as Google.
For example, while working on the new classifier, I did some tests using redback, which is used
by Gmail, this went off for sure, so it's a vectorization model, taller and too attacks,
and to train a Kaggle dataset for spam filtering to like two hours on an M2 Max and a Macbook,
while the FtRL proxy algorithm did that in seconds, and the accuracy improvement was 0.3%
which for the deep learning mode. So for Google scale, that is millions of emails, but for
the small guy that's not that much, it doesn't justify. So if we want to reach Gmail's
accuracy, we need to use something that understands content, the intent of the email, not just
counting words like linear models, are doing. So the goal is to match email level accuracy and
efficiency. So so we're already supports integrating LLMs. You can connect to Open API, or
30PT, or Cloud, whatever, or run your own model locally. But of course, the slow, it's slow and
expensive. So we need something in the middle between linear models, something like the linear regression
models and LLMs. And one potential alternative, while the bird models that could be combined with
redback, and that would be great to understand. So these models can assist the linear models,
not replace, but as more info, to decide if an email is spam or not.
Then for what is clustering, store work has been tested in the lab, of course, with a setup
with one million users. It was done done on a 10 node cluster, but of course, it's a lab testing.
But so far, all the work in store work was implementing new protocols and new features,
but now store work is feature complete. So from now on, the focus is performance and scalability.
So yeah, so from now on, it's all the fine tuning comes.
So to get to Gmail scale, after version 0.6, 0.16 is released, the idea is to use some
credits, a world by GitHub, to create a very large cluster to test if possible, something like
Gmail. So as use and burn those credits to simulate a Gmail deployment, of course, before
something smaller will be attempted, right? And then go from there and to see how far we can push it.
So the roadmap for store work, as mentioned before, is now store work feature complete.
However, there are a few pending user enhancement, a user request and enhancements.
Versions 0.16 is under development. It will no longer use tunnel for settings.
It will be configured with JMAP with a new future preferences protocol, several settings,
protocol that will be hopefully standardized at some point. So yeah, so the goal is to make store
where a scale was as Gmail is an ambitious goal, but even if that means writing a store work
search engine to rich.go, so that is the plan. So well, this is a current summary,
basically of what we've been talking about. I believe we're like almost there, basically because
of all the great technology of their foundation, the VNSA-1, but there's a lot of work pending, okay?
All right, thank you very much.
Questions?
First of all, you're standing, yeah, sorry, it's all and how much
localization do you recommend to someone for action?
Okay, the question is how large an organization that recommends store work to be used or
well
Oh, if he's taking in production, why is the largest deployment?
The largest I know is a 300,000, but they do mostly outbound. Then there are smaller
requirements of 50,000, for example, in Germany, there's another 30,000 in France, and also the
search engine khagi, I don't know if you know them, like it's a paid search engine, it's also using
a store work, but I don't know exactly the one that are not, some users may be using it for
with more mailboxes, but if they're not sharing that, if they're using the open source version,
we'll never know. Those are the ones I know, yeah.
Can you repeat how large I think?
Oh, how large is a standard my expected to become? I don't know, we need to ask
the standard people, but they're running our beta, and they haven't shared the numbers,
but if my guess is 100, because they're hundreds of thousands of users based on the donations that
receive, I'm sure it will be reached at level, and also because I think there will be
also offering free email, free email accounts, so that will also increment the number.
Oh, because email is evil. Why competing with?
The Gmail also is you don't download your data, they're using it probably to train models,
and so it's not complete because Gmail is also a service, this is a product,
I probably run out your server, so you get to control of your data, so instead of hosting your email,
Gmail, you can host it at your own server, and yeah, and I'll prevent them from using it to train models.
That would be a good reason to release a stalwart to release stalwart.
Yeah, yeah, it is, it is out there, it is on GitHub.
You rest your happy phone now, okay?
I'm sorry, you may provide their season still worked in Europe or in the United Kingdom,
like here for commercial suffocates.
I don't know, he's asking is there any mailbox provider, you know, the kingdom?
Yes, I know one in the Netherlands, a super space, I think it's called, yeah, yeah, yeah, yeah,
it's great. If you rewrite the writing a search, and you can also replace Google,
yeah, that would be the most.
I'm wondering about your benchmark in size, because for me, it's like a bit of an aberration,
I have an integrated protocol, but then everybody's using the segments, so I want that benchmark,
and I would love to be, to see how easy it can be with one of the instances
of multiple users, how easy it is to set up, how easy it is to set up,
how to verify the instances, how easy it is to set individual instances, compared to something very large,
is that correct? Yeah, that is a main, that's what Stau is already doing really well,
which is running like very easy to deploy, you can handle your DNS records,
was so far only the TLS, the Acme records, but versions of 16 will handle your decim,
everything in a related, so it's pretty simple to set up a small instance.
That's one final question, the background, sorry.
We use all work, which is fantastic, the one mission, Stau works, we build by Stau work,
and that's right. Yeah, that's correct, because before it was in the UK,
and until the EUing company is type of company is available, it's currently been migrated to the EU.
Also, it's going to be brought to Europe. Yeah, it's already managed from Europe,
it's the LLCs for investment from the US, only, but it's not, there's no control of the US.
One of the key levels of sovereignty is that it's been built by a European country,
whether European business, located in Europe or so, and why only LLC might get brought to some
country, but in your movement, Europe, that's great. Yeah, yeah, that is a plan, it's already,
there's already a UKLTD managing, but the idea after the Brexit,
there was a problem, the company is slowly winding down, so this is a temporary structure,
until the EU cooperation is available. Yeah. Okay, thank you very much.