We are going to present auto merge plus key hive, the design of a view.
Thanks for your patience everyone.
We are going to talk about auto machine key hive and local first infrastructure.
I am Alex.
I am Brooke.
We are going to talk about auto merge and key hive.
Thank you everyone.
I am going to start by talking about the problem that we think local first infrastructure
can solve.
I am going to do that by talking a little bit about a little story I would like to tell
which is how about five years ago I wrote this application which is a little
tiny little Android app for tracking work.
It is basically a nice front end on a spreadsheet because using a spreadsheet in the gym is
pretty hard work.
I was very pleased with it and then a friend of mine a week later said,
could I have that and I would like to use it to sync with my laptop.
I thought about why we need to do to allow him to be able to sync with his laptop.
He would have to write a server-side component in some language.
I would have to.
It is back.
It is eventually there.
I would have to write some sync code.
The most important thing is I would have to run a server somewhere.
He would have to run a server somewhere.
I think this particular story is a very small piece of software that I wrote almost as a
throw away.
It is not a very important thing.
I think the fact that I would have to host a server somewhere in order to do this represents
like a loss or a difficulty in it.
You want me to speak louder?
Yeah.
All right.
I represent a lot of something that I think is really important in open source software.
The most important thing to me about open source software is the autonomy that it gives me.
If I don't like a change that someone makes to a piece of software,
I can make a change to it or someone makes, I can refuse that change.
I have a lot more freedom than with the closed source software.
As soon as you have a server-side component running,
whoever is running that server is now controlling something about your experiences.
Now, I am able to force choices on you that you don't like.
I think the normal answer that we think of in open source to this problem is say,
well, you should run every part of the stack yourself.
I think that works for some things.
It works for big, important things.
But for these little things, this little app that I wrote from a workout tracker.
I am not going to do that.
I am not going to run a server for every little piece of software on the right.
I think that the analogy to infrastructure here is a little bit like electricity.
Factories used to build their own power stations.
And that is great for factories and many still do.
But I don't want to build a power station to use my Hoover.
I want to have a piece of infrastructure that I can plug into.
And this is the problem here.
Self-hosting is work.
It means that most people don't self-host.
Even if the whole stack of a piece of software is open source,
we don't actually get the autonomy that we want it.
So we are in this room.
Obviously, local first is the answer.
This is what I am going to say.
You expect.
And there is a lot of appealing things about local first there.
I know servers needed.
That is the principle behind local first is the local version of the data.
It is the thing that is primary.
But servers are actually really useful.
Even if you are in a local first peer-to-peer world,
servers are always on.
If you don't have a server that is always on,
if you are just peer-to-peer between our laptops,
we have to make sure that whoever has the data has their machine on.
Also, modern networks are kind of impenetrable for peer-to-peer.
Use cases a lot of the time.
And having a well-connected, well-known location where something is always on,
makes connectivity much easier.
And it is easier to do back up and redundancy on a server.
So servers are really useful.
I don't think we actually get much benefit from saying we shouldn't use them at all.
Instead, I think we want a couple of design principles that make it possible to work with servers without sacrificing an autonomy.
First of, you should always be able to use some server.
It doesn't matter which server you should be able to stand one up.
You should be able to, any kind of server you can point out,
it should be pretty fungable with a placeable thing.
Secondly, you should never be required to use any particular server.
Like, if you're using your things server finance, you'll be able to use mine.
We should still be able to collaborate without having been forced to use the same ones.
And a server should never be able to do anything more dangerous than just not serve data.
And so those kind of principles are behind how automates works.
So I'm going to just briefly introduce automates.
This is a very high level over the architecture rather than a workflow of how to use automates.
This is a very brief intro, but automates is a JavaScript library.
Well, it's also a couple of other platforms, but this is the JavaScript version.
And it's centered around documents.
So you create an automates document.
The document appears just like a JavaScript object.
Except that when you want to make changes to it, you call this function to make changes to it.
And you can see here we've given a callback where we modify some attributes of this document.
And then you can merge that document with other versions of the same document.
And conflicts are resolved automatically.
I'm not claiming that this is magical.
It's similar to the way that JavaScript works in a lot of ways.
It's all dubbed based.
Like last break wins based conflict resolution.
But we hold on to the history.
So if you want to if you want to go and look at how that conflict resolve change it, you can do that.
Great.
How do we actually do that?
That's the interesting part I want to talk about here with respect to the infrastructure arguments.
Is that we already have a local first application where servers are pretty
Fundable, which allows you to work with local changes.
And that's Git.
It gets really, really good of this stuff.
And one way, it's also really, really bad of this stuff.
Everyone hates Git for good reasons.
But the core model is powerful.
And one way of thinking about what to imagine is it's like Git for JSON.
And what it means by that is the under the hood.
An automated document is a commit.
Dag like this.
Where instead of in Git, where the contents of each commit would be the changed files.
Well, actually, you can tie a snapshot of the file system.
But in auto merge, the contents of each commit are changes to an auto to a JSON document.
JSON like, there's some extra features over JSON, but that's not too important for now.
And this has some nice properties.
Most importantly, the server just has to ship commits around.
It doesn't have any special logic.
It doesn't.
You know, you actually have to care about the contents of those commits.
So, see later, that means we can actually encrypt the contents of the commits.
And the server doesn't need to know anything different.
We can examine the history.
So, we never throw away any of these commits.
That means that you can always, someone, I can always say,
Oh, no, at this point in the commit history I was doing eggs.
And I can go back and look at it.
I can compare the difference between two points of history.
So, I can dip, dip the document just like you can with Git.
And it's hashling, which has some really useful properties when we're talking about infrastructure.
Which is, one, if you and I both have the same,
I've derived the same hash for some commit.
We know that we're looking at the same thing.
Right? Like, we don't have to trust anyone in between,
correctly, shipped, day around.
And we, correctly, doesn't just mean that they weren't malicious.
It means that they weren't buggy.
And that's really useful.
Because a lot of, we're talking about concurrent changes.
A lot of the educators are pretty fedly.
Knowing that we both have the exact same data.
If we know that and we've seen something different,
we know that we have a view, probably,
and the implementation is correct producing the wrong view,
but we have the same data.
And it also means that we can sign over those commits.
Which is really useful.
I want to, I want to talk about one thing here,
which is that we don't throw away any data.
Right? So we're collecting every keystroke you make.
And there's been a tool we can various tools previously already about,
compress about, compacting or throwing away data.
First, this is a delayed virtual choice, not to throw away data.
It's, it's really useful because it means you can always look at what was different.
And when you're talking about local first software and concurrent changes,
being able to see what happened and what changed,
so that you can disagree and change your mind,
is really important and liberating.
But it also has a lot of overhead.
So in order to make this possible, we take this commit law on the right.
And we encode it as a column of compressed format.
So we get, on the, in the commit format,
you get overhead of, you know, one commit hash per keystroke.
Yeah, there's quite a lot of overhead there.
There are actually a couple of hashes because you have to reference the parent commit as well.
But once we've done this column of compression,
we end up with about, a kind of a one bite overhead,
a typical, typical payload.
So you end up, as long as you're being smart about how you're compressing the stuff,
you don't worry too much about the history size.
And this is like an analogy to, to get as well.
Get has, no one really worries too much about the fact that get doesn't throw away all of its,
the history, or track everything.
Like that's a useful feature of it.
And there's a lot of stuff going on under the hood to make that possible.
But I, I think it's a feature, and it's, when you talk,
it's a tradeoff that's very important to think hard about.
And we think that the right side of that tradeoff for the kinds of things that we're working on, is to hold all of history.
And what about complex?
Well, there, there was all of the client side in the sense that,
we'll choose, for some, if there's two conflicting rights,
we'll choose one such that everyone chooses the same one,
who has the same data.
But we also give you all the history in a way to resolve them differently, if you want.
Yeah, other dubbed by default, and history retained.
The principal here is service of dumb pipes.
They're not doing anything except shipping around commits.
They can't interpret your data, and they can't.
And she's read it, it's just forward things.
In practice, this works pretty well.
We have a lot of, we use this a lot ourselves,
and we have a lot of, maybe, strange qualities.
Like, for example, we have, in our web applications, we often have a
Sync server, which is actually a service worker, talking to another Sync server.
Just because it's nice to have a homogeneous architecture, and it's fine.
The main place where this becomes a problem in practice,
is when you want to authenticate and authorize access to your documents.
And this sort of becomes a point when most people are using,
or to merge it, end up having to have a central central Sync server,
that authenticates the reader rights.
And so, that's why we've been working on QI, which I'm getting a hand over to,
but great things.
All right, QI, not the actual logo, but we thought we'd modify the auto merge
logo because I want to logo too.
So, the question is, what would signal for documents look like in this local first world?
We need group and to end encryption and post-confirm my security.
So, we have the security level where if somebody breaks in and you don't detect it,
we still kick them out automatically.
We need mutation control.
We need to protect and prevent people that aren't authorized from editing your documents,
obviously.
And we need peering rules.
We don't want people to be able to toss you or pull down copies of your data,
even if it's encrypted.
And I'll talk a little bit more about that on the next slide,
but that's actually a very important property.
So, all of these things actually end up being a really nice fit from,
for a local first.
We've been pulling ideas and techniques from different fields.
But as Alex just illustrated, this is very helpful if we want to have interchangeable sync servers.
So, why prevent peering?
Well, we need defense and depth.
So, on every node, because, you know, the money you have locally is the same as the thing running on the server.
We have our data encrypted.
You know, there's a little envelope there.
And if you leak the keys, so by accident,
you post the key for the file to get hug or on blue sky or wherever.
Well, we're going to not appear with them because they're not in our,
they're not authorized to actually ask for those bytes.
If somebody breaks into the server and they don't have the key,
you're still okay because the bytes are encrypted unless somebody looks the key.
And so, this means that you can still access your own data,
but it radically lowers the bar to running one of these servers.
You can run it poorly.
You can run it.
You know, just been one up temporarily.
You can switch between servers and all of that will just work.
And you don't have to worry about these really tricky things like how do I seek your access to my data.
It's just built in.
But access control in this world happens at right time.
We're doing encryption.
This sets a ceiling on how secure your document can be.
The choices that we make mean that we're going to impose a security model on your application.
So, this is the trade-off.
When you think about these things as infrastructure,
is any choices that we make are going to affect all of your applications.
So, just a quick rundown comparison in a traditional system.
So, if you think, say, oh, off.
The servers, the thing fundamentally control in this model, you are in control.
In a central server, you know exactly who has access to do what right now.
But because local first, you can have people being offline.
Well, that might, you might have an out-of-date view and reconcile that later.
Server can block access to anyone.
In key hive, the off travels around with the document no matter where it lives.
And you don't need a live connection at all.
We can do over Bluetooth and this continues to work.
So, fundamentally, you become the authority, you can switch servers,
and they are completely interchangeable infrastructure.
One of the components that we use to do this is called a capabilities model.
This is a nice primitive for building up other things that people are more familiar with.
So, the way this fundamentally works is the document itself,
and all the users in the system have ideas of a public key.
And we call this a delegation.
And we are going to write in operations that say things like Bob can edit the documents.
And sends that to, sends that between different users.
And as we, you know, the document delegates this to Alice and Alice delegates to Bob,
we can follow this chain back through and know without having to check any particular place.
We all have a copy of all of these operations, all these certificates.
We know exactly who's able to do what on this document.
And then we take all of this and we sign over it, cryptographically.
Everybody has their own keys.
They can sign it just like how you sign your git commits.
I hope. And that makes it verifiable everywhere.
Out of this, you can build lots of different things.
So, you can build your role-based access control.
You can say, you know, Alice is a member of in-consuitions.
So, he's allowed to write to these documents.
These documents are in this directory.
If you have access to this directory, you've access to all these other files.
And you can also build other systems.
People are, you know, some people really like attribute-based access control
and these other systems.
You can build those all up out of these simpler primitives.
There is, because the know-sensual off, we have out-of-order off.
So, if you do revocation, this means that we have to think both this on two timelines.
We have the original set of delegations and who we're bringing into the document.
And if we kick somebody out, we're basically reaching back in time
and changing what they're able to, what they have access to.
This means that we have to take the edits that they made on the document.
So, let's say they're sort of defacing the document,
or writing all kinds of horrible things into it.
We can go back in time and remove those edits after the fact.
This looks basically the same as they're bringing somebody in.
It's, again, an operation that we sign over.
So, there's no system-wide, super admin-baked-in.
This is just, whoever creates the document gets access to it automatically,
and by defaults, groups can have admins.
If you want, they don't have to automatically.
These are a concepts that are accessible in the system, but not mandatory.
And authority always starts with a document creator and flows out from them.
So, the grand vision.
This is all infrastructure.
Local first, CRDT is all of this, sync service, et cetera.
There's things that we all benefit from.
We all depend on.
No one can take them away.
And they're going to last beyond us or a single project.
So, our users don't lose all of their stuff.
It's not tied to any particular app, any company,
any one country, or anyone host, no matter how enthusiastic that host is,
in their self-hosting.
It should be invisible as TCP-IP.
And what we hope, isn't the future of this sort of ship directly in your OS,
as just another system service.
So, where are things at today?
Today, auto merge.
You should use it today.
It's very actively maintained.
It's written in Rust.
It runs everywhere and has bindings in J.S.
Swift, Go, and there's Python in alpha.
And K-Hive available soon.
TM, the code is open source.
You can look at it.
But it's security software.
So, let us do some audits first.
Thank you.
Very much both of you.
We are a bit in a difficult situation with time.
We have one question online.
Jonas asks, does auto merge keep the whole data set in RAM?
If yes, how can I keep the RAM usage to load for big data sets?
And have you considered a database mode where bits of data are loaded into RAM on demand?
Yes, it does keep the data set in RAM.
That column in our compression is pretty effective.
It took a lot of work to actually be able to use the column in the compression at runtime,
as well as just on disk.
But RAM usage is not since that changed since we were able to do that.
Too much of an issue.
But we are also thinking about being able to partially load bits of the document.
It's a bit further down the roadmap.
Unless it's important to you and you have enough money to sponsor us to do it.
But we will get there eventually.
And to add on to that, a lot of people want to do things with multiple documents at once.
And that we do load in just with your request.
Another question?
One.
Two.
No question, perfect.
Thank you very much.
Thank you again.
Group.
And that next one, motommage.
Yes.
Thank you.
Good job.