So I'm a tough way between academia and industry, so tough as a time, I'm in academia,
so the talk will be also tough way.
So the general idea is that there is like some lost chapter of computer science data
synchronization in our apps.
We use like typical commodity data structures like maps, vectors, whatever,
and there is no way the rest of the generic technology to sync those data structures.
Like once you have two devices, we obviously immediately get the need to sync data.
For some strange reason, in a computer science textbook there is no accepted way to do it right.
For some reason. Like it was discussed in ages, about the money when the other way.
So this story was unfolding at a very slow rate.
I started participating in 2008, so it is much better now.
But the general idea is like, we currently can have like,
we have like wool outlet with IP packets, right?
And we want to have wool outlet with data structures.
Like I changed some stuff here in my app, and it goes by the bias,
and where I don't think about it, it does it in background,
and it gets distributed to all the replicas, if somebody clicks something somewhere.
It gets back to me.
So I just connected the network.
This is the idea.
Very well defined problem, very generic, everybody needs it.
There is nothing.
So general approach, like we use the database.
My app is saving to the database, your app is loading from the database.
What if we deal with the loan running app?
You have two positive database, which is not good.
And general speaking, if we lose connection to the database,
that's it, we are screwed, because we have no local copy.
And another way to look at this problem, like, it is git for data.
We have the data we need, we have it like locally, we autonomous,
but we also think in real time, like the vision for it.
In the application to database theory, like,
some ideas have been said like in 1884,
and it was obviously a summary of discussions of the 70s,
so the general idea was like in the air, but the money went the other way.
Because obviously, if you bring out the data to one big central server,
then you can own the server, you can make a lot of money.
So, I mean, the logic is just straightforward here, like this.
It is about technology, if it makes me lose money, right?
So, but if you look from the other side, like from the client side,
then it is exactly the opposite.
Then, why can't we use air sink, for example, for such thing?
Why can't there seem to be our lip sink for such a kind of general things?
Obviously, it works well, it synchronizes binary blobs,
without understanding what's inside, so it is generic enough.
But, there's sink on your works one way.
It can update replicas, right? It can update mirrors,
because if you have two files, same name, different date, different size,
watch other sink do.
I mean, is it like, this is probably the new version,
and maybe this kind of like concurrent changes?
It is unclear, from the context.
And also, the sink is enabled to seem a file, which has some internal structure.
So, obviously, to understand which is first, which is second,
or maybe the concurrent, we need to track changes.
Let us revision control, right?
Obviously, git has history, branches.
It knows which version was derived from, which has a version,
and using that metadata, it can more or less reasonably merge the changes,
most of the time.
Well, with the exception, for the cases, one, it cannot merge them,
or it will merge them, but that will be a bit incorrect,
because it has, like, very, very understanding of the semantic of the data.
Yeah, so, we cannot use git for this purpose, for this simple reason.
Git has not deterministic notion.
Quite often, git has to appeal to the developer,
to manually resolve the merge contract, right?
It's not deterministic.
We cannot have an application, which asks the user,
actually, Google Docs was doing that in its early days.
One, they had concurrent edicts in the same paragraph.
They would show a pop-up like, which version would you prefer?
If you speak, for example, of Apple, their office program,
it was even a funny story, because it was saying,
like, which version do you want to keep?
And if you click the wrong one, a week of work is disappeared forever.
That actually happened to me.
So, what is, they match better with this?
So, then, a nice new technology,
that is about 2,000 to 11, 30 is.
Complex theory, pick it at data types.
They call those commodity data structures,
which, like, everybody is using to build their apps.
And make mergeable versions of them.
We use some metadata.
We annotate each piece of data, like,
which was added by Joel, like, yesterday.
This was added by Jane, two days ago.
So, we, more or less, understand which piece of data came from,
which side, and which is the recent area,
which is the old data, and so on.
A algebraically, it is very, very nice and smooth and clean.
And it always, like, merge very reasonably without conflicts and so on.
I will explain the quotes.
So, basically, who invented this series T?
The series T wasn't invented, it was coined.
By the time the term was invented,
the technology was already being used.
So, and the very term series here,
I mentioned it as conflict theory replicated data types.
Actually, there are also commutative replicated data types.
Conversion, replicated data types.
And many other probably, see,
are these, they mean more or less the same,
but they're, like, equivalent in some sense.
So, basically, it is the last library of replicated data types,
of why those kinds.
So, that was, like, a happy day when,
that in reality, in publishes,
the paper of the library of replicated data types,
have, of which the boroughs for some of the deals.
But finally, we have, like, that for library,
which we can use to build apps.
So, finally, we can solve that problem,
it is not too engineering, right?
Sort of yes.
But the problem is,
CRBT, resource, all these merge conflicts
and are other issues of dealing with concurrent changes to the data.
It is solved by adding metadata.
So, for each piece of data is annotated
who added it, when added it.
And that, basically, can ruin everything for the simple reason
that we can have more metadata on data by many times.
So, we can only compare with technology to Cassandra.
I mean, I'm on, like, massively used technologies,
which are, like, industrial, and massively used.
The only thing we can compare,
is Cassandra. Cassandra is last right-piece,
but it is using metadata.
I understand, Cassandra, right?
It's a cloud database, which Apple is using,
in farms, like, of 10,000 servers,
because, like, all the iPhones are pumping data
into the cloud, and they have to put it somewhere.
So, Cassandra is, like, very decentralized.
And it has the same molest metadata problems.
They kind of cheating.
They're putting, I believe, only one time
step on the tuple.
And in case of CRDT, we have to put, like,
one logical time step on the cell.
And a logical time step is idea for replica and time.
So, basically, two integers.
So, if our cell is one thing integer,
we put two integers of metadata on top.
And also, because, like, most typical piece of data
in real-world applications is a small integer,
and our logical time stamp is typically
larger integer, and user-ready is also larger integer.
So, in the end, it will be, like, 10 times more data,
metadata than data.
Which sort of ruins the day?
Because, like, it works correctly,
as a runa conflict, but you have to,
you need 10 times more time to pump all the data,
than 10 times more servers to store it.
And, no, but it's all correct.
On the contrary, it's synchronizes really good.
So, basically, this was the story, like,
two thousand telets, one, two thousand,
fifteen, fifteen, so it would, like,
top-ish ones agenda.
And, another piece of metadata is version vectors.
In case, I mean, we can also refer to one, like,
most reviews in the astral system, they name it DB,
which has a small, that's a lot of problem,
but they don't have much, that's a problem, because,
because, version vectors is a constructs.
Like, if you don't have, in a linear history,
you can, like, in post-gross, rogue,
DB, whatever.
Any classic database, my SQL,
we can have, like, transaction,
one transaction, two transaction, three,
three, three, four, and so on, linear.
In such a database, because you don't have
central servers, you don't have linear history.
So, we have a vector of versions.
So, we say, like, from Jane,
we've heard everything till version 10,
and from Joe, we've heard everything till version 12,
and so on, and in the worst case, for each user,
we have a number.
So, if we have million users,
then our theoretical descriptor of our version,
is, like, one million cells.
Which is not good.
And that was also a problem, that was,
but, both in many people, for example,
some people from the Portuguese team,
I think, University of Braga, which he had Braga.
They wrote, like,
and stack of articles on optimizing version vectors,
but, like, yes, it was a problem.
Like, the name of DB,
it doesn't have a problem because, for them,
one replica is actually one server.
So, in case you have 10 servers,
you have 10 cells, and a option vector.
And this will be, like,
not service, in total, but, so,
we're just touching a particular piece of data.
So, it wasn't much of a problem, for them.
So, like, we have nice technology,
it's more just swell, it's synchronized as well.
Well, actually,
me and friends might,
a collaborative editor, which was running in production,
into files on 12, using exactly this version team.
But, it wasn't an issue because, for each particular document,
user-side collaborative, collaboratively.
Write a picture not many users.
If you have a hundred users, that's not a tragedy.
Even if you have a thousand users per document,
which is really rare, it is not a tragedy.
So, this is only a problem in some cases,
but when it is a problem, it is a big one.
So, all these metadata problems have been
more or less solved by this day.
Basically, the correct approach was to use,
how to say, to version data in bulk.
Like, you can ship in envelopes,
you can ship in what logistical containers.
So, the idea is to ship data in logistical containers.
Once you version logistical containers,
you need much less of metadata.
But, what I'm describing is,
like, progress of this field of research for,
might be like, 15 years, just like.
I mean, when I describe it as a spectrum,
it is like, what a full, heavy beam,
but it wasn't just me, it was like,
a team in Portugal, a team in France and many other teams.
So, the idea,
we can compare,
with the entire story,
to myself run all the same data basis,
which actually, in nature,
really, version of the data,
because they have,
how to say, it is T files of all data,
and like, less old data and new data and so on.
So, the nature to keep all the new data separate.
So, the trick is used
to then, that part of me,
to data is not a problem as well.
So, August question for people who are bored of,
bored of academic struggles,
I could be in describing.
What can I actually use?
Somebody,
who has an idea,
what can I actually use for sinking my data in my app?
Please raise your hand.
Like, you have clear idea,
you're using some kind of,
one, two, three, four, five,
six, seven.
So, basically, this is like a relevant audience.
Good.
So, currently, we have like,
the local first world,
which is mostly into JavaScript, mostly.
Some have, are asymptim limitations,
but they are not working that,
quite easily.
So, mostly, from 2014,
I think that,
a lot of first story continues,
like for 10 years,
and to this day, most of it is like JavaScript.
Then, there are solutions for sinking data,
basically.
Like, if you have Postgres on the server,
you have,
SQLite on the client,
and you want to synchronize it.
And some of this actually,
funded companies.
And,
the thing I like to talk about is,
LiPurDiX,
it is like,
my fin.
So, a LiPurDiX,
like, replicated data exchange format.
That is, like, JSON-ish format,
which synchronizes really well.
So, why do I want to talk about,
because synchronize the bull JSON,
it is like, for 1214,
and it is not used really much,
because,
but what is the good thing about,
again,
a problem is about metadata.
What is good thing about JSON JSON,
is easy to read, easy to edit, right?
If we put our data in JSON,
we can always read it,
we can always debug it,
we open console in the browser,
we see JSON,
we understand what's going on, right?
So, if you add that,
if you add it to JSON,
it becomes,
like, really complicated,
what's going on,
impossible to edit by hand,
because if you mess up the metadata,
that's it.
So,
ah,
good part.
Ah,
Erdix,
replicate the data exchange format.
So, it is like JSON-like format,
which has metadata,
but the model is built in a way
that you may export it in a platform
and then import it back
into metadata reform.
So, basically,
as a user,
you'll never work with metadata,
annotated data.
Also,
it is not exactly JSON,
because it is sort of optimized
to make it nice, clean of jubric.
So, for example,
there are, like,
primitive types,
first float into jubric
and string term.
And,
the rapplex types,
collections,
basically, tuple.
Ah,
linear vector,
you will reset,
and multiplex is,
like,
version vector,
or counter,
collaborative counter.
So,
ah,
the idea is,
that you can nest it all,
arbitrarily,
for example,
this is set.
In the JavaScript,
that will be a legal,
to,
for example,
one entry,
or,
or entry without a key,
and here set is a set.
So, basically,
you can put whatever you want into it,
and,
this key will appear is actually a table,
and this part,
and this pair is also a table.
So, basically,
it is, like,
algebraically,
optimized JSON,
where everything commutes,
and everything can be nested into,
into everything.
So,
basically,
how I merge it,
you don't produce
an illegal construction.
How,
how,
that's how,
you,
what,
I've already read compiler,
hit get it,
DB. It is prototype quality, but the idea is once you have all this constructs nice
clean algebraic, you can just add this more shakerators to record it, but it becomes
not jable-syncable serdity database. Like this is a very primitive hack, and on
it works, because once again, all the more shakerators for all these types, like single
pass, very much like more shakerators. And then there is this library which I write in
very particular dialect of C, which is called ABC, algebraic, break-lain C. It is a very
algebraic dialect. People who read it, they always mention lists, whatever, and so, the good
part is it is like systems level stuff. So, lipordrics, lip, lip, lip, lip, lip, lip, lip. Just to
sync data from any up to any connection to any server, that is the idea. I might
make it all nice, but because it is in C, I challenge obviously, and then in any other language.
So, in this world of data synchronicnowages, we now have a different problem. We have different
standards for those data synchroniclets, very much like in the real world, but otherwise life
is beautiful, and please questions. Thank you.
You have a first question, right? No, no, no, no. Any question?
Yeah, of course itself, when you sit there, get as something that does sync, and that you can
use, and you did it. No, theoretically, I might put my stuff in text files, and my pricing
can get with this. Some people are actually doing that, for example. Can you repeat the
question, please? Can you repeat the question? Why don't I use git? Why do I even consider?
Why are you listening? Because some people are actually doing it. For example, they put
issues and the commands and stuff into their rep of the main project and they have some scripts,
which like I forgot was the name of the project, which is doing that, but they are actually using
git to sync data. They didn't, they just not correct, but it works to some degree.
And the sync is the same. So, basically, I'm just going like less semantics, more semantics.
Ersink is just, GIT is basically also blobs, but it tracks history, the blobs, and it can
merge changes to the blobs, it's out on the standard semantics. And see, already, everything is
that, everything is that, and also it has particular semantics for the extractor. So, it can merge
it also set, and vector is a vector. So, primitive to sophisticated.
You brought up the merge problem earlier, but I don't know if I missed any proposed solution
on semantic merge conflicts. So, even without the X or what not, was your proposal for,
I changed the word, the cat, you changed the word, the dog, and they come in together.
Obviously, a little bit. I mean, it was like a joke for like 20 years and this is the main,
like we need artificial intelligence to solve semantics merge problems, because there is a
semantic. And now we have artificial intelligence, so it is no longer a joke.
Basically, they will produce a mess, but they will produce a deterministic mess, so they will
all produce the same mess. So, they want to resolve that mess, they will also resolve it for the others.
At least algorithmic level, like the maximum possible thing.
Did you have a solution or is it at a level above for compaction or garbage collection of the
version vectors? Yes, yes, yes, yes. Basically, I will send that a basis, they do garbage collection
nature really. Like the ways of work, they always do gc, like with every compaction, they basically
compact. So, more or less the same thing. Some amount of metadata still stays. It is inevitable.
Unfortunately, but in the general scheme of things, yes, there are compaction and I am
like inevitably all of some blades of it. Thank you. Oh, finally.
So, how do you actually handle, is it ends up being last write rins for actual pieces of data,
like let's say between if it was named Alice Bob, like there is a conflict in the actual values?
As a result, basically, in many cases, it is last written depends on the data structure.
It is of an array. For example, the mark with somehow it didn't uniquely identify the insertion
place. So, if somebody, instead of data into the place, we don't care about any concurrent insertion
solutions or deletions, we can always find the place in the new version. That is the matrix of array.
So, it's, they have more or less last written ones, semantics, but they are a particular key.
Counter, they have a different semantics. Like in version version, we had like Bob 5 and the other version,
we have Bob 7, we know like this one is later version. So, Bob 7, that means like Bob contributed
7 units to this particular counter. And values from Alice, they will go like in a different cell.
So, basically, concurrent counters, they split it by contributions by the source.
Not the mess up the things, because they track who contributed how much. And so on, semantics differs
by the data type. Yep. Sorry, I have another question. Should I understand RDX as a being a CRD
team? So, could I say, when I choose between oddermarge, YDS, that is just, and then this, this,
this, this, this, this, this, this, this, this, this, this, this, this, this, this, this, this, this,
this, this, this, this, this, you see? Clip, yeah, it is working in Cambridge. Yeah.
Ah, the oddermarge. So, is there also something about RDX that makes it different or
or higher up the stack, or larger than YJS, for instance,
is something RDS is more or less like a small business
doing the discrete data synchronization
for collaborative online applications,
which was my state of interest in 2012 to 2014.
And now I'm doing systems level of top.
So YJS, yes, let us see at another solution.
Speak of after March, then basically
they're trying to use unchanged JSON.
And I'm basically saying like there's no way
to make JSON mergeable, let's make a nicer JSON
like format, which is like sufficiently how to say it.
All jubric to merge it nicely.
So more likely, quiz.
So this is different solutions, but very more or less
like overlapped, heavily.
But very, very powerful, maybe.
Thank you.
One less question?
Is this like a mathematically proved
that this problem is solved?
Is that the bottom part?
No way.
Why do we focus?
Yes, yes.
Why do we still have things like merge conflicts?
Basically, it was public.
It was a chain of articles starting to file on six
I believe.
And the concept here was coined in 2011.
And it was always, yes, we had a couple of proof.
And it was only one of us, we had a couple of proof.
So for the last 15 years, the problem was practical
applicability because of the method of data.
And in theory, I would have been more just nicely
over a few courses on the rainbow.