Jim, we'll be talking about a programming language perspective on
and replication. It's super high level, super interesting. Let's go.
Let's welcome him.
So then talk about the scientific environment.
I can have a perspective as a post-conceptal tool researcher at the TV.
So I'm the expert in the programming language perspective.
So we're talking about the applied module.
I have these vision data types.
So they have been talking about two parts.
First of all, look at the problem that we've made.
How we attempted to work, try to solve them for the method teams.
And then try to go a bit into my personal framework.
I've been able to test these things very well.
So a lot of the problems, actually, that we have to do.
We can get them to get them strong.
Well, it's right that we can't remember.
We need a distributed system.
A lot of these pre-porting systems.
We've been able to build a built-up team and partition tools.
We're going to take it into a liquid system, right?
But, actually, in reality, we always need more people, right?
We always have to assume that our system remains correct.
If we have something, what does it get?
It's not as much as the method.
What that means is that we have to have a built-up system in the state.
And then we'll be able to build it.
So that's why we have to do this.
But the reality when the development is replicated,
it's not like this.
The consistency is poor.
You get it's right.
The replicated setting is used to be strong.
We practically get it.
It's a contextual tool.
But it is not impossible.
I'm going to go pick over several little problems.
And we'll put that into the system.
So the first thing is right.
Selecting the right to begin the device for your application.
So, if you're building an application, you have to ask yourself,
do we want strong consistency to want synchronize or data?
Do we want that strong advantage of consistency?
We can tolerate a bit of delay in our synthesis.
We're at the gas consumption rate of data.
But eventually, it converts.
Or do we need just causal consistency, which might be different?
Let's be glad that.
Thank you very much.
All right, I'll just recap the slide.
Just for the online audience.
So basically, selecting the right or the T is difficult.
We have to select the right consistency level.
And based on that, we also have to think,
what invariant in our application, we have to guarantee.
What does our application need?
And often this is very specific to what you're going to do.
So one of the things that we've been developing,
and actually this was done by Kevin, who gave a presentation earlier,
is that we let the computer decide.
Basically, you encode, in a formal specification,
in a high-level language, what you need,
what the semantics of your data type have to be,
what invariants you have to uphold,
and we run competitions on this.
So basically, what happens is you give a scalar-like spec.
We let the three wrong on this together with some analysis tools.
It gives you back a lot of bunch of results.
Based on this, we can automatically generate
a scalar code, JavaScript code, other code,
which will allow you to synchronize it.
We have also proof that whatever the computer decides
to generate what data type generates out of this is correct.
So a really cool part of this is that it allows you
to offload a bit about thinking how to design correct
to serialities, data types, or strongly consistent data types,
because not only can verify, make it a serialities.
It also can tell you for this replicated data type,
you cannot have a seriality, because we will never be able
to uphold this variance, and you will have to sync on this point.
So this gives the developer a bit of a flexibility
to be more confident about what they have.
So that's important.
The second part is that in a replicated data type,
you often have all these events happening concurrently, right?
This is why you need serialities to have convergence
or any other mechanism.
But so one of the ways that you want to implement a seriality
is by tracking causality, for example,
tracking what links you have between events,
what operations happened before other events.
And so this actually leads to a problem,
because you have to kind of track this metadata
inside of your replicated state, so either as data,
or often when you are transmitting packages over the network,
you want to track logical timestamps,
about the data or what other data variations
this might grow with the size of the events in your system.
So basically, we have a data management problem, right?
So what we've been doing is looking at ways
that we can have distributed garbage collection
and serialities, that can happen in the background,
and it makes it more efficient.
We move to data when needed.
And also Tomas here has been working
on having Delta vector close, which allow us to have
in efficient representation of called timestamps
which are replicated in a network.
Finally, designing a sensible CRT is hard.
So earlier, you saw that we can let
computer decide how to make CRTs.
But also, we have to kind of think that if you want
to have a general set of CRTs that people can directly use,
we have to think a bit about or semantic
that we give to the end user.
So for given a set, for example, a set,
you cannot replicate directly because if you have received
either remove operations in different orders,
you get different results.
So this makes that you have CRTs, for example,
be add-win CRT or remove CRT or last writer-win CRT's.
They allow you to force in certain ordering
that will make that your system converges, right?
But even if you pick one of these, implementing them
still can lead to strange behavior.
Imagine we have two replicas.
And then when we add an item, and remove it,
and we do exactly the same in an order replica.
Now I say we have add-win semantic system
in a concurrent context.
We will let ads win over removes.
That would mean that in this particular case,
if we merge everything, we might end up with the item in it
in the set because the concurrent ads
would cancel or out the removes.
But sequentially, actually, we have removed the element.
So depending on your interpretation,
you can get to very strange results
that might not also make sense to the end user.
So building on the automatic generation
of what we saw in the first problem,
we've been also working on an extension of this,
where that we actually are trying to create this building blocks
and different building blocks that we can provide
to the end user to build replications
that where you first will analyze, for example,
the data type, you will automatically detect
the complex different behaviors,
and there's different properties of your sequential implementations.
And then you will propose to the user look,
I find five ways that I can produce a serity,
and this is to be these behaviors, right?
And so we can give these components
and users can also pick what states best.
In the first version that I showed,
the computer will give you something that solves the problem.
But here, we're actually trying to give the end user
the ability to pick a bit and see what fits best
to their application.
And the realities that often see what it is,
may not be the best case, but at least we give them
a way to envision this.
So this work, excitingly enough,
we're doing this also with two important figures
who worked on the original serity paper,
Carlos Bacero, and no producer.
So we are really getting important feedback of them
as well on doing this correctly.
Finally, in the understanding problem,
well, understanding or it is hard,
but it doesn't have to be.
If you're a master student or a PhD,
you can go to the Ford Summer School on this route
or replicate an environment where you can get direct class
from some of the experts in the domain.
So what I want to say about this first part
is basically that well, consistency can be easy
if you do have the right components.
And so this is the research that we're doing at university.
We're looking at ways we can make serity more efficient
in what ways we can target new settings,
for example, imagine that we are in a system
where we transmit packages over Lora.
We have a very small bandwidth, right?
We cannot just broadcast data at random
so we really have to consider how we package our data,
what representation we use,
how is our causality linked,
so that we can have as efficient data structures as possible.
So this finishes the first part,
where I give a bit of a view of the problems.
And I will go back into FLEC,
which is my personal laboratory for experimenting with seritys.
So I started my research on seritys actually back in 2018.
And I was using a lot of Lua code to implement my seritys.
I had a small little framework.
And later, it is actually evolved into a TypeScript
based implementation because I wanted to use
assembly scripts to compile this to WebAssembly.
And we were running this on embedded microcontrollers.
In the end, we just focus on the TypeScript.
And so what is this?
What FLEC is actually is a small tool,
which gives you several libraries.
Let's say a causal delivery middleware
a serity-based serity library
where you can implement state-based seritys,
operation-based seritys.
And we have this important layer called
pure operation-based serity layer.
So pure operation-based seritys are like a known ad hoc way
of designing operation-based seritys.
And what's really nice is that we can reason easily
about stuff like when we should drop certain metadata
in the network.
So this framework is built on something I call TSAT,
which is actually an implementation of what you call
ambient oriented programming, which was a bit popular
in 2009 on top of TypeScript.
So I'll give a bit of an example of that
and show you how we use this.
So the most basic example that I can show you
with just the not-slag but TSAT
is that we're going to have a ping-pong application.
So on one side, we can make a new actor.
And we can export an object, a ping-pong object
on the network.
So here you see ping-pong.
And so these nodes can be using web port
to see peer-to-peer.
You can have shuffle a bit to the middleware layer outside.
You could even add a Bluetooth layer if you wanted.
But the idea is that you have a full mesh network
where nodes can discover all objects exported by the order
nodes.
And so that's what we do exactly in this computer.
We're discovering, we have an actor,
which is discovering a ping-pong object.
And so we got here is a reference.
And what's really cool is that this reference is actually
a four reference.
And any operation that would be originally synchronous
on the other side is automatically
related into an asynchronous call.
Even at type level.
So here you would see the reference,
and you would get a full type of it.
But let's say if the original sequential implementation
would return string, here we will get a promise to a string.
So we can safely type and implement asynchronous applications.
So on top of this, we have actually implemented
and the serity layer.
So the most basic example is that we can, for example,
we want to have a simple shopping list, the most basic one.
We say we want a net wind set.
And so we say this at wind set should go online
with the name shopping list.
So now anywhere in the network, I can discover this shopping list.
And so that's what I do here.
And discovered the shopping list.
I set a callback on the shopping list
to get updates on when the serity itself
can rely on serity is updated.
And so here you can see I add some items
as you would do in a regular sequential data type.
I forgot to go over my arrows, but OK.
So the implementation of the yet-wins itself
builds again on top of, let's say,
or pure operation-based serity theory.
And we kind of basically defined these rules,
these redundancy rules, which map actually directly
to their specification as you can find in academia.
So one of the goals when I was building this framework
was that I could easily experiment and change it,
changes to these frameworks, new concepts for serities.
For example, I introduced redundancy relations
that allow you to access data directly
that is cash in the reliable causal broadcasting layer
to allow you to have more efficient updates.
And so that was my goal with building this little laboratory
for myself.
How do I make something that I can easily extend and play with?
So this is a bit of unclear, so maybe I should give a visualization
of what actually happens when we receive operations
and what redundancy relations are used.
So imagine that we have a serity, a pure operation-based serity,
and internally we keep a log of the events that happened,
only the ones that we need to.
We actually compact this log regularly
and we keep a compact state.
But basically, whatever we keep in the log,
is the frontier of operations, which
are not causing stable, which we don't have information
if they have been observed by other replicas.
So let's say we receive a remove or a operation.
What we'll first do is we'll check if anything in the log,
the existing log of the serity will become redundant, right?
And we do that using the redundancy rules of the pure operation-based
serity framework and well, or implementation.
So basically, we say that any existing item is redundant.
If either the newly arriving operation is a clear operation,
or if the existing item has the same arguments.
What's important to note is that all these operations
are arriving in causal order, the underlying framework
will ensure that they're causally ordered.
So in this case, we can make the ads for a redundant.
And then, before we put the element itself in the log,
we will check, actually, do we need to keep it.
And in this case, for the ads, we actually
do not need to keep removes.
Because the serity will converge without it.
And so what happens is that actually, then, in this naive version,
we will just iterate over the log and compute the difference
and compute the state.
And this result is here T, which contains B and C.
Now, in an actual implementation, so in black,
what we do is we keep this compact state.
And whenever operations are causally stable,
we apply on sequential version to ensure
that we have as little internal data usage as possible.
And the different techniques, actually,
to clean this up and force it.
But so what was important for me
was that we had these different layers where I could
poke and play with, and so I built this extensible implementation
where we, at different levels, have different hooks
that I can hook in.
For example, in the TSAT core layer,
I can hook into message shunning, message receiving,
the ide generation of your different nodes,
when events are received, updated, et cetera.
So that's really nice that I had these distinct ports,
which helped me reason about CRT's replicas, network messages
in a flexible way.
And that's where the name flag actually also come from.
So for the CRDT layer,
I could hook into when nodes would go online.
I could have hook into when operations were generated.
Operations were received.
And in the buffer, the causal buffer,
and also finally applied.
So different layers.
And on because I have these hooks,
now I can build on top of this.
And that's where these, the pure operation layer
and the RCB layer, build on.
So the RCB layer then adds this causal buffering.
And the pure up layer adds this method
for dealing with all the redundancy layers.
So actually, when we really fast through my presentation,
and this is my last slide.
So I'm giving you some buffer time.
But so I will conclude with basically saying
that designing and implementing replicated data type
is hard.
This is why you often will go to one-fit-all solutions
that are out there that are made by very small people
who have already talked about a lot of the problems.
But sometimes your problem might not fit this one
for fits all solutions.
So we've been working at getting like computer-generated,
provenly correct designs out there.
And this is important because we can optimize for efficiency.
We can optimize for correctness.
And we can also really try to make
a series ofties that sometimes makes sense, right?
Because sometimes we want to build an application.
And in reality, concurrency forces you
to make compromises.
So we want you to develop, or at least,
to be understanding what compromises they will make
and what it means for the end user.
So yeah, the core is we're trying to distill
the core building blocks horserierlyties
and let the computer do as much calculation as possible.
Let's see end of my presentation.
And thank you for listening.
Thank you, Jim.
I'm sure we have questions.
Yes, I was thinking about you.
Yeah, I wish that's getting closer.
It might be a bit of a...
Is it on?
Yeah, it is.
The remote here is neutral press.
It might be a bit of an assumption of question.
But I was...
My trainer thinking is that this is still highly dependent
on some elements of time date.
Do you foresee, or do you imagine,
some way of achieving CRDT, you know,
just consistency as all, without that?
We're dealing with time and date.
Without human data, without...
Ah.
Well, the thing is what's really important
with developing strategies and application
is that you verify the correctness, right?
And so the benefit of the tools that we develop
is that you give the sequential specification,
which is always based on the business logic
of what your application needs, right?
And so that is still something that has to become up with a human
or maybe these days you could do an LM,
but I don't know how much you could trust it to do the correct thing.
But you don't have to, or...
You much less have to think about designing the correct CRDT
because the semantics will be automatically...
that's the goal verified by the computer for you.
So you get something, you can see,
look, we can offer you this, these strategies based on your spec,
and well, you decide if this fits your application.
But at least you don't have to think about
how you go into the details of it.
Thanks, Jamen.
Things are really, really great body of work.
And as a creator of a one-size-fits-all CRDT,
I think it's an empirical question.
We don't think it's the right approach.
We think it's an expedient approach to try and test whether
you can have a one-size-fits-most approach.
And we don't think it's optimal, but it's useful.
It's got pretty far.
I, in a past life, did a lot of database stuff.
And what I noticed is nobody actually understands how to use databases
and the vast majority of postgres features
are never used by anyone except consultants.
I see someone from the postgres community having a chocolate
front of me.
It's very true, though.
It's very true.
And so I think historically, I would have been worried
about this, but I'm curious now, it's been a few years since Flek.
Has anyone been making any progress on type inference,
static analysis, so that we don't get
sort of either the laziest or most conservative interpretations
that could actually use PL techniques to find the weakest forms
of consistency necessary.
Does it feel like a promising direction?
Yeah, so I maybe will go back to this slide.
I went to four, sorry.
Two, sorry.
No, let me see.
Show navigator.
Uh, I won't.
Yeah, you're just fine.
Yeah.
So it's a very good question, right?
Because there is no right solution for very many things.
We're just trying to create something and see if it fits.
So in this last paper that we were working on,
we're trying to really analyze the building blocks of what
makes a replicated data type take.
Where they core essential ports.
And this is why we're also working together at like Carlos Bacero
because they have a very good insights on what is important
and what has been needed for them to come up with the concept
of series these in the first place.
And so it's a very difficult problem.
We're trying to analyze as much as possible given,
let's say, some goal, some business goal,
some application, right?
What is the minimum we need to make something
that you can automatically compose these components
and get something else useful?
So in the end, the one size set all solution is actually
really good because you get these components
that you can play with and build it and easily iterate.
So we're also trying to do that somehow,
but create it by some algorithms that help you formulate
this in a way that fits this package, right?
Is it still bringing this to Papak this year
or will we have to wait a little longer?
So this paper is going to be submitted
in the initial version of this paper is going to be submitted
in to Eco?
OK, great.
Well, hopefully it's going to come out.
Hopefully it gets accepted, let's say, to I get first.
Because obviously we have to be verified for correctness.
It's okay if it's the key of the show.
Yeah, but we don't want to burn the paper immediately
for sure to try to get it published and then we see.
Well, thank you so much.
Thank you for your question.
Another question?
Yes.
Next question.
Yeah, thanks.
This was a great presentation.
I think, very information dense.
So I'm mainly wondering, do you have either these slides,
but like links to the papers and all the work somewhere,
or I saw the end, you have like a page for the faculty
like to disco, is it all on there or like back?
And I find more information in the end, indeed, to the end.
Animations make it navigation slow.
But so basically, if you go here, that's where all
our publications will list as well.
So some papers and I had listed there
because there's still an owner progress,
but you can find through this site, probably a little bit,
or you can look up my name at like the name of the...
Ah, on the page of your song?
Yeah, yes, as well.
And also really great is cdt.dech website,
which lists super much resources on cdt's
and of very prominent goods researchers.
Well, nothing, I'm not good, but I mean,
more famous researchers, like Martin Klappman.
So cdt.dech is really a very good resource.
Good things.
Yeah, I'm Willem.
System where, basically, you don't want the programmer
to manually specify the CRDT, but you want it to be
generated automatically to a lot of extent
just that depend on the use case,
and are there maybe invariants or properties of the use case
that somehow need to be encoded to render this derivation
and is there tooling to help in identifying that?
So that's actually a really good question
because that's exactly where the problem lies, right?
In variance, dealing with applications, specific constraints.
So if you don't have any constraints,
you can easily build a CRDT.
However, as soon as you add invariants to the mix,
or you have data relations, it becomes much more complicated.
And so because we have a solver that can understand this stuff,
the goal is to have indeed that we can automatically
generate a CRDT's, which even have data dependencies
among them, and we have to respect certain rules.
Now, the first bit that we're going to publish
doesn't have this invariant port yet, because it would
be too large, but the goal is to have this in a subsequent
releases, and we've already been working on the solution there.
And can you help the programmer to specify the invariant?
So the specification for the invariant is basically
you implement, so the spec is given in a scalar-like language,
and there you can describe how your business rule should be
in contrast to operation that are applied
on the data type itself, the sequential one.
So you're supposed to write there, basically,
some of the invariants, but you do it in a high-level
language, and not in something like Koch.
And it's like open source.
Flake is open source.
It's not like something you should use to build the application,
because it's a laboratory, mainly made for myself,
but it's open source, and you can find all source code
on the GitHub, which is linked to my pages as well.