All right, everyone, if you could help me in welcoming Andreas Andreas as a core maintainer
of P2 Panda, he's going to be giving a presentation on walkway systems.
These are infrastructure-independent P2P architectures that allow applications to switch seamlessly
between internet, mesh, and other transport layers.
So if you'll help me in welcoming Andreas, everyone.
Thank you.
Okay.
I'll try that again.
Hello, yeah.
Yeah, so they walk away stacked.
That was this term, which was brought up during the D
Web seminar, which was an event I helped organizing
in the internet archive in San Francisco last year.
Yeah, about me, my name is Andreas.
So ADZ on the internet, I'm a musician and a developer
since a long time.
And yeah, as we already heard, I'm also one of the maintainers
of peer-to-panel, which we started
like also roughly six years ago.
Cool.
So this is basically like a kind of like a mini-compressed
version of this lecture I tried to give.
Some concepts might be known to you, some others, not.
And bear in mind, this is if you're a radical framework.
I will, you know, it will be rough on some edges.
So most of the time, we want to send a message to someone, right?
So here we have Bobby, Bobby wants to send a message
to Daphne, you know, we have the message here and Bobby.
And Bobby probably wants to send it over, that's
the internet.
And, you know, they sent the message to a server or to another
note.
And then what we usually want to do is like, yeah,
we want to process that message.
You want to look at it.
Does it look correct?
Does it have the right shape?
Then we want to store this message in a database.
And then we want to do queries on that database
so then finally render it into the UI.
So now, of course, now we want to move this into a peer-to-peer
space and there we don't have servers.
And maybe we want to use other authentication strategies
then then checking your email address if it's valid or something
or your phone number.
So what we use is public key cryptography.
We sign the message.
We attach the public key to the message.
And now wherever that message arrives, we can now check that
signature and we can say, OK, cool.
This message actually has integrity.
It's a guarantee we gained now.
The integrity guarantee is no one has tampered with the contents
of this message while it was underway.
The other thing we also gained is provenance.
We know where that message came from or authentication.
These are guarantees we gained from this.
And this is, I would say, some base guarantees
we want for a walkway stack.
Of course, we need other systems to map Bobby
to actually that public key, but this is another problem.
Yeah, and because I'm trying to paint the model here,
let's try to make this even more abstract.
So we can say somehow this message of Bobby needs to be sent.
So it needs to be sent over something I would call the connectivity
substrate, like the internet protocol or a USB stick,
or we don't know, it doesn't matter.
But it somehow needs to be sent.
And this message, that's our data type, right?
Like something we define.
And then we can now even more simplify this into,
let's call this bottom part the event delivery layer,
because this is the thing which brings the message
to somewhere.
And then we want to process this on the event processing layer.
This is another simplification.
And then finally, we have the application on top of that.
Cool.
And now, some people happen to like local first links very much
and in local first, we always want to do everything
like right on the machine locally.
That basically means that now everyone turns into a server.
Every computer actually buys this, including.
And we want to store the data there.
And the problem, which we have to now,
it's a think about this when everything is stored on your machine,
like it's that you might be missing out on someone else's data.
And that can especially happen when there is network partition
or network fragmentation, literally things
like there is no internet.
And so now we end up with like an inconsistent state of the system
because Stephanie didn't get the messages of Bobby.
And then now we have a problem.
This can be easily solved with something
like people call a sync protocol or like a replication protocol,
which is basically like two peers asking each other
what they're missing of each other
and then, you know, it's usually expressed as some sort of state vector.
And this is enough information for the other peer
to compute the difference between these two sets
to then sense the missing messages to the other party.
And what we gained from this now is like eventual consistency guarantees.
Like these systems will now converge to the same state
and that will make everyone in the local first world happy.
And now if you look really close to what we've just done,
it's actually already a CRDT.
So CRDTs can be very complicated to understand
but this what we just saw was already a CRDT.
So we have a set of messages like all the chat messages
which can be sent in this group,
which were sent in that system and, you know,
eventually everyone will see all the chat messages, right?
And it also kind of doesn't matter
in which order they arrived,
they will be just sorted by timestamp afterwards anyway.
And also it doesn't matter if they arrived multiple times
because, you know, like we just ignored that.
So here we have a very simple CRDT already.
It's called a grow only set.
And that's interesting, right?
Because we're suddenly in like this world.
We didn't even notice that we did it
and I think many of you implemented these things
and maybe accidentally already implemented the CRDT.
Another terminology I introduced here is the replica.
So now we have suddenly like replicas,
which are basically like the local view on your state.
And now we can talk about like, you know,
peers having replicas of this data.
There's different CRDTs.
Very simple ones, but they're all really interesting.
Like the grow only set, we looked at a last right wins map,
kind of like a key value store where, you know,
the next item will overwrite the other one.
There is the append only look,
which is something people also call a hash chain
that's linked list where the next item points at the previous one.
And then you can have that also expressed in a tree
where multiple items can point at the previous one.
And then you end up having a directed acically graph
or a hash graph.
These are all like nice, basic CRDTs.
And there's proofs that there are actually functions also like CRDTs.
We can, they can converge.
And they have the same properties we want mathematically from them.
So let's extend this model a little bit now more.
So now our event delivery layer suddenly became,
suddenly got a sync protocol.
So now it's not only connectivity substrate,
like the internet protocol, but now we also express the sync protocol on top of that.
And we can kind of agree that might be on the event delivery layer, right?
But then at the same time, the sync protocol is pretty closely tied to the data type.
We are syncing with because, you know,
you might want to sync a deck differently.
And it depends only look differently than a set.
There's different strategies there.
I'm going to talk about later.
But we have to pick some data type.
And we call it, in the walkaway stack,
I call it like a base convergent data type.
It's basically, yeah, like this core data type.
And it can be one of these, or maybe something else, a mercury or whatnot.
Yeah, and this brings me now to even more simplification of this model,
which looks like this.
So we have an event delivery layer, as I mentioned,
the base convergent data type.
And then we do the event processing on top.
And this is getting now quite interesting, if we start thinking about applications
like with this sort of model.
And let me walk you through it.
So we start with the event delivery layer.
You know, some concrete connectivity,
stop substrates.
We could pick here, it's like we just heard,
I wrote, could use valid, could use the torrent network,
it to be what, you know, you pick your favorite substrates to connect to peers
and exchange messages.
There's different sync protocols.
I mentioned already, depending on your data type,
there might be different efficiencies.
You gain from them, for example, for sets.
And maps, there's the range-based set reconciliation by Alirasha Maya.
And the rebuilds is like something with bloom filters.
And the block height comparison, that's a very simple strategy
to compare the state of two append-only logs and then compute the diff.
I'm going to talk about this a little bit more.
But you see, like there's already kind of like a little zoo of things we can do.
And maybe the thing I want to point out here is,
it kind of is important what base data type you pick,
because it kind of influences the replication protocol
and therefore also your event delivery layer.
So they're kind of somehow tied together.
And then now we can also look at this base conversion data type.
And, you know, some people are like, yeah, I mean,
append-only logs are nice, but like, or sets, but I want a graph.
And my answer to that is, yes, that's cool,
because they're completely unrelated.
Your base conversion data type is just a carrier of your data.
And then you express other data types on top of that, right?
It's a little bit like the IP packet or the data gram, right?
It's just a carrier of something.
And we can understand them,
similar like here as well.
It's kind of like a carrier of any information we want,
including graphs or mercletaries or whatnot.
So we can sync DAX over append-only logs.
And you see this is basically like some sort of like enhancement of my message.
So this base conversion data type now gives me the cryptographic integrity
for hashing function, the provenance,
through a digital signature algorithm.
And it protects now my payload and what we gain is all of these get guarantees
and then on top even eventually consistency for our like local first worlds.
Okay, let's move up to the event processing layer.
So now we are observing these messages coming in, right?
And some people might call it eventsourcing.
In any case, we usually want to observe these events
and then we want to materialize state on top of that.
We kind of like, you know, message comes in and says like,
create this thing, then I'm creating this thing,
I put it into a database and someone comes around the corner and says,
I update that thing, then I change the value in that database.
This is usually where my materialize state lives.
And then I want to have queries on top of that for my application to,
again, render it to the UI.
And what is interesting with event processing is,
and this is, I think what I'm very excited about is we can layer the concerns.
And we can enhance our data types with certain processing methods,
which enrich my data in a way so I can then get even more guarantees higher up the stack.
Oh, God, I used the word stack too much.
I don't want to sound like, oh, as I model or something where you have these layers.
This is a much more pluralistic, much more, yeah, like, yeah, much more colorful worlds, I think.
But in any case, some data types, you want to enhance the certain features here.
We basically introduce something where I describe predecessors.
So I point with like a vector clock, which is cryptographically secured to like past items.
And now, suddenly, we composed on an appent or single writer appent on a locally composed multi-writer directed acyclic graph, right?
And with this information, I can now express a processing pipeline,
which, again, does the same thing.
My message arrives in the system.
I validate it.
It does everything look fine.
It's a signature correct.
And then now I can do ordering.
And, you know, like topological, like, basically like a topological sword,
I can check out my dependencies met.
If not, wait, if yes, forward it.
And then I end up with a topologically sorted serialized sequence of my events.
And then I get my partial ordering guarantee, which is also called causal ordering.
And that's awesome, because literally every CRDT kind of needs that.
And so once again, we are in a world where systems can converge.
There can be fully decentralized, and we can still get the guarantees that we eventually
will end up in the same state.
And all of this through like some nice layering and kind of optional extension of my data types.
Some needed, some other stones.
And this may be an invitation to kind of think about it as like a general class of things
we might be able to compose.
And there's many problems we have in that space, many, many problems.
It's not only ordering or signatures or integrity checks.
It's things like access control, permission management, capabilities.
We want to think about, of course, attacks on the system.
So we need to worry about Byzantine fault tolerance.
We need to think about garbage collection, so data shouldn't grow forever.
But we at one point also want to get rid of it.
These are like forms of coordination.
And we do want to do key agreements.
We want to agree on secret materials.
So we can have a shared key and then encrypts data with it.
And so on and so on.
So there's a lot of things we can do on that layer.
And this is, I think, which is interesting.
It's completely detached from event delivery, right?
This is like its own complete ecosystem, almost like a state machine.
We can just describe it as such.
And yeah, I think this is a little bit like a provocation.
We're like existing CRDT frameworks.
I think there's a future where I don't need to opt in
into a whole ecosystem of like a super CRDT.
But actually, I just want to text CRDT part.
And everything else can be done by other frameworks.
And maybe this even will be pluggable and interchangeable,
depending on my needs.
We're not there yet.
This is just an idea.
But I think it's for me it's personally quite exciting.
Yeah, and then we'll finally at the definition of a walk away stack.
So the walk away stack is basically like moving all of these application
concerns up to the event processing layer.
And then we can move away from the event delivery layer whenever we want.
And this is really beautiful because now we are not dependent on the technical infrastructure
that layer imposes on us and also maybe like the politics around this technical infrastructure.
Including internet shutdowns and censorship and whatnot.
And someone in the seminar called it power in balances as well.
So you choose your own power in balances.
Because we're always exposed to systems where there is some sort of like power dynamic going on.
But maybe if our applications don't break when we move to another system,
then we have something like walk away and choose our own power in balances.
So this is a little bit the idea behind it.
Cool. And now I talked a lot about direct connections, the internet and so on.
But I'm also here in a room where there's a lot of people doing mesh networking.
And now I want to combine these two worlds and see like okay I aren't they actually the same in some way.
If we think about it.
So in a mesh network usually let's I'm taking a very radical example.
I have an antenna and this antenna is just broadcasting.
Right. It's just like I send data. I just emit it.
A little antenna sending data.
Yeah. And you know it sends the data and it spreads.
It's a little bit like you throw stone into the water, you know,
and it's the ripples of the water just, you know, like move.
And at one point that little wave will hit another antenna somewhere here.
And message received. Right.
We have a one directional communication happening here.
And then we sent an answer and the answer comes back in the same way.
And here we go. We have our same scenario again of some communication between two parties like Bobby and Duffney.
Okay. So now how can we get eventually consistency guarantee? How can we make this local first?
So as I just said, we want to converge to the same state.
At one point every peer has to see every message. Right.
This is the guarantee we want to achieve.
How do we do this? We could literally just repeat the message all the time.
Until maybe everyone has it.
We don't know if they're going to have it. We don't have any form of acknowledgement.
But I can just do like best effort. I just repeated it forever.
And that works, of course, that's a fine synchronization protocol anyway.
The problem is here maybe the more data you created in your application,
the more crazy it gets because now if you repeat all of these things all the time.
And there's an answer to that. You send a pent-only look.
So an pent-only look is like a strictly ordered sequence of items.
And I know my current state because I am at the sequence number three.
Definitely is at sequence number one. Now I know that Stephanie is missing two items
because we can literally just compare these two numbers.
And this is something where we can simply just do by broadcasting our current highest of the lock.
And definitely just broadcast, hey, I'm at like lock height one.
And I receive this signal as Bobby.
And this is enough information for me to compute the difference between this and just reply.
And I do this until no one says anything anymore, basically.
No one announced their lock height anymore.
This is already, you know, now we're already in exactly the same space I just mentioned before.
And with the help of a pent-only looks.
There's other strategies as well, but this is like one example.
And very concretely, this is done in the project called Tiny SSB.
You can check it out.
And Tiny SSB goes even that far to do like a packet ready over short wave,
and with Skywave bouncing over the Atlantic Ocean with a pent-only looks.
And there, of course, like we all have the problem now.
As soon as we move this world into like,
yeah, mesh networking, we might have too much data in our payloads.
So we need to be smart with like what do we actually want to send.
Tiny SSB also has some interesting ideas of like a form of compression,
where we omit kind of like redundant data.
We can actually also just compute ourselves because it's deterministic.
The interesting project you should definitely check it out.
Okay, so now the mesh networks learn something from local first.
Now the other way around, how can we make like,
internet or IP based like stacks more like feel more like mesh networks.
So in, of course, on the internet, we always think about direct connections.
But we can express basically broadcast overlays on top of that.
We heard that already before something like I regoss it.
And this is like a very interesting sort of provocation also to developers
to think in these dimensions of like, okay, what if everything is broadcast, right?
And the new, if we still maintain direct connections in our system.
But the high level abstraction is broadcast.
And yeah, exactly.
So this is like my provocation.
Is it the same, maybe not.
But yeah, maybe we can actually express this in this high level sort of IPA API.
So we just subscribe to it topic, organize ourselves around like an identifier.
And then I just get a TX and RX like a way to send a message,
a way to receive a message.
And this is just like a sort of like almost like a broadcast interface.
This is very simplifying obviously.
We need to do so much more, what's in terms of privacy and security.
But I think it's like a provocation in this direction to think about these systems more from this broadcast,
even when we are on the internet.
Another thought around walk away is something,
some people call relays, some people call support notes,
some people call store and forward.
And this is like something we also want to have in these systems.
And we can actually also leverage quite well from mesh networks.
We want to relay data, of course, by intermediaries.
You know, like you are in a protest situation.
There is a building around in front of you.
But there are some people on the other side of the building,
how do you get around that building.
So basically what you do is you relay from everyone's smartphone,
basically from everyone's Bluetooth,
energy, capability devices around the building until the message arrived.
This is some sort of relaying problem.
And this is awesome and we want to have it all the time.
So basically what I do is like I have my message.
I encrypt it in a ways because I don't, you know,
I don't trust any intermediaries to intercept my message.
So I need to encrypt it beforehand.
It arrives on these relays and they are just forwarded until it finally arrives
at the receiver, which is basically definitely again.
And the property we want here is like this sort of, I mean,
fairly dystopian scenario, but I think a very important one is
relays can go down and we want still this network
to somehow still be able to communicate even when things get really shaky.
One idea we can maybe combine things with is like,
of course, how do you agree on this symmetric key right in the beginning?
I think like people from MeshTastik,
they usually do some handshake in person.
This is a sort of side channel where you agree on that key.
And then afterwards you get into your like secret communication.
We could also use the direct connections, actually,
and we still have the internet to use the CRDT to coordinate key agreements.
And then when we go out into the Mesh World,
we could then use that key to continue talking.
So this is like a very concrete example.
I think call is also doing a lot of work here in this direction.
And yeah, this is a little bit like a map I try to paint.
You see this is a very theoretical approximation of a very vast space.
There's a lot of stuff missing here, which doesn't fit on this slide.
But I think we kind of have a lot of things already happening
in all of these directions.
But maybe there is much more space for like actually interoperability
and collaboration across projects from this like MeshFear,
the local first sphere.
I think it's definitely possible.
Cool. So thank you very much.
I think this is it.
Yeah, we have time for one or two questions.
So is anyone have anything they want to ask?
Well with that, let's give another applause then.
Thank you very much.
Thank you for showing us this.
And if you do think of questions,
feel free to find a mastering talk to him.
Thank you very much.