We are going to talk about how we did playing, how he did playing the collaborative
team, his wages and production.
All right, folks, there was a lot of delay, sorry about that.
So, let's start. How many of you have used YJS or any other CRDD algorithm in production?
And how was your experience like did you face any issues at the start or was it all smooth sailing since
the start? So, we are going to talk about that, how we did it at playing and let me start
with a short story. So, last year we upgraded one of our core dependencies from a major version of
grid. And guess what? The next day everything was on fire, all the servers were crashing.
The Kubernetes was just restarting the pods again and again and traffic was redirecting.
It was chaos, but the best thing about CRDDDs was that our users didn't face any of it,
like they didn't lose a single bit of data and it was all super awesome.
So, that's what we are going to talk about the past, the present and the future.
So, yeah. So, YJS? Well, it's magic, but magic means more fees long.
So, yeah, a bit more about playing.
Playing is an open source project management tool. We have projects, wiki and air, everything in one workspace.
And you can sell for a set, air gap it, run it wherever you want.
And wiki is where the YJS lives. So, a lot of people get this a bit wrong when they start with any kind of CRDD.
Especially I've seen this in a lot of forums where people are super confused about this.
So, I wanted to start with this.
So, what's your idea of storing the YJS stuff like? Do you want to store the JSON or also the metadata?
What do you think? To get the collaborative part in action.
A lot of people think that we can just get away with storing JSON and HTML on the server.
And they expect that collaboration will work, but that's not the case.
And while I implemented it, luckily LLM passed through the docs well.
And I got to know that here I have to store the binary and not just the JSON.
So, yeah, we like wideboarding a lot at play and the aesthetic.
So, previously when we started, we only had a normal restrict set up in which you could not really collaborate.
And for that, we were using JSON.
But, with time as we wanted real-time collaboration and we did an on-demand migration to wideoc as users started loading the pages.
And let me give you a nice culture on this.
So, when you do this for the first time.
So, as you can see over here, the server needs to be converted into binary.
And if you want the local first experience, you use index TB as well.
So, as the client converts the JSON into binary and stores it into the index TB of the client, everything is all good.
But say, if the client doesn't really make any edits, then the underlying library that we use, that's focus focus.
That's the wisest web socket back end.
If the user doesn't make any change on to the document, it really doesn't trigger a save call of the binary.
So, you can see how where we are going.
So, if the user just leaves the page, comes back to it after some time, reloads it or anything.
The server again in the server's logic, we have the check that if the binary is not in the database, we convert JSON into binary.
So, but the client has the binary in their index TB, which was generated by the first visit.
So, now on the client's new, on the second visit, the new binary and the old binary come into index TB.
And wisest tries its best to merge things.
And what we faced was someone having like 1000 paragraphs and it getting duplicated in a really weird way.
So, the gacha is that you should store the binary immediately after conversion of the JSON for the first time, else there would be chaos.
All right.
So, the binary needs to be used as the source of truth.
For all the other needs, say for crawling, for indexing and all that, you can use the derived views of JSON HTML, etc.
And let me just, yeah.
So, the migration code is very simple.
You just have to encode the binary and store that in your database and load from here the next time.
And all right.
So, remember that fun little story that I told you about.
That's when the server's catching firepot.
So, let's dig into that.
All right.
So, it was a very nice Friday and focus focus V3 drops.
We were super excited that multiplexing works finally.
And there were a lot of issues in V2 that were fixed in V3.
So, we were very excited.
We did our homework, test data lot of stuff, did staging, QA, did everything.
It was just two lines of change on our side of code when we ran the code mod.
And we deployed it on a Friday.
I mean, what could go wrong, right.
So, we actually treated it as a minor release, but yeah.
So, I even made an arrogant post on Twitter that, hey, software engineering is so.
What you see is what you get types.
And that weekend was chaotic.
So, let me tell you what happened on that exact day and the couple of days following that.
In just a couple of hours, our servers started crashing out of nowhere.
And a huge customer also on boards on the same exact date.
So, they are using massive tables, hundreds of rows and hundreds of columns and multiple such pages.
And our memory just, our server just starts crashing like anything.
And it goes into a huge loop of crashes.
Well, you know, Kubernetes, it was just doing its thing.
So, luckily, we had our fallback logic.
So, this is super important that on the client side after if the server disconnects,
then we do something like this.
If it is disconnected, we fetch the binary from the database and then apply it on the client.
So, in certain intervals, we were basically sinking the client's slowly.
So, yeah.
So, it basically converges and we were using this by the way.
This was for a couple of months before we started using Hocus Progress.
This was our primary way of sinking stuff.
And it worked flawlessly thanks to VHS.
And yeah.
So, there was no data loss at all.
So, suppose that incident, we started monitoring all these things.
And this really helped us moving forward.
So, what was the bug?
In the Hocus Progress, the bug was that the documents were never getting unloaded out of memory.
So, with time, the memory just kept getting loaded and loaded.
And on the high load, a node just gave up.
And why was this?
That was because of the process dot next thing.
And we were depending on a lot of event loop stuff.
But yeah.
That was eventually fixed and now things are good.
Due to lag of time, I might need to skip this part.
But yeah.
This is how the client side code looks.
Like these are the different states that we have on a client side.
That helps us manage all the weird states that a client code go through.
An able offline thing.
And this force closed thing is something that is triggering a flow, force closed from the server.
If the page gets too heavy or something.
So, the local first UX.
So, we have index DB ready on our report.
That's what enables us to do.
So, for first loading, I wanted to show a really cool demo.
But well, it's a chaotic.
So, this is how we do it.
So, these are the different things.
So, first we check if the server is synced.
Then, yeah, show the content.
Second, if say the server goes down.
Like I just mentioned, then we are offline.
Or if the user is offline.
We show the cash content from the index DB immediately.
And most importantly, if the cash is ready.
And we know that we have a cash version.
We show that immediately without any loading states.
So, yeah.
So, this is the flow that I just described.
So, this was the super cool part that I wanted to share with you all.
The server side editing bit.
And this is not a detail we need to focus on.
But sometimes you want server side editing as well along with your client side editing.
So, say if you want to do operations.
When the doc is not really loaded by any user, we need to do server side editing.
But the challenge with server side editing is that you need to do.
It's such that it doesn't override users changes.
If they suddenly open the page somehow.
So, a really neat use case is that say if you have a nested page sub documents basically.
And if someone deletes the sub document, then the reference of the sub document from the main
document has to be removed in the content.
But to do that, you need server side editing.
So, we follow the shadow wide of pattern that I came up with.
And this also helps us to some cool AI stuff that I wanted to show.
This is how it works.
We create a shadow wide of.
And we calculate the data between the live wide of and the shadow wide of.
And then we apply those updates back.
So, that now the updates are super safe.
So, if you want to take any important points from this note, is these three points.
And you should be good to go with your production stuff.
And thank you.
And let me just see if I can show you folks a quick demo.
All right.
Yeah, just a short demo.
Is it fine?
All right.
Okay.
Any questions?
All right.
Hopefully this was visible.
Yeah.
So, this was the server side thing that I was talking about.
That enables us to do super cool stuff, even with AI and stuff.
So, here you can see I'm asking it.
Can you create a detailed blog on snapshotting in wages.
And it starts to.
The AI starts to generate content.
I switch the page and work on something else.
While it's doing it in the background.
And it's a super cool like.
And you can control it via an API call as well.
So, you can trigger this exact same thing.
So, you can create a API call like over here I make a curl request.
Like this.
Can you please remove all the headings.
And it will do it in the background.
And you can then accept or reject changes.
But yeah.
Like, server side editing.
Really enables us to do a lot of cool stuff.
Yeah.
That's pretty much about it.
As you can see.
Thank you so much.
If somebody needs to leave.
Let's do it very quickly because we have people waiting to enter.