So, I'll skip this side.
So, most of you are probably familiar with the way mySQL and Postgres to replication.
So, they have a single node. Everybody writes that node.
The transactions are committed.
It's written to the binlog in mySQL.
And in Postgres, there are like 500 ways to do it.
The right-ahead log being one of them.
But the concept is roughly the same.
So, the transaction commits.
So, you can think of it as a queue with a single producer.
Multiple consumers. That's the abstraction.
New big deal. Quite straightforward.
Oh, sorry.
TIDB is the database that developed by Pink App.
So, it is a multi-writer, but not in the sense of Postgres and MySQL.
It is a disaggregated compute and stored.
So, you can have n number of nodes, n number of compute nodes for SQL, for example.
And n number of nodes for storage.
It doesn't have the concept of a primary node.
So, the data is automatically sharded.
And the shards are spread across the storage nodes.
And each of those storage nodes, think of it as a page conceptually.
It's called a region or a tablet in Spanner.
It's inspired by Spanner.
And each of the pages is a rough group.
So, the scale at which TIDB is usually used is about,
let's say, 250 terabytes is quite common.
And with the default page size of 256, that's roughly about a million pages.
So, that's a million rough groups.
Each page is made up of three other pages that are part of a rough group.
So, for TIDB, it's not as simple as Postgres or MySQL.
So, you're mental model has to change.
And if you understand that part of the rest of whatever,
I have on the slides will make more sense.
So, that's the challenge we are solving.
And the other thing that I want to highlight that is very different from both of these,
TIDB does online schema change, which means unlike MySQL and Postgres.
The replication doesn't block.
But in MySQL and Postgres, once it's a DDL, you have to wait till the DDL is finished.
And then the rest of the changes move.
TIDB doesn't work like this.
In TIDB, DDL is online.
It's happening in the background.
And so, if the changes coming in the replication stream from changes to the same table or other table,
the replication stream is still being propagated.
So, that's how TIDB works.
So, for TIDB, all this happens external to the storage nodes.
So, that's entire CDC is a distributed system in itself,
and HA distributed system in itself.
It's not just connecting one socket to another socket running somewhere and just streaming TIDB.
It's not as simple as that.
There is an entire HA system that has to store some of the state.
So, that the back pressure, because the nodes that are producing the data also have their own draft logs.
And you don't want to block them because you can't read the data fast enough.
So, they have to store some state.
And if they crash, they have to resume and not have to start from the last checkpoint.
So, it's a far more complicated and complex problem.
So, if you get that mental model correct, the rest of the slides will make more sense hopefully.
So, in this diagram, you can see lots of data coming in.
They're all writing to the draft nodes or TIDB nodes.
Then those events are collected and they are sorted.
The two main events that are important parts of the stream are the sorting because you have multiple writers writing.
So, you need to order those events so that when you send them downstream, they receive them in completely ordered fashion.
So, that's very important. That's a huge cost in this system.
The other thing is, you have to handle the schema changes.
So, I'll get to that in the rest of the slides.
So, how does it plug into the system?
It uses an observer pattern.
So, the two parts of the rough protocol, the way I can be uses it.
So, you get the events coming into the rough log.
Once they're committed, then they're applied locally on the storage nodes.
So, you put an observer in there that is observing these events and then it's pushing it to the CDC system.
That's the mental picture you must have.
That's how it does it.
So, this is, I want to highlight this up front and rather than do it later,
because this is very important for how to explain one of the bigger challenges that CDC solves.
So, assuming you have an insert, then you have, let's say, an alter column,
and then you have another insert.
You have to set the barrier for when the DDL takes place.
So, one of the things in distributed systems is, I mean even standalone system.
You need some kind of monotonically increasing counter.
If you have that counter and you can have a strict less than relation,
or which in distributed systems would be happens before, you can solve most of these problems.
And TIDB does that through a what's called a TESO or a time stamp.
So, it has a component called PD, which generates this global time monotonically increasing time stamp.
And that is unique and it's stamped on to every transaction.
So, the second thing you need in any database or transaction or distributed system is,
you need to know when that transaction started.
And when it was committed, or when the changes were externalized.
So, you want to propagate what was externalized.
You don't want to send data across that hasn't been committed yet,
because downstream could be a system that is not TIDB.
TIDB can probably handle it.
But let's say we are sending data to, I don't know,
somebody writes a converter for mySQL or one of those things.
And that may just work on ordered committed transactions.
So, once you're armed with these three things,
the start commit and like a,
some kind of counter in this time stamp or any counter,
which is unique in your cluster.
Conceptually there it becomes very easy.
You read the events, you do a happens before and you do a sort and Bob Jarenkel.
It's quite straightforward.
So, conceptually it's not so difficult.
The engineering of this is quite difficult.
It's the scale it's HA, these are the difficult problems.
And handling things like back pressure.
Because as the rough log is being created,
there's a lot of space and other pressure.
So, you have almost like a garbage collector that has to
truncate those rough logs.
So, otherwise they just don't keep drawing.
But you can't truncate it unless the changes have been pushed across to your CDC
and across to wherever they have to go.
So, that's the challenge here.
Once you have the time stamp, you know that anything that comes,
that are cannot push, that changes that were impacted by the time stamp.
We have a log service as part of this.
It sort of squirrels it away on the side.
It waits for the barrier as that DDL barrier time stamp
to be equal to whatever is up to the dissolved time stamp.
And then it's not pushing the entries after that.
But anything else that is not touched by this is being replicated,
which is not the case in my SQL and Postgres.
So, that's why I wanted to highlight.
This is a very important part of what it achieves and what it does.
So, we also learnt our lesson.
The current architecture just didn't happen,
like some kind of a maculate conception.
It was a lot of pain to get to this.
So, you have to start somewhere.
Nobody was an expert at this.
This was a new challenge for everybody.
So, the first solution was, okay, we have a node.
It reads the data, sorts in memory.
And it had a polling mechanism.
Most of our frontend software is in Go.
So, this is in Go.
So, they all use Go channels and whatever else Go has.
Sorted in memory.
Then they had ETCD in the background for HA,
where the metadata was stored.
And these worker nodes would poll data.
Do whatever it takes.
But the idea is growth.
And the kind of data that the customers are storing.
We would get OEM messages and all kinds of other problems.
I didn't scale.
Second step was, okay, you can have a hybrid spillover to disk.
Then suddenly people had 6 million tables.
We would run out of file handles.
Third, final solution.
Before cockroach DB, change the license.
We have a fork.
We write it into LSM.
It's written in Go.
So, it works quite well.
And you can index it.
So, current model is.
It uses Pable DB to store the data.
So, if you have a lot of pages.
As I mentioned, so let's assume we have a million regions.
Across three nodes.
You will probably need more nodes, but let's say three nodes.
So, if you are attaching to, that roughly is about 330k pages per node.
Now, I imagine you have to do an observer on all of these.
So, you have to have 330,000 observers.
Each with an independent stream trying to stream it.
It won't scale.
So, this is one of the channels.
And then you don't want to poll after some.
It's just, it just doesn't work.
But conceptually it will work.
It doesn't scale.
That's the hard engineering problem.
Memory pressure.
Huge problem.
CPU overhead.
So, the CPU overhead also will get into the pipeline later.
So, these are real bugs that we had.
So, these are this.
So, sit down and think.
You need decentralization.
So, even though today, all this microservices is not fashionable.
For this, you need something like that.
So, that you can scale independent components.
Especially the part that does the conversion from the internal format to the SQL or whatever
raw format that is required by whatever your downstream is.
That is a very heavy compute operation.
And you want to be able to spread it as much as you can in a service that you can have multiple services running
which read in parallel from some store.
You want it to be event driven without polling.
Polling even in a general programming sense is works where you can hammer the system with
and always keep it busy.
So, that polling, there is no busy way.
But it's not always the case.
So, polling puts a threshold.
You can't go faster than that.
If the events are not faster coming fast enough.
And so, PSAP operation can serve the extended engineering.
So, you would be divided this whole thing into four abstracts of the things.
There is an upstream adapter that talks to the Tai KV.
We have a log service which is stateful.
A downstream adapter which is with a conversion.
And the coordinator is what coordinates the cluster.
So, they are like four main services.
So, if you go a little bit deeper.
As I mentioned, you need the timestamp.
You need the watermark which is up to where the progress is on the rough log.
You need global aggregation.
You need to know across your entire cluster.
What the minimum is so that you don't delete anything which is.
This is quite similar to how things like Aurora also work.
With the no DBs under log.
You can't.
You have to look at where the read view is across your entire cluster.
You can't just look at one node.
So, you calculate that you use events.
You sort them.
And then transaction reconstruction is basically you map all the events of the transaction.
All the changes to the transaction.
You need.
So, Mount there is misnamed.
But that's what it is in the source code on GitHub.
Mount there is like a transformer.
And this is the compute heavy part.
So, it takes raw data because you want to sort on your opaque data because it's more compact.
And it's a big.
It's easier to manage.
Once it's converted into its SQL type, it becomes bigger.
And it's more difficult to sort.
So, the amounter it's a transformer is the part that you can have multiple.
And you can do lots in parallel.
So, this is what it roughly looks like.
You have a log service where all the chain feeds are coming in.
It writes into PableDB.
You don't do any polling.
And the number of tables is no longer a problem.
We don't store them based on table.
We just store the events based on the changes to the pages and the transactions.
So, some of the other optimizations we have to make,
where that if you look at the number two,
rather than have like the naive approach of multiple connections for each raft,
you just have one connection and then you multi-plex it.
And you do a connection per node rather than connection per that will never scale it.
But you have to start somewhere and so that's how it started.
So, you also need to order some of the events.
For that internally, it's like implementation detail.
It uses, I think, a go-be tree.
Some be tree implementation grow.
Also, you can put bounds on the memory that you use because now your storage.
And you have the ability to put back pressure and slow the ingestion of your service.
But you don't want any of the services to die.
You're much better off telling the service to slow down.
And that the user just needs to put more resources so that they can fix it.
That's the basic idea of back pressure.
I'll skip that part.
So, this is the part about the transformer.
So, the rough idea that I gave is the storage node.
You have the puller is also misnamed.
It's actually pushed.
Then you sort the data, then transform the data.
And the sink is where it receives the data, which does a more magic.
And then it sends it downstream.
So, what does it do?
It decodes the key value pairs, which is what the storage knows about.
Storage doesn't know about SQL.
It also needs to know the schema.
Because schema change is online, you know,
Thikevi can have multiple schemas.
And so, you shouldn't have to go back to the server to check the schema.
That's an additional cost.
You want a local version or cache of a schema, which is,
you can apply an update.
And when you see the DDL, you do the update and then you use the latest schema.
Because some of your data will be working on old schema and some will be on the new schema.
So, you have to maintain a local schema.
So, so by decoupling the whole encoding logic thing,
you can independently scale.
And that's where the microservices thing comes about.
So, how do you start all this?
Because it's ETCD, when you start the system up,
the instance is a register.
And these are all the different services.
They register with ETCD.
They elect a coordinator that does more like housekeeping things.
It doesn't do anything much more than that.
And then you schedule the change fields that's going to talk to Tikevi
and just get the whole system running.
So, because it's an HSA system,
it also has to do all these other tasks.
Like dynamic registration.
It has to do automatic recovery.
If some service fails.
So, that's the other aspect of this.
So, that's, it's a complete system.
It's not just like connecting sockets.
That's the point I wanted to make.
So, when there are failures,
you know, when you're running the scale at which Tikevi runs,
there are nodes going up and down.
There's always some kind of problem.
But you don't want the service to be affected.
So, it handles all things like brain split
and all the other things that come with any distributed systems.
Also, it can do.
So, in version one,
as I mentioned, the crash recovery had.
There was no intermediate state.
The crash recovery had to go all the way back
through a whole intermediate state
and that RP over Tikevi forever.
So, now, because it has local state,
it gets a leader.
It knows I need to get the latest data from
such-and-such log service that's running
and then it rebuilds from where it left off
and then carries on from there.
These are more or less.
So, upgrade down here.
It can handle partition tables quite easily.
Let's skip this.
Otherwise, I'll run over time.
Yeah, this is also good enough.
So, I've mentioned all this,
but I just want to go over this again.
So, the schema is also stored in the state service,
which is the log service.
The events are also there.
The table, so it,
table stream you can ignore.
So, any local event in that cluster is also stored here.
So, there's no metadata in ETCD.
This is as the state of the system.
So, the downstream adapter is for the events
that are coming from the log service.
It's more or less like a reader for the log service.
Then talks to the downstream adapter.
What data has mentioned just does,
it's internal CTC node changes.
So, what can it do?
You can do massive change feeds and scales very easily.
It's designed to scale by adding more services.
Large tables can be split for more efficient storage in the service.
Transaction integrity is preserved because the entire time stamp
is exactly the same that comes from the cluster.
So, there is no ambiguity anywhere in the pipeline way.
You don't know how to order the events or order the transactions.
So, it also supports go plugins, other hooks,
and it has an extensible architecture and we have connectors for Kafka
and all sorts of division and whatever else is the fashion of the day.
So, how much time for QA?
Excellent.
Okay.
So, anyway, there's the GitHub URL.
So, anybody wants to look at the code,
contribute, learn, go for it.
So, it scales linearly.
It has very high throughput.
It's a clear architecture relative to what it was before.
And it uses the last city of the cloud to scale very easily.
It's quite easy.
So, that's it.
Thank you.
Any questions?
Anyone?
Yeah?
Yeah, it's got all the checks on everything.
Yes, that means.
This is used by people who run very large.
Oh, how do you check the quality of the data?
So, it has all the other details like checks on whatever else is required.
Yes.
Okay.
Thank you.