So, basically, I'm from Pascho, I'm developed advocate for yoga by D.B.
Daniel Finneiden, I'm a technical support engineer at Pink App, which is the company behind TITIB.
And we look at the different layers. We will start at the SQL layer.
Yes, of course, as well layer, there's a big difference here, because there's both squares in my SQL.
But another very important difference here is that we decided to be my SQL compatible, but not to use any of the mySQL source code.
So, we implemented the protocol from scratch. The good thing about this is, like, we were able to do this in Go, which is a really nice modern language.
The drawback of this whole thing is that you don't get any of the things implemented for free, because you have to re-implement all the parts of the protocol.
And the advantage of re-using postgres, we have a lot of features that they come like this, but it's need to find significant errors, and that makes it a bit more difficult to make.
So, that was about the SQL layer about data distribution, because that's the goal of distributed database.
Yeah, so, with TITIB, we have TIKV, which is a CNCF project.
So, there are multiple TIKV servers, on which we are actually storing the data.
So, we don't have to make sure that, like, the data gets distributed for all the servers, and not that, like, one server has all the data, the other server has no data.
So, we need to make sure that the data is distributed, but also that there are, like, multiple copies being stored.
To make sure that if one server goes down, that everything, like, still is able to continue to run.
Which also means that, like, you need to take care of, like, a availability zones, because you don't want to have all the copies in the same availability zone.
And very similar on, on UGAB-DB, basically, this is based on the Google Spanner paper, where the table rows and the index entries are distributed, charted on their key.
And in UGAB-DB, we have also range sharding, so range of key, and hash sharding.
When you have only equality, queries, equality predicates on key, then you can apply your hash function, but you cannot do that when you have ranges.
I don't know. If you have more questions that just tell us, we cannot go into all details for all layers.
So, if you have questions or tell us, maybe I would just mention that the name, when we distribute the table rows on the index entry, we call the short tablets.
And, basically, the idea is to store that in our LSM-3 in RoxDB.
In a small volume, enough that it can move to one server in another in case of any problem.
Yeah, also with TIDB, we are splitting the data, we are not calling it a tablet, we call it a region, but basically it's the same thing.
So, these are, like, parts of a table, like 100 megabytes, those are the things that are distributed over all the servers.
And by doing this with a smaller volume, then, like, a complete table, you can easily, like, move takes in such a way that things are distributed equally.
And then it can be rigid distributed. I think it's the case in both case. You can add new nodes to your cluster, and then it will be rebalanced to have all cluster running your queries.
Yeah, another thing to think about with data distribution is that if you have one table, there can be, like, say, there's YouTube videos in there.
There might be some very popular ones, there might be some ones that are, like, old and no one really looks at them.
So, you need to be able to, like, move things around in your cluster, that the data is distributed, but also that, like, your reads and your writes are distributed throughout the whole cluster.
Then I have a question, do you distribute, then, not only on the size, but also on the number of reads and write operations,
we just continue to guide, we look on the other side.
Yeah, we try to distribute based on, like, many of these things.
And that's also, like, one thing there to take into account is that, with my style, you have an auto increment, which ends up always in the same data region.
So, you, all the new writes to a table, might go to the same place to the same server.
Usually, that's not much of a problem, but that's also why we created the auto random, where we, instead of a unique idea that increments, get our random ID,
then we can distribute writes, even more. Now, I'll run up to take the question.
So, the question is, in your guide, when we move the world, what's DB instance, or just a part of it, in your guide, each tablet is a root DB, and we move the world of it.
So, the goal of the rebalancing and auto splitting is to keep each tablet between 10 gigabytes and each gigabytes, that doesn't really matter,
but the goal is not to have one terabyte to move, when there is something to move.
Okay, that was about the distribution.
There's a question there.
So, the auto random, it really gives you a random ID. So, that's not really sortable by design.
There's even another thing there, is that with the auto increment in a distributed system, like, we have the option to either have like a central service, giving out the IDs, make sure that they increment,
but also like by default, we assign ranges of IDs to servers, then they are given out, which means that for my client, you might see like different gaps of numbers,
if the IDs are given out by different servers. So, the central one might be better for my scope compatibility, but might not be the best for scalability.
So, with distributed systems, you sometimes have to adapt things a little bit to take real advantage of the resources that you have.
Okay, so that was about how we showed our ranges of keys that go somewhere in a tablet or in a region.
And then, before storing it, we went to replicate it, because that's one goal in distributed SQL database is to replicate the data in different data centers, different zones, so that if one is down everything continues.
And then, what do we use to replicate?
So, rough this, what's being used for data replication. So, this means that, like, if you're writing a something, it will also send this write to like two other nodes,
and then wait to get acknowledgments from the other nodes to make sure that, like, the data is always stored on multiple machines.
Yeah, so the rough consensus, the idea of the rough consensus is to get a consensus between multiple nodes, and the idea of rough is that you have a leader and you have followers, which makes it easier, because the leader can synchronize many things, and in a SQL database, we have transactions, we have many things to synchronize.
So, all the reads and writes, go to the leader, then the writes are sent to the follower, and it writes for the core of it.
So, when you look at it, it looks like a primary standby database, except that this leader is only for a small part of the rows because of the distribution top of it.
Question for both of you, and like we're going to assume that in a rough group, all the nodes are necessarily on the front of a particular node.
Yes, this should all be on different availability zones. Well, one thing also to consider is that when we're talking about nodes, like, we have a data region, and the data region, that's a rough group.
So, if you have like three physical servers, and many like data regions, you have many, many rough groups. There's not a physical server that has one rough group.
Typically, those deployment are very nice in the cloud, where it's easy to have free availability zones. You can also do multi-region, but then you think about the placement of your data, because you don't want to pay the multi-region latency to often.
Yeah. So, my question is about rough and consensus. So, who might extend this kind of very vague about it?
Think that when you write, you go through rough consensus. So, if you write multiple round groups, or, like, establishing the consensus and stuff, or, in some cases, it's more tricky than that.
So, there's the writing to a rough group, and there's a whole consensus that happens with, like, leader, election, et cetera, if the configuration of the group changes.
So, if you're writing, you're writing to the leader, that is this is sent to the, to the other nodes of the rough group.
And then they acknowledge back, and I think as soon as you have a majority, the right goes through.
Yeah, the idea is that if the majority has acknowledged, then you know, so, the leader has acknowledged, and one of the two followers have acknowledged, if you have in replication factor free.
And then if a follower is down, you still have the leader with the latest value. If the leader is down, the two followers can't all together, and know which one has the latest value.
They need to, to all together, to know which one, but then they can. So, that's the idea, which means that the rights involve the latency from the leader to the, the fastest follower.
Okay, so, the only successful part, people can be worse than human classification. So, that's a very good question.
So, the question is, when everything is up, then it's not worse, that's the synchronous replication.
Next up, that in synchronous replication, usually in traditional databases, you wait only at commit.
In yoga bites, we wait more often, because we want to be resilient, if one note is down, you can continue your transaction without holding it back.
So, we don't synchronize for each write, we batch the write, and we synchronize when we need it.
For example, if you have reads and writes, then the reads must wait that it is synchronized to go to the other.
Yeah, and if you're doing like an update statement, you're probably not even dealing with like one rough group, but with like 10 or 100, because there might be like multiple data regions that you need to change.
Yeah, secondary indexes, and maybe checking some foreign keys, and yeah, another question.
These are all draft limitations. The distribution is mostly about your ability of the message rather than applying it to the follower notes.
It's not the case, you're both cases as well. So, that if I insert the raw, the draft protocol, just negotiate the fact that I requested it, rather than actually waiting for the answer to be acknowledged and applied from the written code.
Yeah, I'm not completely sure the exact details, but as far as I know, it needs to be like persisted, but also because it's the leader knows about it, and it's the leader where we by default are like reading and writing.
It's not like you're seeing outdated data, or you don't have to go to like multiple layers of something.
For example, in Uganda by DB, when you write, so when you write it writes to the draft log, the writer adds log for the tablet on the leader sends it to the followers, the followers write to their log and acknowledge.
And then when you have one that has acknowledged, for the majority that has acknowledged, then you can acknowledge to the client and the leader then writes to its memory table.
So, it's really written, after it is on the log of the majority.
So, this is the beauty storage. One of the common components here is ROXDB.
But both IDB and Uganda by using ROXDB as an LSM storage engine, which is a library.
And for us with IDB, it's the IKV that's actually using ROXDB to process things on this.
I think in both cases, we're not using just normal ROXDB, but always with small additions, small changes to make the performance better.
But it's still one of the common components.
And of course, some modification on ROXDB, ROXDB is very good to get in search, maybe a bit less at reads.
And in a SQL database, you have a lot of reads. Even when you write, you have to read for the phone key, duplicate key, all that.
So, a lot was improved for reads. The case also for food tables, cans, you don't want that of food tables, can just use all the cache and remove all the other.
So, it's based on all the DB, but with some modifications to make it efficient in this context.
And ROXDB is also one of the things that you can actually use with my SQL, with my ROX plugin. So, that's an interesting thing if you're interested in LSM.
OK, and then there are transactions, because in SQL database, it's not as easy as just writing what you update.
You need to do that within a transaction. And the big thing that is difficult is that when you read, you need to know the commit time.
But when you write, you don't know the commit time. You need to come back and change the commit time.
So, basically, for all the data, all the data gets a timestamp attached to it from the transaction, and all the transactions also have timestamp.
So, based on this timestamp and based on the data, you know whether or not this transaction needs to be able to see this ROX or not.
So, that's why it's really important that it's mostly correct, but that is really correct in a very small timeframe.
And in a distributed system, you have multiple nodes, and they all have their own clock, but here you need a time that is cluster wide.
So, here we do things differently. You can explain how you do in TIGB to get a timestamp that is cluster wide.
So, in TIGB, we have a timestamp Oracle. This is one of the things of the PD role of the cluster.
So, it's a single node that's also a rough group with two other nodes, but this one gives out timestamps.
And the timestamps which are given out are the actual normal physical timestamp, like the one you get from a clock, with a logical increment.
In case there, you need multiple timestamps for the same second.
And because it's always given out by this one component, we can guarantee that the order is always the same.
Like any thing that gets a timestamp later than the client before, always gets a higher number.
And for you, Gammaid, the choice was different, was not to rely on a single point of truth.
So, the way I under the clock that can drift, but adding a LAN port clock to it, when the messages exchange between two nodes, they update their clock to the latest one.
So, it's a neighborhood logical clock where you have a physical component that comes from NTP, but may drift and may have a high skew.
And an additional logical clock that doesn't synchronize all nodes at the same time, but at least those who exchange messages.
And to do that, and to guarantee consistency with that, you need to define the maximum clock skew.
You know that you have a clock skew, but cannot be more than 500 milliseconds.
You read something that was written less than 500 milliseconds, then you read it because you are not sure that it was done before or after.
But more and more in the clouds, you have access to precision time protocol or anatomic clocks.
And then you can read you that, for example, on AWS, we can reduce this clock skew from 500 milliseconds to something very small in microseconds.
And both databases are based on the idea of spanner.
And spanner itself requires like good quality, physical clocks.
But if you're deploying on cloud, it might be available like on AWS, but that don't guarantee it.
If you want people to be able to run it on their own data center, you can also not assume that the clock will be high quality low precision.
And do you need to be 100% sure or you have in consistency? You have a question?
Can I just use the timestamp or if you have a no objective? Why not? Do you have problems when the nature is changed?
Because you know what I'm going to say.
So with the placement driver, it's a rough group of free notes.
Like if the leader changes, then there might be like a very short time that like the timestamp might be a problem, but I don't really see that like overness like causing issues or at all basically.
Sorry, I'm going to go to good news and distributed as well.
Is there like a centralized place where all the locations for your data are stored, or is that kind of split out?
So I'm struggling with something you've said to, of course, how will you know what's going to be?
That's a very good question. So for TIDB, you might have like nine notes. You have a table of a few hundred megabytes.
It's split up in chunks. And then we have like in a central place, we keep like, well, this range of keys is stored in on this machine.
This range is stored there. So that's why we can like really do a quick look up if we don't like the primary key of a table.
So if I can ask, if that's your ghost document, it's a record of this entry start here in that entry start there.
So I'm not sure what you got by it does with with with IKV.
It's the data region that sends heart beats with all kinds of metadata. This is part of this metadata.
So if anything happens there, like it's easily rebuilt.
And the other thing there is that in case like the TIDB and the SQL layer has the wrong information about what key is where.
It doesn't really matter because it might try to write to a note that is not a leader for like that region, but then it will just get an error basically to refresh the metadata it has and try again.
So it's quite safe.
The idea is it's to not have a single point of truth. You depend on so each tablet or each region are really independent rough to group sending a bit and trying to solve the failure by themselves.
And we can move a bit more on transaction.
If you have the time.
Comment in question regarding the leader of the election, which is likely that the 90 makes for a rough to do an election.
You've wanted it, but I think it was more than clock school.
Question does it mean that every week we could request the TIDB and the city also includes another hope to be clocked service to tell you of giving us a question.
In many cases, yes.
I don't know the actual details, but what I said, like that there's only like one PD.
Of course there are two other PDs.
I think you can like delegate a few things to there.
And for some things, delegated times then might be good enough.
But yeah, I'm a bit lighter than details are.
This is typically a trade off having one time stamp or clock school.
And last topic about MVCC multivation control.
So this is also based on the time stamp, not the same time stamp, but the idea is the same is that if you update data, someone may need to read a soft previews read time.
So you need to store all the versions.
And it requires this time stamp and it requires storage of all the versions for some time.
And that's on point some garbage collection.
Typically in your garbage DB, we set the retention to 15 minutes.
If you have long queries, you can increase it.
But basically you can read as of any read point in the last 15 minutes.
So your query is consistent.
And garbage collection is done because it's whole DB, there is compaction.
So the removal of the old version is built in the compaction process.
Yeah, and we try to be will allow you to see like old versions of a table.
So you can just say like, I want the version of this table as it was like five minutes ago.
It will give you the data of the table of course.
But it will also give you the schema at that time.
So if you drop a column or edit an index, you will see those changes as well, which is quite powerful.
And the other powerful thing that we've done there is that if you drop a table by accident, that of course never happens.
But we allow you to just do a flashback table.
And because it was not garbage collected yet, we just restore it to the last version that we had and your back.
And so that's one of the like big advantage of like a garbage collected set up.
No, for backups, we have a different way of doing this.
So if we take a backup, all the type of storage nodes are writing their own backup.
And then we rely on the NPCC as well to be able to have a consistent backup of the whole cluster.
So it's a combination, you take snapshots, but the snapshots may be from different point in time in the different nodes.
And this is where MVCC allows you to restore at a specific point in time for all nodes.
Typically, when you divide the B by default, we have 15 minutes retention for MVCC.
When you do snapshot every hours, it will be garbage collected only after one hour.
And besides all the components that we talked about, there are so many other components that are shared between distributed databases.
And not just Yugabyte, TIDB, it also features, for example, like GRPC and many of the other things there.
Any more questions?
Anyone? How do you do the discovery of the storage nodes from the JETS?
So for us, we have the placement driver, the placement driver knows of all the machines.
So you connect to the placement driver, you get all the information, and that's how things work there.
And something similar, there is a catalog, a map of the nodes, and where are the tablets that must be shared.
Because we have something that we call master, I don't really like the name, it's more a master that orchestrates, where are the nodes.
It's a component that must be shared, so you do not want to rely on it for all operations to be scalable, but this mapping is shared.
And we also store the Postgres catalog in it.
So it's basically a two-phase commit that's being used.
I think the same is true also for a Yugabyte.
We have an optimization.
I think Yugabyte is one of the best things to do.
So it's basically a two-phase commit that's being used.
I think the same is true also for a Yugabyte.
We have an optimization.
I think Yugabyte has the same one that if you're only hitting a single data region or tablet,
then we don't do the whole two-phase commit or don't require you to do so.
But for multi-shot transaction, you have a transaction table somewhere that is also distributed.
That has the status of the transaction.
So when you read, you need to know if it's committed or not from it.
Okay, thank you.
Thank you.
Here, Mike.
Yep.