The leader node for that draft group. So that's roughly how it works. So let's say
take a V2, volume 2 goes down, there'll be a local election only on that region, it'll
be elected and you'll leader and then let's say you've done big in, select, insert whatever,
the transaction will not roll back it. Behind the scenes, there'll be a election and
the transaction will still succeed. Provided the other two, the majority of regions
are still up. To get application won't even know that there's been some problem and behind
the scenes, tidy, we will start moving the other nodes around in order to maintain the
safety for data safety. So that's roughly how it works. If you have any questions,
it's not a little physically unit, it's more like a logical unit, the region. It's
ordered, so you will always get the same rows of the same table, same index and the same region,
but they'll be ordered. That's what the region is. I have to finish in quickly, that's why.
So it uses percolator, it supports the read committed in an abstract isolation, level
it's compatible with NWDB's repeatable read. There are some minus subtle differences,
but overall it's more application than just work. PDB is the, as I mentioned, is the, sorry,
the PDB is the better data server that tracks all the region across the cluster. It does,
originally tidy, we was written with optimistic currency control, but we found that many people
struggled with that. So by default, it's disabled now and it does pessimistic, currency control,
but it is still used in auto commit, optimistic currency control. So this stuff works pretty
well. What are the challenges that we face? So compaction causes a huge problem. So now
let's say you have a multi-tenant system or you have and it's all running on the type
of a cluster and the compaction kicks in because of some application that's doing heavy
rights. It affects everybody. Can I get some water? So you want to avoid things like
compaction causing jitter in your database. So that's one problem that we face. Provision
can take a long time. So let's say you add a new node. Data has to be moved around behind
the scenes and that causes problems. It takes a long time to add a new node, even though
it's automatic. You want it to be instant. You add a new node and instant it should be
available. That's another problem you want to solve. We have people asking us, or
we're talking about 200 petabytes of data. There is no cluster that I'm aware of, that
even more than six petabytes. We want to solve those kind of problems. I don't think anybody
has the answers, but we will are talking about these kind of clusters. People are consolidating
lower to reduce cost. And they don't want the applications to interfere with each other.
They want to isolate them, but they want to have one cluster for all these applications.
Reduce cost and reduce complexity of managing these large clusters. You want to solve that
problem. So in any distributed system, a trade-off usually becomes a knob. Somebody wants
something, then you add another parameter to fiddle things to balance it for whatever you
want to solve there. The other thing is it should be on demand. It should scale down as quickly
as scaling up. So one thing you can't do is storage you still have to pay. But compute,
you don't want to pay if you're not using it. So that's another thing you want to solve.
So you want compute to be cheap. So you don't want expensive storage and you're not using
it. Let's say you have like a 20-pater by database. The hot set is only 100, one-tater by
let's say. You don't want to be paying premium for the entire data set. You only want the
hot set. And the cold data you want to pay as little as possible. And for companies that
are on which type that reduce the IDB, that one at petabyte scale, this is a big cost for
them. Use cost. So how do we solve this? So this is like really the bulk of the talk.
So we want developers to start small. It doesn't mean that this because the system can
handle, I don't know whatever, one-pater by it, that you have to set up a one-pater by
you should be able to start with 100 megabytes or 10 megabytes system. Start small, but
it's scale easily, very easily. So that's one target. It should have, you should be
able to use any kind of storage if you want, but ideally the cheapest, cheapest is not
just in storage. Because if you look at the cloud list, I guess 3, it's cheap to store,
but it's extremely expensive to access. So that's another thing you want to fix. You want
to grow with demand. It should be elastic. It should give you the, whatever, 29, the 5,
9, whatever it is, resilience. These are very difficult things to achieve at scale. And so
if Amazon or Google or whoever has solved that problem for you, you want to leverage it.
Don't want to create your own data center in your deployment. You want to leverage the good
stuff that they've already done for you, the hard stuff that they've done for you. Distributed
systems and notoriously complex. I would say they're probably the most complex thing that
I have ever worked on. That doesn't mean that there are not other, but I have not worked
on anything more complex than this. The other thing is that what we see in applications
is that applications don't just talk to the database. They talk to many other services.
So in the cloud, you want to use the cloud to leverage all the other services that the
clouds provide. And the last point I've just mentioned. So in the system or the data is
not being accessed, you just want to pay for the cheap storage. So they should be the cloud.
What cool thing? So this is roughly how the costing is on the cloud with S3 and blocks
stored like EBS. Storage cost of objects storage is low. EBS is high. Objects stored
like S3 is unlimited. EBS is sort of limited. Durability of S3 is very high. Lock storage
is less. Latency is higher for S3, but it's low for blocks. How do you get the best
of both? That's what the cloud-native database needs to leverage. How do you fix this problem?
And the cost is higher for request cost is high for S3 and it's low for EBS. So this is
really the environment in which this has to work and leverage and make use of it in the
most optimal way. So TIDV has something called serverless. It's multi-tenant. So it has
a disaggregated storage. So we've replaced Rock's TD. In the same way, anyone can download
TIDV to pull Rock's TD out and start cracking on it and trying to the same thing. Hopefully
I'll be able to get that across in some and waving way. So it uses a local disk as a cache.
That's the core idea. It uses S3 as its primary store. So think about it. Now imagine
you have a stateless compute nodes and you have a storage layer like S3 and you have
let's say EBS as a cache. If I fire up a compute node, I don't need to do any balancing
moving data around. I just connect directly to S3 and instantly it's available. So that's
one huge thing for people who have to fight fires in emergencies and there's like a spike
and you just go, okay, spin up. And you reduce the latency so that the user doesn't see
anything. You have a pool of these hots, what would like all of them? Spot nodes, spot
instances, instances. So if you were doing it yourself, you would have some kind of Kubernetes
thing with a pool of spot instances and they could be all kinds of different shapes, sizes,
whatever. You fire one up and just connect to S3 and it just works because S3 can handle massive
scale of throughput. Without affecting your quality of service, it's like 2019 in its
fantastic. So this is what you want to leverage and this is really the core idea here.
So it from a shared nothing architecture of like any, it becomes a shared architecture.
Simply because you use S3. Now what does the EBS do? It's like any other cache in a database
like above a pool. So in theory, if you want to reduce the latency further, you could write
your own memory, shared memory, thank you very much. You could write your own shared memory
distributed cache. So if you're storage engine models, let's say an abstraction of a distributed
file system. So when it does the right, it first writes to S3 for durability, that's guaranteed
them then it writes to the cache. It'll use it like a right through cache. And so your
hotspot, your hot data will always be in your cache. So depending on how much effort you
want to put in and how much latency you could have. EBS as a cache, you could have a, like
a sophisticated, distributed memory cache, it's up to you and they could even be spot instances.
If you do it, just add some more EBS, increase your cache. This is the core idea behind
it. And to achieve it, if you want to write your own, just download Typev, rip out Rocks
DB, and start working on it. It's a very simple API from Typev to Rocks DB. It's like a
key value store. It's like, get port delete and few other things and let's say. One key
point. So in our case, which makes sense for anybody who wants to do it, we use, okay,
so how do you guarantee durability? So you have one thing that we do is we put the raft
raft log on EBS. So you want low latency. You have a currency, I haven't tried S3 Express,
but I don't think it has the latency you need for something like a raft, a raft log. But
that you use EBS. But a raft log is not a large data object. It's, it just needs to be active
to make sure it's like any other log like a right or a log of NTV. You'd fronkated when
you don't need, you know that all these entries have been applied to your fronkated. So it's
very cheap on storage. It just depends how much, how much of the log you want to keep.
It's a small, just enough to get the appliance nodes to apply whatever changes you're making.
It's not a big deal. So the raft log is on EBS and that is what gives us the guarantee
of or durability. So if the raft log says yes, it's been distributed. It's an atomic
commit. It's all there. It can lazily from there be applied to S3, no big deal.
Okay. One of the problems with the data structure that you see, we use is LSM. I have not
seen a beetry that stored on S3, so I won't go into that. So LSM we think is the best thing
for the indexing technique for this. The LSM is very easy to move around. They're immutable.
So once it's on S3, nobody's going to change it. So it makes things so much easier. The
pricey pay is latency, but then people have bloom filters and all that's going to stuff
that makes it faster. But one problem they do have, and we see it in production, is
legitimate. And how does this architecture solve the jitter? You can have spot instances that
work directly on S3. And because it's immutable on S3, you can run the spot instance as many
as you like doing all the merges, compaction in the background. It will not affect your compute
nodes at a serving business request. That's all the huge problem in LSM too. One thing
that is not in my slides or I'll mention is that in our serverless design, we have query
push-down. So what we've also done is we have spot nodes for what we call the remote
co-processor. So the optimizer, we'll look at the query, created
lag for aggregates and index scans or tables scans or whatever. And push it down to these
remote co-processor for long queries. And figure out, okay, this query is like a big one.
This is a small one. Small ones go directly to EBS, read data, send it back, process it
themselves. But for larger queries, we can fire up spot instances, which are called remote
co-processors and offload the query to those. So that the short requests are not impacted
by this. And all the larger queries are all done by these remote. So the only thing that
the remote, I don't, I've bit of this code. So I know this a little bit. The only thing
they need to do is, because the latest data is in the mem tables inside the storage nodes,
it can't get that from S3. You have to synchronize the mem table contents from. So that's
a bit of latency. But because these are large queries and if there's a few milliseconds
synchronizing the mem table for the data, you need to sort of engage in practice. So that's
the extra step it needs to do. So you can keep, it's a design to be scalable. So you can
keep adding these remote co-processor, you can have bigger machines with more RAM to do more
processing of these remote co-processors. And they send their aggregated results to the computer
nodes and of here. So it's designed to be really, really scalable at every aspect of
it. So this is what I've been talking about. So you have the compute nodes, it's a
stateless. You have the storage there. And the storage there, it should work theoretically
you can work without a cache. Cache is just an optimization technique. It's not really
requirement for correctness. So if you look at the diagram, you have storage nodes that
are writing directly or accessing the S3. And the part it's keeping is the rough log, which
I'll have in the next slide. And the resource pool, which is, you can have Kubernetes doing
this stuff. You have MPP would be your Typhlash node that do parallel analytics queries and
stuff like that. DDO. So there's a DDL worker. So we also have distributed DDL, which
does, if it follows Google's F1 online scheme, a change, which can do parallel DDL across
all your nodes. It's not in the serverless one, but in our regular Tydb, I think we've
got 40 terabytes add indexes like half another something. So it all the nodes are working
in parallel. So it's really impresses stuff. So this is what the general architecture looks like
at the hand way we level. This is the same thing, but in more detail. So in our deployment,
you want to, like, a more sophisticated production to form, you need gateways and you need
like manager server that are managing your pool. So that's where the complexity. I won't
talk in detail about because that's got a little, do with the storage side of things. So
you will need all these things. You'll need some kind of pool controller that goes and looks
at the stateless pool of compute nodes. And this is operational stuff that is very specific
to how people want to run it. But you will need some layer that makes it automatic and makes
it easy to manage. So this is the storage part. So if you look at the storage node,
Tydb at the top would be the raft part of it. Sorry, that's the interface it has to
the compute nodes. And the raft engine is on EBS. And if you see the green part, so that
the EBS drives or EBS storage. And the raft engine writes to that. So you have automicity
in consistency and all that sort of durability come from the raft log, the right to EBS.
And there's a raft engine on each of these nodes. And your KV engine for the data will go
to S3. And as a cache, which is missing from here, it would also use EBS for cache.
So this is what it would look like as a complete picture. So on the left is how the current
Tydb gave you that you can download from anywhere works. And it's more or less what I
explained. So that's a multiple instances. You have the main tables. So if you know how
LSM works, you have a main table. And then once that fails, I think RoxyB has 2 by default.
So then it makes it like an old main table. And immutable puts it on one side. And then
when the second one feels out, it writes it. The immutable one to L0. Then once it goes to
disk, it's immutable. And those little pink things with the dotted arrows are the ones
on disk. So this is the current one. And this is the more cloud enable architecture. And
we actually have our own version of this, but anyone can take a code and rip it out and
do their own thing. And so in this case, you can see the raft engine is right into the
raft log. That's an EDS. The main table part is common to both. Except in this case,
if you want to even offload your expensive or large queries, analytic queries to separate
spot node, then you would need some kind of synchronization with the different main tables
of the regions. And they would all go to S3. Anyone can, because it's immutably, you
just go pick it up and get massive amounts of data. So what does this do? It reduces our cost
by using the cache to reduce access to S3. So that reduces the cost on EBS, because the
only keep the hard data there, just by design. You don't have to do anything special.
The more you access, whatever you're caching, LRU, whatever caching strategy you use, based
on that, you'll get, and you can have, if you, just like a regular buffer pool in a
single load engine, if you have a smaller cache, and you have page misses. If you have
hits, the small EBS works for you. If you have misses, just like a regular cache, well,
then you have to fetch from EBS and there's more cost. So you increase your EBS cache,
just like you're increasing the buffer pool, exactly the same. And so the only mental
thing that you need to be sort of think about is that rather than you just memory for
cache, in this case, you can also use disk. So it's like a CPU L0, L1, L2, L3 counter cache.
Except the weird thing is that the analogy where it breaks is accessing, accessing memory
is not expensive. S3 is expensive. That's it. That's it. There are the latencies
I, but there's no extra expense, but where is it? In the cost factor, accessing S3 is extremely
expensive. So that's what you reduce. So you have multiple, so the rough plot is exactly the same.
There's no difference. So if you have your own draft implementation, you can use that or you can use
the one we have, it's not a big deal. So the remote storage and services is essentially
hot spot instances for doing compaction, for doing backup recovery, so allowing that.
So one of the things that we struggle with initially, when we went to like 4,500 terabytes,
the instances is backup and recovery. That's a huge problem. Backing up one terabyte,
500 gigabytes is not such a big deal. Try doing a one petabyte backup. Even incremental
will be hundreds of terabytes. How do you solve this problem? And how do you solve that problem
and not impact by using S3? It gives you such fantastic bandwidth. They've solved those problems
for you. That's why you want to leverage these facilities and reduce your own complexity
in your database design. It is a lot simpler to, in my opinion, today it will write really
scalable databases. You can go all fancy using whatever time sink and whatever other stuff
that Google or AWS offering, use atomic blocks now. And write your own protocol to reduce
latency to the same in the database. It's not so difficult. I think that's my last slide,
probably. Yes. So I think whatever this QR code is, if you use it, it's fighting open source
developers. We give credits to use our cloud service for free. I don't know how much.
Yeah, there you go. $2,000 credit? $1,000. If you want to see this architecture and action,
you can use this. If you want to write your own, just rip out rock TV, label the rest of
this stuff and write your own engine. And run Minio because it has an S3 compatible layer.
You don't have to go to Amazon to write your own engine. Just write up one Minio and start.
And write a little lesson and start using it. And that's about it. So you have any questions?
Feel free to ask.
Thanks for presentation. So you said multiple times that anyone can take open source
like we take rocks to be out and implemented like API on a S3. So like have you actually
seen anyone trying to do it in open source? No, so this is the first time I've said it,
but we've done it and I have worked on it. So I know it's not difficult. Okay, yes.
How's your mind? Anyway, the guy who asked the question actually hacks on Tygavey.
When pushing. Yep, please. Object on S3. Do you have an optimal size for the
object? Sorry? Do you have an optimal size?
Oh, so the size can vary. So and there's a reason for that. So the PD, one of the bottlenecks
in this design is still the metadata server that does the moving of the object. So the bigger
the object, the less they have to manage, the smaller the objects, the bigger the overwritten
managing so many objects. So you can have any size. So yeah, so what we do currently
is that when the hot, we know that this is a hot region, we start splitting it very aggressively.
So it doesn't even work that it has to reach this size and split. We'll just keep splitting
it more and more.
All right, one last question.
Hi, thanks for the great talk. So my question is like, how do you handle the batching
when writing or reading from S3? Because the S3 cost model is something different. Sorry.
The S3 cost model is based on number of calls rather the storage. So how do you handle the
batching of write and read to, is that a problem for the initial write, we need to do the
rough log apply, right? It really related to the previous question. So how do you find the balance?
So you do you want to write, collect it and write it fully, like bigger chunks or you want to send
the smaller chunks as you split them. So this is the job of the apply thread.
So even though you're aggressively splitting the thing, right, but that is, let's say you got a very
heavy load, right? And you, this could come from many mem tables. It's not coming from one, right?
So you can batch different mem tables together, create a bigger batch to reduce the request.
It's like let's say a read to log of into DD. If you're just going to write one little log at
that time, it's extremely expensive. It's like group commit. So you group commit as many
mem tables and fields as you can together. And the PD knows where they are. So you can slice and dice
whichever you like. Thank you.