So hello everyone. Thanks for sticking by. It's the last talk of the day I promise. I'm
in the Andrew. I'm a database engineer at Bolt and today we're going to be talking a bit
about replication in time to be and how to move data into and out of time to be with some
sample cases that we've used at Bolt. You probably heard of us, we're a right-hand
company. We also do scooters and e-bike rentals and food delivery and we've been growing
massively over the last few years. It brings about some challenges in terms of database
scalability. So I'll probably skip this. You probably got enough of TIDB today. The main
thing about this is that it's my SQL compatible and where are my SQL shops. So that was
one of the main reasons why we chose to migrate from my SQL to TIDB for the compatibility
that means that our applications needed no or little read writing. Another thing that helps
us was it's quite easier to run at scale than my SQL. For example, if you want to do
DDL on five terabyte tables, TIDB is your friends, where's my SQL can do it with ghost,
but it's painful. So these are the scenarios that we're going to be looking at. The first
scenario which most people run into first is migrant and data from my SQL into TIDB.
For this we use a tool called DM. Surprisingly, it stands for data migrator. It's a separate
cluster from your TIDB instances. It has mostly two parts, DM master and DM worker. The
DM master is basically the brains of the cluster. It receives commands from the user, task
management, and monitors the workers. Basically, the DM workers are basically the BZBs of
the system where they connect to my SQL, dump data, and also read the binwalk to provide
ongoing replication. The reason that you see multiple nodes, for example, multiple DM master
nodes is for high availability. So if the DM master goes down you have to other nodes
to fall back on and your replication doesn't break. Some instructions on how to set up
the DM cluster. I wanted to much into it. It's basically you define the nodes that you want
and you deploy it and start it up. Once you have your cluster running, you need to define
some sources. Basically a source is just a connection to my SQL instance and you can have
multiple sources on your cluster. A SQL cluster can read from multiple my scales and write
into multiple TIDB clusters. You don't need an individual cluster per migration.
Here are some examples of how to do this. Once you define your source, you don't actually
have any data replication moving for that. You need to start a task. Basically, all a
task is telling it, I'm going to read from one or more my SQLs and write into a TIDB
cluster. It's also a powerful filtering system which you can do funky stuff with, for example,
merging multiple shards into a single TIDB cluster. You can ignore SQL commands, for example,
you can ignore the weeds so that if you want to archive all your data, then you can do
it with this. You can do filtering, so just replicating part of your data or renaming
of tables, so it's quite powerful filtering system. The task mode, yeah, it's basically
you can do a dump on reload with no replication. Have that end replication or have just
replication from a specific point in time. It's a very flexible tool. This is an example
of how to monitor your task from the command line and you have the usual stuff. If you
familiar with my SQL, you know what seconds behind master is, it's a replication way. You
can also do this from the built in graphana monitoring. We didn't need to set up any of
this. It comes built in with TIDB, so it's quite a good monitoring stack there. So some
tips after you start your migration or your replication, you want to do usually one of the
data consistent check. There's this tool called sync diff inspector, which basically compares
the table definitions and also the data at a specific point in time. This also works with
TIDC which we will see next. Another thing that sometimes happens is my SQL will purge
some binwalls that you need in order to get replication running. So there's an option called
re-way log in DM to copy the SQL binwalls locally so that you don't run into this problem.
Another very useful feature is this ignoring ghost tables. Anybody that's done largely
the else on my SQL using ghosts. I wish for a new thank you. You'll know that it
creates a temporary table, it duplicates everything. So DM is actually smart enough to ignore
that and apply the changes downstream without meeting any of the intermediate stuff. And
the filtering I've talked about, one use case that we've done is we had a very large table
in my SQL with int primary key and we're running out of addressing space. So this was a
good candidate to migrate to TIDC. What we decided to do is actually change the primary
key at migration time. So you can create the empty table schema with bigings, primary key
and then use the filter to ignore the create table, start replication and it will write
into this new table with the bigings primary key. So different scenario, now how to that
was to get data into TIDV. This is how to get data out of TIDV. Now it uses a tool called
TICDC and this tool is actually part of an existing TICDC cluster. It reads data from
TICIV and it will send it to a downstream sync. It can be another TIDV cluster, it can
be my SQL, it can be Kafka. So it's quite a flexible tool for change that capture.
So yeah, you can add your TICDC nodes either to an existing TIDV cluster or you can add
them when you're deploying a new TIDV cluster and some of the limitations or requirements
is that your tables need a primary key or a unique, not null key in order to, you
need to identify the rows that you're replicating. In terms of network latency, it's recommended
under 100 milliseconds between your TICDC nodes and your downstream, mySQL, TIDV, wherever
you're writing it to. So this is one scenario that we use at Bolt. When we do TIDV version
upgrades, major version upgrades, the upgrade itself is online and rolling, there's no
downtime. But if something goes wrong and we want to switch back to the previous version,
there's no way to do an online rolling downgrade. So our strategy is the same old strategy
of having a replica or a replica that tried to be a cluster in this case with the old version.
And in that case, if something goes wrong, you can just switch a traffic over to the original
version. And setting this up is quite simple, you just load your data, you do a USB
R to do a restore. And then you start your change fees from a specific point in time,
you might have heard the term TSL, a timestamp oracle. And replication start from that
point in time. One gutcher is you might want to increase your garbage collection lifetime
on your source TIDV. The default is 10 minutes and usually that's not enough. So after
you load your data, if it takes more than 10 minutes, then replication doesn't have enough
back data to fetch from. So we usually put 24 hours and then set it back to the original
value. If you're from the MySQL world, this would be more or less how many binwalls
how far back the binwall goes. So TIDV C has the same filtering capabilities that DM has
and you can do all the same tricks there. To verify, you can check it from the command
line and also from your final, has expected. And the next scenario is similar to the previous
one, but using a different AWS region or another cloud region. The reason we use this is
more for disaster recovery. In the unlikely event that the whole cloud region goes down,
we want to have a running database system that we can switch to. It's basically the same
thing as a previous scenario, but there's a few gotchas. Since we're running on different
regions, latency is higher. So there's a possibility since 65 to put your CDC nodes in the
downstream cloud so that the closer to the downstream TIDV cluster and you don't have problems
with latency. We ran into this into infinitive lag because of this problem. And also another
tip is rather than having one single change feed from source to target, try to have smaller
change feeds. When we switch to first-key mid-change feeds, our latency problem will reduce
almost nothing. So that's quite helpful. I won't be talking about TIDV to Kafka because
that's used by a different team in the company, but you can also do that. And the last scenario
is replicating from TIDV to my SQL. You might be wondering why would you need this? We're
ready. I'm TIDV. Why do you want to get back to my SQL? The reason, again, is for insurance policy.
So I'll show you a scenario that we've done. One of the migrations we are doing multiple
shards into TIDV. So you can see we had multiple DM workers reading from separate shards
and consolidating them into TIDV. But then you see we used TICDC to replicate back to my SQL
cluster. The reason we did this is in case the my SQL to TIDV migration went south,
we needed to fall back into a mySQL based system. So this is why we would use TICDC against
the mySQL targets. And yeah, I wish I had a little bit more time to go in depth, but yeah, you can
see some of the pink capsules that are available for replication. And I don't think we have any
time for questions, but I'll be here tonight and tomorrow in the mySQL room. So if you have questions,
feel free to grab me. Thank you very much.
Yeah, I think all the end users are here. So if you have any questions,
we'll stick around until they kick us out. And thanks for coming to our very first cloud native
databases, everyone. Thank you.