So, we are going to talk about Vites which is a different approach to creating cloud
native than you have seen before and we are taking an approach of explaining how you
can migrate massive databases into Vites to flesh out the architecture of Vites.
My name is Roy Thnak, I am a core maintainer of Vites for the last five years, I work
at Nanick's scale along with the Mathias.
I am a CS, I work also at Tanstier, I am in customer engineering, so whatever he builds.
I get to implement.
So, Vites was created at YouTube in around 2010 when people felt the need to distribute
lots of cat videos, they were running out of bandwidth, so they built Vites on top of
my SQL, so this is a very different architecture from what you have seen before, it uses
my SQL as the storage layer, so they are not reinventing the storage layer, you get all
the benefits of my SQL and fill in the gaps that are needed to make it into massively
scalable database.
It is a graduated CNCF project, it is open source, so I will go in the next slide over
what makes it cloud native since it is a cloud native room, so Vites was built from scratch
to work on Googleborg, which is the predecessor of which is inspiration for Docker and
Kubernetes, so it was built to be cloud native from ground up, and we will see what that
means soon, it is in supposed to massively scalable, so that is given highly available, and
one of the core principles was that it has to be my SQL compatible, so that my apps should
be able to talk to it like it is a regular my SQL database, underneath you can have several
databases, they can be vertically scale, meaning you have lots of your databases across my
SQL databases, or horizontally shared it, where one table 6, one row will be a one
shot, but the rows are distributed across multiple my SQL databases, that gives the
massive scale, that is one of the key things of it is the horizontal charting, and
because it gives you a single connection, the entire database underneath is seen as a logical
database, you still get a single connection string, so database frameworks like Rails,
Larawail or ORM tools, everything just work WordPress for example, a lot of very large users
just to give you a flavor that it is actually running on massive scale, most of a slacks
traffic runs on Wittes, by runs on Wittes, Hub sort square cache, Americans were here
who use square cache, that is entirely on Wittes, Shopify and some gaming companies, so
it runs at really high scale, planet scale itself, where we work for run managed Wittes,
and we have, I know, several hundred thousand my SQL servers for across clients, and
the largest customer we have has probably right now two six shots, we are going to move
to 500 to 12 shots, so it is pretty massive, so since the cloud native data room, let's
devroom, let's see what is meant by cloud native, and why Wittes, what are the principles
that we used to make Wittes cloud native, so Microsoft is oriented design, all the
components that I use will see it soon, the different components, they are decoupled,
they communicate to each other over GRPC APIs, so you can containerize each of these within
a Kubernetes container, for example, and so you can scale it vertically by making each container
through hardware of the problem if it's, you know, you need more performance, or horizontally
sharded, horizontally add more of these components, so what that lets you do is you can leverage
commodity hardware, so I think a sunny talked about price performance, and that's really
important in cloud because you are running these massive number of instances, you need
them to be replicated, so you want to cut costs, so when you are able to use commodity hardware
and shard, so horizontally sharding, you spread your the cost across the database bandwidth
across multiple my SQL servers, and you can leverage cloud scaling, cloud allow you to add
new instances, drop new instances, very fast, so when you want to upscale, for example, around
thanksgiving event or you know black Friday, we have many clients who will just ramp it up
two, four times, and then bring it down immediately afterwards, and witness allows all that
very quickly, you will see how that is done, high availability is done, so we are using my
SQL, so we get the benefits of my SQL, the replication and semi-sync, and on top of that,
we have orchestrator, which is a cluster manager, which looks at all the instances that
are going on, there is a health check that is going on, on everything that is running, and
if like a primary fails and you cannot write, it will promote a replica automatically,
Kubernetes integration there is a Wittis operator, which has the IDs for all the different
components that we are Wittis components, so you can use just Kubernetes regular Kubernetes
for scaling, self-healing, give declarative information on how you want to cluster to be,
and Kubernetes will create it for you, global availability, so there is a concept of availability
zones similar to AWS, so you can have your clusters spread across different zones,
you can also be on different cloud providers, so there are users who really need it, I mean,
who want it because they want really don't want downtime at all, so I am just going to explain
the architecture, so we have very little time in general, but I want to give a flavor of what
Wittis is, and what does all this magic, and using an example of user real user, it was
a massive user who why they wanted to move from RDS to us, to Wittis, so they started scaling,
they reached the limits of their vertical scaling, you couldn't get bigger instances,
they tried manual sharding, manual sharding is hard for anyone who has tried it,
they had costs were getting really high because you are using these top to the line instances,
and they wanted more control over when to upgrade, when some of the configuration options were not
available, this is specific to one user, there are different reasons people do, just giving a flavor
of it, Wittis was important because it was my SQL compatible, horizontal sharding was one of the
key reasons they adopted it, because then they could use cheaper hardware mentioned before,
and the high availability features, another feature which is for schema change, right, when you
want a DDLs, your tables keep changing, you are developing all the time, so the time it takes
for a schema to be changed, you cannot run it directly on like a 400 terabyte database,
because it will just take hours and days, and you cannot it will bring down your system,
so you have the system like ghost and online schema change tools, so Wittis has something
online DDL which is built on top of the old orchestrator, that does the distributors schema changes,
and there is one of the reasons why this customer used switch to Wittis, so
I am now going to move to see it, sort of animation of what are the steps involved in
migrating to Wittis, this is the initial stage, right, and we will just using it, so first thing
you do is you set up a Wittis cluster, so everything in orange is a Wittis component,
normally you have like many of them, like Wittigates, okay, I will go into each of them step by step,
so XCD is the topology service that we use for coordinating between all the components
and my SQL of course, right, so initial focus will is on the external key space that's here,
so a key space is a logical database within Wittis terminology, Wittigablet is a component that
controls the database, the my SQL instance, so Wittis components never talk directly to my SQL,
they talk via Wittigablets, there are many features in Wittigablets, one which we are going to be
looking at right now is about how it does migration and replication, but it's also used for
connection pulling, query caching, a protecting against bad queries, etc, so here the database is external,
right, it's ideas, so we call it a unmanaged tablet, it doesn't bring down, when Wittigablet
comes out, it doesn't look at the health of RDS, it expects it to be up, so it's not controlling
RDS, but it's used so that the other components can talk to RDS, all it needs is a connection
string to the RDS cluster, now there are workflows within Wittis, which are launched on the
control plane, so this is a control plane demon, there are command line and Ui and GRPC
APIs available, so you start something called a move tables workflow, which moves all the tables
from RDS into Wittis, now realize that this is happening at scale and the system is online,
right, so we have users running at millions of QPS and terabytes of data, so it doesn't happen
like instantly, it takes days, sometimes hours, days for the whole thing to happen, so,
right, so we let's say the user, this is a very simple case, but I'm just talking to two
shots, just to show that it's a shattered architecture, so you tell us take a table, which has auto
increment ID, now what you do is you hash the ID and the space 64 bit space of that,
split into two and say okay, first half goes into minus 80, so 0 to minus 80, 0 to 0, and the other
is going to be other shots, so a very simple basic sharding scheme I'm explaining here, and we need
to do this when the data comes in, so the originally RDS is not sharded, so we run like sometimes
we are moving, now we are moving something to 32 shots, it's not this one, so several terabytes
are going into different 32 shots, so that's filtering is happening, okay, we'll see by how
the time, so, right, so we start the workflow and data, the app is still talking to RDS, nothing
is changed, but there are these workflows, processes, there are started on each of these
shards, which is pulling data through this and filtering out the, is asking only for those
rows that should reside on this, right, and this VT tablet makes the calls to RDS, it converts it
into internal objects called V events and they talk on GAPC, now the copy is eventually consistent,
before we, I mean of course, it has to be consistent when we switch, but initially we do what's
got a copy face, where we do bulk copies of tables, right, so we take each table, select chunks
of it using snapshots and these are all sent to the target, here you get the V events,
which are then converted into SQL and the queries are applied on the mySQL database, and at
this time there is still load, right, those rights are happening and the corrosi of copied
changed, so mySQL has something of bin logs, I'm not sure how many of you are aware of it,
but people know what bin logs are streaming, okay, so essentially it's a transaction log of all
the rights that are happening, right, that's provided by mySQL itself, so we tail the bin logs
and those are also processed along with this copy face, once you have copied all the tables,
so this happens really fast because yeah, so we copy it, then tail it and at some point you will see
that we have reached, we are very close to where the RDS clusters, so we call it the V replication
we replicate the module we use for this, so the log is very low, we can now switch, so typically
you can switch all traffic or you can say first switch just the reads and reads are specified by
you can specify at replica to a database and it will go do reads or if you are the separate
connection string for reads, you would only change that to point to bit as, this is the time
when you point to bit as, all rights are still happening into RDS, right, because there are some
I'm not going to go to into detail because of time, but there are these we call it routing rules
which are in the topology which are created when the workflow is created and so we take it
knows all the VT gets there are typically hundreds of VT gets running if you have millions of transactions,
so it knows and it only pulls does reads selects from here, DMLs are going there,
you can at any time just reworked and the replication continues or at some point you say fine
now I want to write switch rights, so rights which reads and writes are all happening from
bit as, however we also run a reverse replication scheme into RDS so all changes are
tailored again from binlocks, set back to RDS, this is a safety feature that if so if you want to
try it out something does not work, you are not provisioned your bit as in half or whatever you
just want to try so you want to switch back you can easily reward everything back and so
so that is one of the key features that gives you comfort that you have that you can move
in or out of it as and we also have tools like VDF which do rho by rho comparisons for snapshots
at a particular point in time and to do actual comparisons to tell you that it is all copied
so I think I will so when you are using everything is fine you just work flow complete and
then you can deprovision your RDS and you are fully in return at this point so this is the point
of the return so Matthias will now talk about some actual migration that we do and what changes
and what special things we need to do for larger customers. Thank you RIT. So in the example
that RIT was showing you only show one RDS instance typically that would be your writer your primary
in many cases when you have a very busy system you don't want to copy from your writer because
that takes all your rights and you want to copy from a replica. If you want to copy from a replica
what is important is that you have global transaction IDs enabled in my scale because otherwise
it is very hard to synchronize when you want to do the cover to the new instance with global
transaction IDs that is just waiting for the transaction ID that you see at the beginning
when you start the process wait until that ends up on your target system and then you can do
the switching easily. RIT also talked about fee events that are the events that are coming from
the binary logs and in order to process those into SQL you need to have the binary log
formats that you row and the binary log image that you fold so you get all the rows in the
all the fields in the the bin log event so you can easily convert. RIT also talked about the two
phases quickly the copy phase of the migration and then the catch up phase so the copy phase is
basically where it runs a long running select like select star from table and start streaming
that to your other site by default it runs for one hour and then after one hour a catch up phase
reading the binary logs is started and all the delta that were processed in the binary logs
of the row to the already copied will be applied any events that are targeting a row that you
did not copy yet can be dropped because we will copy those rows later and you will copy the state
that are in attack point in time. We do throttle the copy phase because if you have a long running
select in my SQL your history list length will start building that's what my skill needs to
guarantee the MVCC constraints and so by default if your history list length is one million
over one million we do not start a new copy phase we will finish the current copy phase but we will
not start a new one until the history list length has dropped below same thing if you're copying
from a replica if your source replica is more than 12 hours behind we will not start a new copy
phase we will wait until replication lag drops and then we will start a new copy phase.
Special case when you have a row of my SQL for Amazon as your source database there are no binary
logs on a row or a replica just no not possible to enable so what we do for that if we need to
copy from a very busy row or a cluster we cannot use the row or a replica we set up a secondary
cluster that does binary log replication from the original cluster and then you use the primary
of that cluster because that again is the only instance that has binary logs because it's
replicates the binary log from your original cluster it rides binary log itself and that is
what we then use to connect our replica tablet to the copying the good thing about the test as a
source this is something we came up with for a customer that had a own premise in their own
data center with us running and they wanted to move into the cloud and they wanted to do this
not by stopping everything on source and starting everything in the cloud because they were
doing transactions and business and stopping everything would be bad for business so we came
up with a chart by chart relations strategy important here is that you have the same
charting scheme for source and target database key space sorry because we then met them one to one
and we can then migrate them one by one you need to have the flag there enable partial key space
migrations on VT gate during the migration after migration you can leave that one out again
because that does a little bit of overhead on the VT gate level because it needs to
barge more things and it also briefly touched on routing rules for this special case
we have introduced something called chart routing rules so we can then create a routing rule
per chart like you can say like chart dash 80 like the one in the example that the road hit
showed earlier will now start going to the cloud fittest and not the own premise fittest
but the other chart will still go to the own premise fittest and we executed this recently
with a customer up to 256 that's not a hard limit that's just a number of charts that they were
running at that point in time after you do the chart by chart migration it's not just the complete
command like normally you would complete and it would clean up everything in this case of chart
by chart you have to do so many opinions afterwards just to be aware there is documentation for this
also on the VTests documentation website so if you want to read more definitely go there
and then another type of migration that we are currently working on is a multi-talent what
do I mean with multi-talent that is when you have a set of tables per customer of yours
per tenant like if you have each of your customers being a tenant in your environment
they have a set of tables and then your business grows and you start off with maybe a thousand
customers or thousand tenants let's say you have a hundred tables so you have a hundred thousand
tables if you go to a hundred thousand customers you have ten million tables in your database
which is not so great for my skill so we were facing facing this and then I was talking to
Rohit and we came up with this multi-talent migration so what we do to each of the tables we add a
unique identifier identifying the tenant like can be an ID like customer ID starting at one
counting upwards and then all your all your queries should include that where
condition where your customer ID or whatever equals the the ID of what you do and then in your
Fiskima which is what Fitas use uses to identify where the chart are
you add an identification the multi-talent spec here and then you say this
fields like your tenant ID customer ID and this is the type of the field and then to your
move tables command you add a tenant ID value when you create it and at that point in time all
your V replication queries will have an extra wear condition so it's always filters on that tenant ID
so you can then again do the cutting forward and backwards per tenant without needing to do the
one big bang for everything and this can merge hundreds or thousands or hundreds of thousands of
schemas into one chart of TASKee space and for this we also introduce a special kind of routing rules
because if you would have normally you would have routing rules per table and that is going to
expose very quickly if you have a lot of tenants and a lot of tables per tenant so we basically
added something called key space routing rules that we can route your queries per tenant
back and forth into the Vitas yeah and then what you best do in this case is chart your Vitas
cluster on that same tenant ID field so then you can spread out your different tenants across
multiple charts and not overload one database server again some members we ran thousands of
your replication imports the ones that are that Roy talked about earlier in Salserv mode so
basically if you sign up for Plan scale and you want to import data that's a V replication stream
is what happens under the hood and examples we have imported for customer in Salserv mode
64 terabytes into 16 charts without much issues and we do multiples of multiple millions of
QPSs across multiple Vitas clusters at Plan scale just last weekend we were working on a
multi terabyte data set copy from a short from a Aurora cluster into a shortest Vitas and we
observed speeds in more than one billion rows per hour in copying those databases so we copied the
entire data set basically over one weekend which was pretty cool the multi tenant migration that I
was talking about were currently executing with a customer that has hundreds of thousands of tenants
on multiple Aurora clusters and total size there is about 250 terabytes which we are migrating into
128 charts currently and we're basically going from nine million tables per Aurora source to 90
to a shortest Vitas so imagine the online DDL you would need to do to manage nine million tables per
cluster now you just need to manage 90 tables in the shortest Vitas and Vitas takes care of the
entire online DDL and then you have some case studies here the Shardbush Hector migration
we posted a case study of the cash app migration for square they did over 400 charts in total
Shardbush hard successfully without downtime and additional case studies are also available on the
website and that was any questions no time for questions unfortunately
all right so Josh are you I don't know what we're Josh