You
Yeah, it's full 4 p.m.
So let's start with the next talk.
Let's go.
Awesome, thank you for the last 4 years for the last 4 years.
I'm responsible for running our POSPASPASPIL infrastructure, which is supporting GitLab.com.
The 8 years before that, I worked at the open source, and especially POSPASPIL consulting company.
Today, I would like to talk to you about how we execute POSPASPIL and operation system upgrades at GitLab with zero downtime.
And I would like to do that by answering a few questions.
We start with POSPASPIL upgrades.
How do they work and why are they hard?
We talk about OS upgrades, what's the deal there, and then I would show you how we minimized the impact for our users.
First of all, POSPASPIL upgrades are major upgrades to be major upgrades.
Why are they hard?
If you have a status service like an application server, you might get away with just changing the running process.
For example, POSPASPIL, a pooling agent, you can just switch out the running process without any interference.
Or if you have basically any application server, you just start up new application servers, paid all the old ones,
and you can easily upgrade without having any impact of your users.
But if you have a service that comes with a state, it's much, much harder,
and especially for complex relational databases like POSPASPIL,
when you do a major upgrade, you might not just change the process, not just the code that is running,
you might need to change the data that is store.
And if you have a large data base, you have to maybe a lot of data to change, or at least to go through.
And for POSPASPIL, a major upgrade, it often occurs that the control structure, the structure,
how the data is stored on this is optimized and get changed.
So you have to rewrite your state, and that's not something that happens instantly.
So, the recommended default method to upgrade the POSPASPIL database,
which is still the default, and if that works for you, you should definitely do it.
You start out with your database.
Your data is stored in a binary format on this, that might depend on the libraries to use on your system.
It depends on your architecture, so the data will look different on an x86 machine, then on a risk.
For example, an Indian is, so this data cannot be easily transferred to a completely different architecture,
and you cannot use it with a different major version of POSPASPIL.
So the default method is you export this data into a logical format that is independent of your architecture and your version.
For example, SPL will do the trick, but there's also an internal format that is slightly more optimal.
So you start out by exporting your data, you go from the binary stored data to a logical representation,
and then on your new server, you just import the data and it gets transferred again to the binary data on your system,
and afterwards you have to recreate all helping structures.
That's mostly indexes, but also statistics and so on.
This is the safest method because you can be sure that all the data is possible,
because it was exported once and then passed again.
There are for many data types, you have special validation, for example, if you have JSON stored in your database,
it gets passed again, so if anything is fishy there, you will notice that.
And also all your helping structures are fresh, you get like square fresh indexes, no blood.
But the problem is if you have a large database that clicks, takes quite some time.
Our main database currently has like 40 terabytes of data, and for us this operation would take multiple days.
If it works for your database, please do that.
You can listen to the rest of my talk, but if that works, please do it.
The next method is still fairly safe, but you have to think about a few things, and that's PG upgrade.
PG upgrade is a tool that gets released with every post-pascale release, and it knows the current data structure,
and it knows all previous data structures, so it will know which parts have to change.
It goes through all your heap data, or your binary data on disk, and changes all the parts that need to be changed.
Quite simple, it's reasonable, fast, it's reasonable to save.
You have to think of a few things, for example, the helping structures like indexes will not be updated.
So if the new post-pascale version comes with an optimized version of B3 indexes, and you want to profit from it,
you have yourself, you have to restore, recreate the indexes.
Or if you would like to go to a new operating system or different architecture, you cannot use this method safely, so keep that in mind.
But still, if this fulfills your need, awesome, use this method, and don't look deeper into the next thing.
So why can't we use this, why can't we use these methods for GitLab?
We have actually done that before, so when I joined GitLab, we would use PG upgrade.
But we had the business requirement before going to a new post-pascale version to make sure that it's operational,
that all our data is correct, and all that the application is not, that we don't have a performance degradation,
and so we needed to run significant number of tests, and the overall downtime for an upgrade was like four to six hours.
So that was for us, not really feasible, and basically just to give you a little insight on our usage profile,
we have over 50 million users around the world, basically in all time zones, and even our team,
we have like over two and a half thousand people who all use the platform for their daily work.
So the requirement we call was basically, we can't afford any downtime, so we had no budget for downtime,
and also the requirement is that after we upgrade any component, if we then realize that we get a strong performance degradation,
we have to be able to roll back.
And many situations, if you do an upgrade, and you have a performance degradation, you have other means of optimizing,
for example, you can buy the next larger cloud instance, if you're running on a hyperscaler, or you can buy the next larger piece of hardware,
because we are basically running on quite the largest machines already, that's not an option.
So if we would experience a performance degradation, we need to be able to roll back quite swiftly.
So yeah, I mentioned zero downtime, but what is zero downtime?
Because we are talking about the software as a service application, and nothing, there's no instant reaction.
So if you press the button, like creating issue, the issue is not there instantly.
It will take like 100 milliseconds, 200 milliseconds depending on where you are.
So we cannot aim for zero, that's not possible, so we need a method to decide, like, we need a metric to decide what is zero downtime.
And what we do is we define that we don't want any noticeable user impact.
And the good thing here is we already have a metric for user, I had a metric for user impact already,
and that is called AppDex, the application performance index.
We do samples for different user interactions.
So for example, for creating an issue, or running a CI job, and we define what is satisfying for the user.
Like, you click the button, new issue, and after a few hundred milliseconds, the issue pops up.
It feels snappy, that's like feeling satisfying.
If you're clicking on create new issue, and you have to wait maybe one or two seconds, that does not feel too great.
You're looking at the blank screen, or the progress indication, that does not feel great, but you tolerate it.
If the action takes significantly longer, the user might get annoyed and press F5, try again, and that's a frustrating experience.
So for a lot of actions within our application, we define these thresholds, and then we take continuously, we take samples for that.
And calculate the AppDex.
So for example, here you see my laser pointer is not really visible.
But you see on the top line, there's satisfied count, so if one hundred percent of all requests would be satisfied.
If that is factory, then we have like one hundred samples, one hundred samples divided by one hundred is one.
So if everything is perfect, we get one.
And if all samples would be frustrating, it's multiplied by zero, so we get a zero, we divide it by whatever we get a zero.
So it's scaled from one to zero, zero is nobody satisfied, everyone is frustrated, and one would be everyone satisfied.
That's the metric we have in already in place, and we page page for this metric.
So if it goes down below 98 point something percent, people get page.
Yeah, how do we achieve that?
And there's a really cool method in post-pressure called logical replication.
So basically the thing we saw at the beginning, where you transform the physically stored data into logical replication,
into logical representation, can be used in a replicating format.
And we use quite some automation to make it actually feasible.
So what is logical replication?
Yeah, or what does it give us?
Unlike extreme replication, logical replication, because it uses the logical format,
can replicate between different major post-pressure versions, and even different infrastructures, different architectures.
So we can just clone our current production, upgrade it, bring it in sync again with the main production system, and then switch over later.
One thing, doesn't come with restriction?
Yeah, it comes with quite some restrictions.
In the previous talk, when I talked about the trivial method, we used modular method, we used before.
I go into more detail of the restrictions.
You can watch a recording if you like.
But for the scope of this talk, I will only go to the main restriction, which is during the logical replication,
while it's enabled, we can't use any DDL.
DDL is data definition language, so create table, author table, drop table.
That's more possible, unfortunately.
For GitLab, we solved that in a way that we have two features for that.
One is a process feature that we block our delivery colleagues from deploying new GitLab versions that would alter the data.
And they also get one week ahead.
We're running, hey, next week or not allowed to change the database.
And also we have a feature flag that you can use to block DDL from happening.
That includes the deployments.
But also we have some background work, for example, that do partitioning.
We have really large multi-terabyte tables that get partitioned in the background.
All these jobs are frozen during the time period.
If you are, if you want to do a zero downtime upgrade for GitLab, you can use the same feature flag.
If you use any other software, you might not have the problem in the first place,
because most standards software does not a lot of DDL in the background.
But you definitely have to check for your concrete application beforehand.
So I work you through a simplified simplified version of the process.
First we start out with our current version of PostgresQL database in this case, PostgresQL 16,
and our application talking to it.
Then we create a one-to-one copy.
In our case, we just create new virtual machines from snapshots of the currently running ones.
And we use the standard method of streaming replication to get the second instance synchronized.
There's a good thing, PostgresQL by design, writes all of its data changes into something called the write-a-headlock.
In the binary format, and you can just stream that to a different server to keep it on the same state as the source.
So our starting system is called source, and the system we are replicating to is called target here.
Then we can stop the replication and upgrade our target system.
So we would run the program you saw before, PG upgrade.
We upgrade the data on this.
We can recreate the indexes.
And afterwards we use logical replication to sync it again.
So now we have the old database version running and new cluster with the new version,
and we get the same data as I am.
And yeah, once we are satisfied, we can just switch the application from connecting from the old one to the new one.
And that's basically the state we had in end of 2023.
And now to give a small look into the actual user impact.
This is our app text.
And yeah, to give you a good static view that's basically the nitpicking view.
I'm looking at the top one percent.
So we are seeing from 0.99 to 1.0 app text.
So here is our degradation as well.
98.8 percent.
So when people get patched, this line is not invisible in this graph.
It's a little bit below.
And yeah, you see that's one week of data.
And you see that it's not constantly at the same level.
We sometimes have degradation.
It can be the application database.
Or maybe a server is 14.
We need to start a new instance or we have more load.
We start more pods, things like that.
And this window here was one of our switch overs.
So you can see the impact on our users is basically measurable.
So during the switch over, we have a short degradation window.
But it's less than the normal noise.
So it's measurable, but it's not significant.
So yeah, we were able to do post-precipeal upgrades with basically zero-down-time zero-user impacts.
But that's old news.
So today, the focus is what did we do for improvement since then?
Or what can we improve further?
And we have two main things here.
One is more of a business requirement.
And that was that the switch over.
When we are switching over from the new version to the new version.
This was a point of no return.
We couldn't switch back.
And in the past, we had the problems that after switch over.
We had a performance degradation.
And the requirement here is to remove this uncertainty.
This risk.
So we need to change the process in order to be able to roll back even after this switch over.
And the second thing is something that I really want.
Because this whole process only upgrades, post-precipeal.
Not any libraries, not the operating system.
So I would like to combine both to reduce the labor involved.
Okay, how do we remove or move the point of no return to a later state?
And that's relatively easy.
All the technology we need is already in place.
Because after the switch over, we can reverse the replication and stream all the data back to the old
old clusters.
So the old clusters kept in swing.
And if we come to the conclusions that we can't fix the new cluster, we can roll back.
In reality, we would put a lot of people into optimizing the queries and making a new cluster perform.
But if that's no longer possible, we have an option left.
So after the upgrade, we would leave the old cluster alive and replicate by a streaming
via a logical replication in the old cluster.
And we can operate in monitor.
And if we come to the conclusion that we really have to roll back, we can switch back to the old one
without losing any data.
So now to the OS upgrade.
Why OS upgrades a problem?
Why can't I just create a new server with a new Linux version of my favorite distribution
and just switch over to this version?
There can be multiple problems with libraries, but the major thing here is something called
Collation.
Like new operating systems, system versions normally come with a new C library for most
it's the GLIPC and the GLIPC provides something called the system wide collation.
And that's the sort or how you order strings.
And on first glance, it should be super easy.
Like if you want to order a single character, like an ABC, it's obvious you have L, first the A, then the B, then the C.
It becomes a much more problematic if you have lower case in upper case.
If you want to order a list of lower and upper case letters, do you want to have first A lower case
or do you want to order ABCD until the end and then start with the upper case letters?
But it's more complicated if you have, for example, strings of numbers, what should come first?
0, 1, or 2, or with special characters.
And unfortunately, there's not one perfect collation that never changes.
Unfortunately, the collation changes regularly.
So when you get a new operating system, you have to accept that your strings will be sorted differently
than before.
And if you have a stateless application, that's not a big deal.
But if you have, like indexes that were created with one collation and all your data was ordered after one ordering pattern.
And then you switch to a new operating system and it starts to order stuff otherwise.
Then you have the problem that you can't find values that already in your index.
If you use the index for search, that's already annoying because you can't find the stuff you already have.
But if you use the index to, for example, and force a constraint, like a unique constraint, you are now able to put tablets in your database, even so they should be unique.
And that will break your data and corrupt your data.
Yeah, how do we solve this problem?
The first thing is, it's not applying for all indexes.
For example, there are some simple data types where you don't have a special collation like an integer.
It's quite easy to sort an integer you can do it in a like a decimal representation or binary.
There's not much to it.
But basically, the simple solution is to rebuild all of your indexes to complex or collation based data structures, mostly strings.
So the good thing is if you do the upgrade, which I suggested in the beginning, the dumpry store upgrade basically, you don't have the problem.
All of the indexes will be rebuilt anyways.
But if you use any of the other upgrade methods, you have to do it yourself manually.
And if you would just rebuild all indexes for GitLab, it will also take multiple days, so that's not feasible for us.
So what do we do to make it feasible?
Along with for the upgrade, we create a new system, we do a test upgrade with a production copy, and then we use a postgresical internal functions called AM check.
And we check which indexes would be corrupted if we upgrade.
Then we take, make a list out of all this indexes and recreate them.
Should it be fairly easy operation and we can just recreate them, also we just take the list and start with our upgrade process.
If the recreation of all of those would take too long, we have to optimize.
You can, for example, try to find if some of these indexes might not be needed at all, or you can try to use different index type, or maybe you can break down an index, maybe a one large index for the whole table.
You can do a lot of partial, partial indexes that are only indexing certain, certain areas, certain, certain spans of a table, things like that.
And also you can do something like lazy recreate. For example, if you come to conclusion, we have really large index, and it's used for to optimize full text search on all of your issues.
And the negative effect, if this index would slightly corrupt would be that.
In a totally weird event, somebody couldn't find an issue that has all the German umlauter in it, and yeah, you can decide, okay, that's not super critical.
We are fine with not creating this index during the upgrade window. We are fine if it takes to the next Monday morning, because we will have low data corruption.
We only will have a slide function degradation here. So if you're looking into that beforehand, you can just make sensible decisions.
So to give you a perspective how you normally do it, we do all these upgrades on the weekend, because that's the lowest load.
So face for us, still not no load unfortunately, but the lowest.
Yeah, and we would do the on Saturday morning, we would start to all the steps required for an upgrade.
And then we have still until Sunday to do additional maintenance operations like index with creations.
So depending on how long the first steps take, we have at least 12 to 24 hours for that.
So we will recreate all indexes, and afterwards we run this internal function A amp check again to make sure that we really have no corruption.
And we mostly have most of the time we also have time for additional running additional sanity checks.
Cool, and now I would like to work through the full process how we did last year with all the improvements.
We are going from possibly 16 to 17 and from Ubuntu 20 to 2022 to 2022 or 4.
Okay, we start out with our database again and the application running to it.
And because it's oversimplified, I give you a little glimpse what's behind the symbols.
So the GitLab acting on the left, that's our application stack, most of it runs on Kubernetes, except our radius.
And the database for this case, it's nine large instances, and one smaller one for taking snapshots.
And we have distributed them across three availability zones.
And we start out with 20 or four and possibly 16. On the top right, I have like a traffic light icon to show you if at the current phase data definition language is allowed.
And for us, it's quite important because multiple times a week, my colleagues deploy new versions and sometimes they want to change to schema.
So one of my requirements was to keep the phase where they can't execute the DL to the minimal minimum.
So the first step is we created test cluster to get all our metrics to know what we are dealing with.
The test cluster can be minimal in our case, it's then like at least three nodes.
Also, yeah, that's not affecting production at all. It's already starting out with the newer S version, the new prosperous version.
And then we do a mock update, upgrade. So we upgrade our test cluster.
We get all the correct execution times, we need to know exactly how long each step takes.
And also we get a list of all the correct indexes because we run the AM check tooling.
If we have all metrics that we need, we can schedule our upgrade, we remove the test cluster and we create the actual target cluster.
In our case, again, it's nine large nodes, over three availability zones and one backup node.
And now comes the step where we have to disable the DL.
Because now we have to switch from streaming replication, like where we just sent the actual byte data from the source to the target cluster.
Now the source cluster has to translate this binary data into a little logical representation and have sent it to target.
And DDL would break the process. So now begins to face where my colleagues are no longer allowed to deploy new schema changes.
That's by the way, that's Saturday morning. In this case.
Now we stop the replication, we upgrade our target cluster, so we run KG upgrade runs like 2020 something minutes, I guess.
Then we re-synchronize again, we have the same state, we get fresh data again.
And then we can do all the additional steps, so we can do a full re-index to make sure we don't have the corrupted indexes.
Afterwards, we can run Analyze, Analyze goes through all your data and creates statistics, which are possible.
It's super important to have fresh statistics, it's a possible node.
Can I do an index? Can I have to read the full table?
Or if you have partition tables, it knows, oh for this query, I have to look into these petitions.
And also, yeah, we run the corruption check to make sure that our index mitigations were successful.
And now we can start with the switch over. The first thing is, we will, the application talks, in this step, the application talks to the source cluster to the PG-161.
And then we gradually start to load balance, read only queries to the new cluster.
So we start with one replica, so all of the queries get scattered across all of the standby in the old cluster, the source cluster,
and one standby of the target cluster gets in the load balance.
And that's really cool because we have a nice dashboard and we can see the performance.
We can make an applet to Apple's comparison.
How much we make sure all of the standby get roughly the same number of connections.
And we can Apple to Apple compare the performance.
We can see the new standby does it have more CPU load, does it do more IOPS, things like that.
In general, if we are going to a new possible version performance increases, but sometimes not because query plan flips.
And then we can have hours and hours of time without any impact that we can optimize the queries.
So during the process, we have like the back and developers on call and call them in if you find the performance problem.
And we can optimize the queries long before we do the full switch over.
Yeah, if we are satisfied, we get all of the read load to the target cluster.
So all of the rides still happen to source cluster, and then they propagate of a logical replication to the target cluster.
And all the read load goes to the target cluster already.
And yeah, we can monitor the performance.
If it looks great, we run our full end to end.
And they are quite quite significant.
It basically uses all of the features that Hitler has.
It creates issues, it does, it does, it does.
Yeah, basically calls calls a lot of the functions, also writeable.
And also measures not only that it works, but also the performance.
And that's really tricky because this construct here, all the data is written to the source cluster and then replicated to target cluster.
It's improved, this increases the latency.
So we even needed to make our QA test a bit more resilient so that works still works.
But now that's a fairly fairly great method to test the complete functionality.
And also to get inside of the performance because we again can compare.
And if you are satisfied with the performance, all tests are successful.
We can do the full switch over.
Now we also, we break the, now we break the replication from source to target.
And we move all of the, we load balance all of the right queries to the target cluster as well.
And this part basically was for two years ago, we also had that.
But now comes the important upgrade.
Now we don't have to, to pray because we still have a way to roll back should perform.
And we keep that running for the full of Monday because during Sunday, as mentioned, we don't have too much load.
So if there would be edge cases that only happen during peak hours, we might not find them on Sunday.
So we still, on Monday when the load starts, we first see a peak on Europe work hours.
So when European start work, we see the load rising.
And then we see when the US east coast starts to work. And then we see when the US west coast start working.
And this is, this is the critical phase. Like we have to use west coast starts to work.
And still people in the east coast works, people in Europe still works.
That's normally our peak hours a day.
And there we have to really have to monitor that everything is optimal or we have to on the fly optimized.
Yeah, and we, we run that for the full of Monday.
Still logical replication is needed for that. So we can keep the old cluster in sync.
And Tuesday morning, Europe time, if everything worked as expected, we stopped the replication, we removed the old cluster.
And now we are fully on 17.
And because we stopped the logical replication, DDL is possible again.
Also my rush quite through that.
So I, I really hope you have some questions for me.
If you, if you want to deep dive into that, we have here an overview about our database infrastructure.
Underneath, I didn't mention that before, all of the actions you have seen are not executed manually.
We have everything in Ansible Playbox. We do that multiple times a year.
It would be super tedious to do it manually.
And it would introduce new human error every time.
So every single incident ends up with playbooks linked to the repo you can see there.
And also for us in a fairly large company from my point of view, I have to coordinate this with a lot of people.
I have to coordinate this with a lot of people. I have to coordinate with delivery.
Some of our larger customers would like to be informed of such things.
And so I have to coordinate with a lot of a lot of different different stakeholders in the company.
And also I have to set up schedules for on call.
And we at GitLab, we use GitLab to organize ourselves.
So we use GitLab issues and it affects to organize such upgrades.
And I have, we have a large template, an issue template.
So when we do a new upgrade, we create a new issue based on this template.
And it has all the checklists to inform what's much requests to create in order to change the roles.
And basically all the steps are in there.
This thing is fairly large and it will most likely not work for your organization is.
But if you have to organize such an upgrade would be really interesting for you to go through it.
Also, I oversimplified a few steps to press it in the time.
If you are more interested in the actual, for example, in the actual caveats of logical replication,
there are a few things about why sequences are a problem I have for recording there.
Yeah, slide text are already updated, upload it.
And I have two versions.
One is the one you are seeing right now.
And one is an extended version with roughly doubled number of slides.
We have additional explanations for you.
Yeah.
And now I really hope you have questions.
You can approach the during the event.
I'm sometimes at the post-presquail booth.
Sometimes at the GitLab booth or running around.
You can also write me something or you can ask right now.
Because apparently we have some time for that.
Yes.
Thank you.
Who is first?
Somewhere right there.
Okay.
I have the following question.
Like you said you are going to 17.
Basically you are staying in one version behind the tip of progress as far as I understand.
But during those periods when you do the upgrade.
Do you also try to see whether you can upgrade to the,
like to the newest version.
Not in order to upgrade just to see whether you will have any problems with that.
Okay.
I summarized the question.
You mentioned that we are going to post-presquail 17,
which is one version behind the current stable.
And you ask me if you have considered going to the latest version.
Okay. Awesome.
Yeah.
In general, we have this policies that we want to go to a post-presquail version before that.
That we want to go to a post-presquail version before the next version comes out.
That's the ideal thing.
But not, but we would not want to go to version before it's older than like half a year.
I really like every new feature.
But I don't have the capacity to find all the cool new bugs in production.
So I'm really happy if somebody else finds them first.
So we have a certain mandatory delay before we go to the new version.
For post-presquail 18, it's a bit different.
Because in post-presquail 18, there are optimizations that we would really love to have.
We have a problem called lightweight log contention.
So post-presquail has explicit logs, which are used in queries.
You can say log table or something.
But there's also an internal construct called lightweight logs.
And that's biting us and our peak hours sometimes.
And post-presquail 18 comes with some optimizations.
So I would really like to go to 18.
So we might make an out-of-band upgrade and basically right now start planning for that.
Awesome.
Thank you for the question.
Hi.
Did I understand correctly that the period that you cannot perform DDL is three days?
And if so, is that not a problem for your change processes?
I couldn't copy a story.
Okay.
Did I understand correctly that the period in which you cannot perform DDL is three days?
And if so, is that not a problem for your change processes?
Yes.
That's correct.
The time frame we can perform DDL is from Saturday to Tuesday morning.
That's the phase.
That's something we choose.
We could make a charter.
But then we don't have a rollback option anymore.
Quick rollback.
Not only before, we only headed to Sunday, Sunday evening.
But we couldn't rollback on Monday.
So for us to trade off was we want to have a larger rollback window.
And therefore we are fine with having the DDL log for longer.
And it's not a large problem.
There are basically two things that can't happen.
We can't deploy new GitLab versions that would change the schema.
And not every GitLab minor update changes schema.
So that's not a huge problem.
And also we can't do background re-indexing.
Which is also not a problem.
We just pause it for the time.
And we can't in the same category.
We can't do background partitioning.
Because we have fairly large tables.
We've been petitions in the background create new new sub tables.
And that's also not a problem.
We pre-create them for like a month or so.
So we could, for this case, we could put DDL of four months.
Or was that any problem?
Awesome.
Thank you for the question.
I have a question related to the rebuilding of the indexes.
And the corruption you mentioned that is maybe possible after the DDLBC upgrade.
Do actually, did it happen to you?
Do you have that experience?
Or is it purely a political?
And in your steps you had a great, you upgraded the database first.
Then enable logical replication and then rebuild the indexes.
So I wonder whether this data corruption could happen.
Whether the audition would be different.
Like first building indexes and then enabling logical replication.
Okay. Awesome.
First question was regarding index corruption.
Did we had index corruption?
And the answer is doing our tests.
Yes.
For the current upgrade going from Ubuntu 20 to 2022,
they are like 20 indexes corrupted.
But it never materialized as a problem because we found out
during our test upgrade and we never went to production with that.
We made the list of indexes.
We optimized and we recreate all of them before we go to production.
And the second question was regarding the sequence of events.
So here when we start out when we create a test cluster,
that's streaming replication because it's the most efficient one.
And then we break it.
We stop then we...
Or that's test cluster.
It's irrelevant.
Okay, here.
We start...
Okay, here.
That starts out.
We create the test cluster.
No.
We create the target cluster.
That starts out with streaming replication because it's just most efficient.
And then we switch it to logical replication.
During here, I mean already here,
the indexes on the target cluster are corrupted.
You couldn't send the live traffic to this cluster because you would get wrong answers.
You couldn't switch over to that.
But it does not matter because we have it in production,
but we don't.
The application does not talk to it.
So at the current state it's corrupted,
but the application does not talk to it.
Answer that.
Is that an answer to your question?
Awesome.
Thank you.
No.
Can you hear me?
So you mentioned briefly that you used the right-to-head log
of post-blastware for managing logical replication.
How do you use any additional tool to handle that replication?
And also managing the NN plus 1 server node use.
Perhaps we're ending a failover.
Just the right-to-head log by itself.
The question is, we use the right-to-head log for logical replication?
Yeah.
Do you use any additional tools?
No, there's something like PG-bound server, for instance.
Okay.
First of all, that misconception, we're not using the right-to-head log for logical replication.
The right-to-head log basically contains the data for the person that I would like one to answer that.
But you have an additional question.
Okay.
Awesome.
Okay.
The right-to-head log is not used for logical replication directly.
For the replication itself, we use the post-blastware build in the future of logical replication.
But we have a lot of different tooling.
For example, we have connection tooling.
We use PG-bound server for connection tooling.
And we have a fleet of PG-bound server because it's CPU bound.
And for management of our high availability, we use a patrony.
And patrony is a post-blastware high availability toolkit.
And it's also controlling the PG-bound server.
There's a centralized single source of trust in our case console.
And if we do the switch over, we basically just say, hey, patrony, switch over to this source.
And patrony tells PG-bound server, hey, post-connections here, and redirects them to the other ones.
Awesome.
If you want to take a look at our Ansible, you see where we integrate, where we hook into the systems.
Awesome.
Thank you.
Do we have a mic here?
Hi.
Oh, you have new people.
Sorry.
Thank you for the talk.
I was really surprised to hear about that collation issue.
I hadn't heard of that before.
I wondered if postgres had ever considered dealing with collation itself somehow
to make this whole upgrade process easier.
Yeah.
Is that ever been considered or?
Yeah, definitely.
First of all, I'm quite lucky that I know what a collation issue for quite a long time.
Because some years ago, I worked for Postgres with Consulting Company and the collation for the German umlauded.
Something like, you changed before, so we had a lot of German customers who had this problem like many years ago.
And now it's biting more people because the collation for different characters was changed.
And yeah, there are new and your Postgres version, there are different methods.
Back in the day, you were bound to use the system-wide collation.
But now you can use different collations.
So there are, I see you.
There are other collations.
They can say, hey, Postgres deal.
Ignore the system collation used this one.
But it also comes with caveats to sum it up in one sentence.
Okay.
Yeah.
Thank you.
The default is that it uses the collation of your G-lipsy library, your system-sy library.
And that's where the problem comes from.
Because if you're making an operating system upgrade, anything changes to G-lipsy,
you are about to find interesting behavior.
Before the switchover, how did you distinguish the read and the write queries?
I did put it a bit closer to the most before the switchover.
Before the switchover, you had the read and write queries.
How did you distinguish which or read and write queries?
Awesome question.
As before the switchover, we first switch over the write queries and then the read queries.
How did we do it?
And that's, unfortunately, for most of you, that's built in G-lip feature.
So in G-lip the application in our Ruby on Redspack and it knows that certain queries
will be read and write and certain queries will only read.
So it connects to two pools, one read, read, write pool, which only talks to the primary.
One read pool where we have a lot of stand-by and the Ruby application talks to all the stand-by.
And the Ruby on Reds decides where to send it.
Okay.
Hi.
Here.
So can you explain a bit more how you the upgrade worked?
Actually, the distribution upgrade, I didn't quite catch that.
For the test, you said you cloned it.
So I guess, do you have compute and storage separate?
So you could just cloned with the 22 or did you do a distribution upgrade?
Or did you just provision empty VMs and then re-cloned or re-sync everything?
Okay.
How do we get to the target cluster?
Okay. We create new virtual machines with Ubuntu 2022.
And we take snapshots from production and we create new machines from snapshots in production.
Storage snapshots, sorry.
So we run on GCP and we take a storage snapshot and then create machines from the snapshots,
which with our currently 48-tera by disks that takes like half an hour,
but it's way faster than restoring a PG-based backup.
I have two questions.
Yeah.
One is how much effort you put into building this automation?
The first iteration of it was more than half a year,
and now we are working for it since 2022 or something.
So we iterate over years over it.
So it's a lot of work.
Because if you make the wrong, there are certain error cases here that brings down time.
And there are error cases that would bring data corruption.
So you have to test it really thoroughly.
And if we do test on our production, it takes a multi-day test.
So yeah, there's a lot of effort into that.
And so it's years, how big of a team do you have?
Currently, it's five people.
And from the majority of our work last year, went into that.
But we have some fluctuation in the team,
so if it would have been a standing team,
maybe we would have had a bit more headroom for other things.
But it's multiple people per year,
if you have a really large system like we have.
What the end of the second question is, would this be reusable outside?
Yes.
So I presented a very oversimplified architecture.
And this architecture you can use for a lot of different use cases.
Our automation is made for us.
It's open source.
You can clone the repo.
You can look into it.
But you most likely would not be able to use our ends of the playbooks one to one.
But the concept for sure.
But to repeat that, if you don't need zero downtime,
if you can take an hour or so of downtime,
I would optimize for one of the simpler approaches
to optimize it to get minimal downtime.
And only go to this endeavor if you have a hard requirement for like zero.
Thank you.
Here we go.
I've got a question over here.
All right.
So before you've mentioned that,
most of the services of GitLab are run on Kubernetes,
I've already used the way to the site.
And I've wanted to ask what exactly red is.
It's useful.
It's mostly for cache and job processing or something.
Why are there use cases on red is?
Okay.
First this play, my focus is postgres VR.
So for my understanding, we use red is for application caching.
So Ruby on red's catcher stuff and red is.
So it doesn't have to create a database for something,
it just asks it.
And for full disclosure,
I didn't thought too much about Githli, Githli is our back end for storing actually
Githli data.
That's also not in Kubernetes at the moment.
It can run.
We have some use cases for that, but for Githli.com, it still runs on virtual machines as well.
Okay, last question here.
What is the procedure for minor version upgrade?
Is it the same?
No, for minor versions upgrade, minor version upgrades.
You don't have the data does not change.
So, for minor version upgrades, we just create new stand-byes,
put the new minor version, put them into the load balancer,
and then fade out the old ones.
And if only the primary is left for only the old version,
we have to do a switch over, which as we have seen,
we can do without noticeable user impact,
then we switch over and decommission the primary.
So, it's a bit tedious because we have to create all these new nodes,
put them in the load balancer, put the old ones out,
but it's something we do just during normal working hours without
a fraction of this preparation effort here.
We've got one more question here, time for it.
Hi, thanks for the talk.
A small question.
I assume that GCP has one more or another of the managed postgres.
What's your considerations for not using that?
Is it okay?
Okay, the question is GCP itself offering for managed postgres here.
Okay, I hope I don't step on somebody's toes here,
but the GCP offering is like an 80% offering,
like for to catch like a lot of people who want to have
just a postgres skill instance,
but for our scale, it's unfortunately not possible.
I guess that's good sum up.
Thank you.