Okay, everybody, we're ready to do it.
We're ready to do it.
And stuff, things.
Yes, and things.
Hello again, much longer talk this time, and only me, so I don't
doesn't get to help this time.
So I'm going to talk about a new piece of open source software.
It was published by the power DNS team last year.
I think you came out in 24, right?
Or did it come out in 23?
Anyway, very recently, which is another way to do
zone replication between authoritative servers, which we all need
to do for various reasons, and we'll talk about that.
So first thing is, we're going to do a little bit of primer here.
I know it's late in the day and you're all falling asleep, so a lot of this
will be old news for many of you, but there's three things that
authoritative server manages.
Data wise, there's the actual content of the zones.
The record sets that it needs to manage.
There's the list of zones for which it's willing to provide
authoritative answers.
And then, at least in modern authoritative servers, there's
metadata about the zones.
How the authoritative server should treat that zone?
So, for example, if it's doing online DNS sex signing, or if it
needs to know where to send notifies two or other things,
those are things which are not stored in the zone itself.
They're stored in metadata alongside the zone.
So there's three sets of things that your authoritative server typically
needs to have to be able to provide the services to provide.
And then, this seems like a silly question, but it's true.
Why do we need to replicate this stuff?
Why do we need to have multiple servers?
Well, there's obvious things, like fault tolerance, right?
If you had just one server, providing answers for your zone,
and that server becomes unavailable for a variety of reasons
and anyone trying to get it results for any of your zones is going to fail,
because your server is not available.
So you obviously want to have more than one for that purpose.
And I have three BGP in there because we all know that when it's not DNS
to each P, so the two reasons why things are broken.
Then, there's the other situation, which is, you know,
the internet's big goes around the whole world and, I guess,
out into space a little bit.
If you just have one server located in, I don't know,
let's say, Western United States, and you've got users in South Africa,
looking for answers to DNS queries for your zone.
That's going to be very slow, because they're around trip time
to your server's long, because of speed of light and other things like that.
So you want to have more servers spread around the world,
so they can get answers more quickly.
And the last thing is, if you happen to be the owner of an extremely popular zone
that's getting thousands and tens of thousands and hundreds of thousands of queries all the time,
having just one server of course would be terrible,
because the server would get overloaded and possibly crash and all those things.
So these are all reasons why people have needed to be able to do replication.
Now, going back to the beginning, and by the beginning, I mean,
way back in the past, and there's lots of people in the room who've never had to do this stuff,
because they're not old enough.
The original authoritative servers, literally, you put a zone file,
an actual zone file, which lots of people still use with bind,
but this is existed forever.
Into a directory, and you pointed the authoritative server
that directory in an answered queries with data from that file, and that was it.
That's all it did. There was nothing else.
There was no metadata. There was no replication.
There was nothing. It was just, hey, here's this data,
and I need you to answer queries for it.
That's way back when.
And the way that you replicated changes to that was you're
self, FTP, or SCP, if you actually had were an early adopter of
SCP back then, or any of these other ways, that's how you would move those things around.
So you'd copy the files to your servers, and then go restart the DNS
demon on all of your machines, and now suddenly they had the current copy of your zone.
We wouldn't want to do that now, because that's a very error prone and slow and lots of other things.
So some wonderful people published some RFCs back in, where did I say that was?
1987. So still approaching 40 years ago.
It's hard to believe that.
So what I said, hey, we should just be able to do this with DNS, right?
The DNS serve the heart to your server itself.
Should be able to send the data that's needed to the other servers without us having to copy
the zone files around and do all of this sorts of stuff.
So they introduced the concept of doing transfer X for initially.
Now we have X for as well.
Using the DNS protocol itself.
And just like copying the files around this copy is the data itself.
There are sets from the zone, not all of the other stuff, just the RR sets.
And so that now, of course, means that you've got to configure all the servers.
You've got to configure the secondary servers to know whether where the primary server is.
You might have to configure the primary server, which I don't know where the secondary server is,
for access control purposes and things like that.
And lots of other access control things like T-SIG and other stuff came along later.
And so that became zone metadata.
Now the servers had to be able to manage this kind of stuff.
And so once you have this set up, then the secondary server would say,
OK, I've been told this zone exists over here.
And I'm going to periodically go doing SLA query for that zone and find out if the serial number is changed
from the one I have.
If it's changing gotten lower, something's broken.
And if that number has gone up, then I'll do an AX for an old pull copy of the zone down
and replace the copy that I have.
And that's that.
And those SLA queries would be controlled by the refresh interval in the SLA record itself.
How often do you want to do this?
One today, once an hour, once a week, whatever is appropriate for your zone.
So great.
We have an improvement now.
You don't have to copy the zone files around yourself.
But that's polling.
We'll know polling has problems.
One of those is, if you change the zone content frequently, more frequently than the polling interval,
then you're going to have to go poke the servers to update anyway.
Because they're not going to pull off enough to pull this stuff down.
So that's not great.
And second, I'm excuse me.
Oh, yes, this is another RFCS gift.
I forgot I took a slide out.
So after that point, so that's as though it was the motivation then for RFC1996,
which is for the server that hosts the primary server, which actually introduced the term primary server by the way,
before that there was no such term in the DNS terminology.
To say, hey, I've just become aware of a change in this zone.
And I know that there are all of these secondaries out there who care about it.
I'm going to notify all of them that the zone contents have changed.
And now they can come back to me and get a copy of the zone changes if they would like to have them.
That's what introduced all of these terms.
Some of which are not in common use.
Like I've literally never heard someone refer to a server as a stealth server in regular conversation.
I think that term is gone by.
Also, I don't think I've ever heard anybody use the term primary master,
even though it kind of does make sense because it's the root of the graph that holds all of your zone.
If you have multiple levels.
And then of course, for various reasons, we don't generally use the words master.
So even more we use primary and secondary, which makes primary and master extremely cumbersome as a term,
because we replaced master with primary.
So now it's primary primary.
So that makes things even more confusing.
So that was really useful because now it makes you could get stuff out to your secondary servers more quickly.
You could just push a notification up to them that they need to update.
This means now there's more zone metadata, right?
It's not just the zone file anymore.
Now you have to know all the secondaries are which ones want to notify.
Maybe there are some that are not listed in the NS record set of the zone.
And you want to send notifications to them anyway.
So you've got to have a place to do all of that.
And what that means now is you've now gone from this.
Just here's these bunch of servers that happen to provide answers to the same questions.
To here are a bunch of servers that actually have to know about each other's existence.
They have to know what their IP addresses are and how what they should communicate with them.
The configuration data for the servers actually becomes important.
And it's sort of tightly coupled with them.
Not super tight, but before they were not coupled at all.
So this is obviously an increase there.
It also introduced eventual consistency, not strong consistency, right?
If you send if you make a change in the zone and notify goes out,
you don't actually know for sure when all the secondaries are going to be updated.
It will happen sometimes soon you hope maybe in the next couple of minutes.
But it's certainly not instantaneous and it's definitely not synchronized.
You're not going to be able to say, I have six servers all around the world.
And they're all going to appear at this exact same moment.
So the next query that any of them gets will have the new data.
That was definitely not part of the design.
So now you had to come account for that.
All right.
So for replication, this is technically all you really need to have.
Right? This works. People do this all the time.
My own personal stuff works this way.
I don't use anything anymore complex in that.
Even though I'm going to tell you about something more complex in that than I used for a while.
If the set of servers that you're managing is relatively static,
having tight coupling is not really a problem.
Because you're not adding in or moving servers all the time.
We're changing their addresses.
So you don't have to update the configuration.
So that's really no big deal.
If the records in your zones aren't changing frequently,
and they have relatively long TTLs,
then eventual consistency is not really a problem.
All the TTLs and all the zones are an hour,
and it takes two seconds for the update to make its way around.
That's fine.
It's a tiny fraction of how long it might have been cached in a result of our anyway.
So it carries if it took two seconds for it to get out there.
And then other good thing about this is two things about this.
Because of the way this works, it doesn't have to be just one level.
You can have multiple levels of replication.
And there are people to do that now.
Hidden primary server somewhere where they manage all of the content.
They push it out to their own secondary servers that they run,
and then they push it further to secondary servers that they rent
from DNS service providers for example,
to provide more redundancy around the world and things like that.
Since this is all done just using the DNS protocol,
that's very simple.
There's no special.
You don't have to have certain specific software or any of that kind of stuff.
So this is all great.
Sounds awesome.
But there are reasons you might want to do something else.
Zones can be really, really big.
Really big.
Like tens or hundreds of thousands of records in them easily.
And if you're using X for every single time,
there's a change in one of those.
That's a bunch of data that the hidden primary or whatever it is.
Has to organize to send.
And then it all has to be streamed to every secondary server.
And they're all going to have to get their own copy of it.
And so that's not particularly fast.
And depending on how you're paying for bandwidth on the ends,
it could be expensive for you to have to do that as well.
And we now we have this wonderful stuff called cloud native deployments,
which have changed the picture a lot.
You might have, first of all, you might have records being created
when a virtual machine gets launched.
And then destroyed when that virtual machine is turned down.
And they need to be visible to all of the consumers of that almost instantly.
You might also be deploying DNS servers in a cloud native type of thing,
like to have something very local next to all of the VMs,
which means if all of the configuration of the DNS servers all has to be updated every time you deploy one,
that's very frustrating or very difficult I should say.
So a solution, which I think all of the certainly all of the open source DNS authority
of server support is to push all the stuff into a database.
So okay, we're not going to do this with own files anymore.
Yeah, we don't have to.
We're going to let you use a database, some common off-the-shelf database to do this sort of thing.
And if you really want to, you can use a super fancy database that has clustering.
You can have nodes all around the world and it will replicate all of the data for you.
And the DNS servers don't have to do any of that kind of stuff.
They'll just say, I've deferred this responsibility.
The database is going to do it for me.
Great.
That might actually give you some things you wouldn't get otherwise, for example,
depending on the database you choose and how you choose to configure it,
you might be able to enforce strong consistency.
And if that's actually important to you, you could choose to do that.
It will have its costs to do that, but you could certainly choose to do that.
It means that the servers, the DNS servers no longer have to know about each other,
because the database is handling all of replication between them.
So the fact that there are one or five or 20 doesn't matter because the database is handling that
and not the DNS server level.
But obviously the database cluster nodes will all have to be aware of each other and configured to be that way.
So that could be the answer for you.
I'm sure there's lots of people here who use databases underneath their authority
of servers to distribute data around.
I would say that if you are the DNS team and not the database team,
you probably don't want to go down this road unless you also want to become the database team,
because managing a global cluster databases and non-trivial thing.
And I put a tiny joke in here, which is DNS itself,
is supposed to be geographically distributed and redundant database,
and you didn't want to have to do the work to do that,
so you picked another geographically distributed and redundant database,
underneath it to solve the problem for you just pushing the problem off to somebody else.
So in addition, all of the databases that are generally used for this,
they're not designed just for DNS, right?
There are relational databases, my SQL and MariaDB and Postgres and lots of other things.
They have thousands and thousands and thousands of features that are in authority
to server with never need to use, so that makes them big and cumbersome.
So I'm going to skip this one because we don't really need to do that.
So there is another option, the thing that I mentioned that's relatively new.
The first piece of it is two pieces of this puzzle, is a very lightweight database.
Some want similar to SQLite, but not exactly the same thing as SQLite.
Called LMDB, which is a relatively single file database
where everything is loaded into memory by the application that uses it.
It's very, very fast, very simple to make schema changes and all of those kinds of things.
And when you use it to back up an authoritative server,
then it holds all of the data that the authoritative server needs
outside of what's in the configuration file.
That's pretty simple.
It's actually like the most trivial part of this.
It just turned it on and it works.
The second part is the thing that does the replication between them.
So there's this tool now called lightning stream,
which is a second process that you're on on your machine,
and you'll see a picture of that in the moment,
which watches for changes in the database,
packages them up and makes them available in a way
that other nodes can get copies of those changes.
So you run one instance on every machine.
It creates these things called snapshot files.
Stores them into a, I've storage bucket,
anything that uses the SREAPI is a suitable location to put it.
And then all of the instances watch that bucket for changes.
I want to see that there are changes there,
download them, merge them into the local copy of LMDB,
and you move on.
So this is, just like X for a notify,
this is only going to give you eventual consistency.
You don't have any way to know exactly when all of those
changes are going to show up,
but the servers don't have to know about each other.
Because we've used a database layer below,
that does know about all of the nodes in a way.
So this is what that looks like.
It's a really trivial thing,
which is application sits on top of a database file.
Next to that, the application in lightning stream
don't know about each other really at all.
There's independent things that are sharing the same database file.
And all the stuff back and forth to the object storage bucket.
It works extremely well.
So there are some benefits to doing this.
The first is, none of those servers have to know about each other at all.
The only thing is that none of the pieces of software
in this equation have to know about the other instances at all.
The only thing they have to know about is the object storage bucket.
You can go from one server to 20 to 50 back to 3,
and nothing has to change on any of the existing servers.
The only thing that changes is the new servers come and go.
That's it.
So that means you can obviously add and remove things at the any time you like.
The bandwidth requirements replication are much lower than the extra,
because the data is compact, very compact format in these snapshots
and center round.
But it does use polling of the object storage bucket.
So if you're paying for bandwidth,
which is the reason I stopped using this,
you will end up seeing that.
And the polling frequency, obviously,
how often you want your dynamic record changes to show up,
is going to drive that bandwidth cost.
If you have them polling every 10 seconds, for example,
because you want stuff to show quickly,
then suddenly you're going to have lots of network bandwidth to deal with.
One other thing that is useful here,
which X for a notify do not support,
is that you can actually make changes to the zones at any server.
If you wish to, if you want to allow dynamic update,
DNS update, for example, on any of the servers in the graph,
you can do that.
There are some risks to using this.
A really, really big one is the object storage bucket is your single point of failure.
If something goes wrong there,
you don't have replication anymore.
The servers are going to keep answering queries,
so you don't have to worry about that.
You just can't get changes out to any of them anymore.
The second one is, and this is something that I ran into,
when replication isn't working properly,
it's not just a regular database.
You can open up a console for and do SQL queries against it
to see what the data looks like.
These bizarre compressed snapshots of data,
and the only tool that exists,
look at them is the lightning stream tool.
So you have to learn a little bit about that.
The final thing is, for now at least,
while this is all open source,
it's also only implemented in the PowerDNS authority of server.
So using this is really simple.
You already have PowerDNS off server installed,
using 4.8, which is the latest.
I think if that's the current.
Okay, so 4.8 or 4.9.
Version, that's all fine.
You configure it to use an LMDB back in,
which is really a really simple thing to do.
You have to build lightning stream from source.
It's very small.
It's really easy to build.
Get that all set up.
On the server server, you have a service,
where you use system D or unit scripts,
or whatever you use on your server.
It's a run lightning stream in sync mode,
and it watches for changes in the LMDB,
pushes them out to the bucket.
On all the other machines,
your run lightning stream receive,
which watches for changes in the bucket,
pulls everything down.
It's really quite simple.
You can run sync on all of the machines,
not just receive,
if you want to do multiple,
multiple bi-directional replication on all of them.
Be prepared for screwing that out,
because I did twice,
before I realized what I was doing wrong,
which was,
if you have two servers,
both of which currently have a copy of the zone contents,
and you set up bi-directional replication
on both of them at the same time,
both of them will say,
aha!
There's a new copy of the zone,
and they will just keep fighting each other
over and over again,
because they,
anyway, it's fun.
Yeah.
So, it's quick summary,
and almost out of time,
lightning stream and LMDB together,
give you a lot of the benefits,
you've used a new cluster database
for replicating data between your authoritative servers.
The resource consumption is extremely low,
by comparison,
for example,
to running a MariaDB or Postgres instance on every machine,
with their associated clustering software.
It easily supports a femoral server deployments,
which is fantastic when you're actually working on
troubleshooting a problem,
you can just say,
it's going to launch a copy of the author's server right here on my laptop,
and go get a copy of the data from the bucket
and see what the data looks like.
That's really, really nice to be able to do that.
And then, of course,
because it's not a complex database system,
you don't have to administer one.
So, and that is that.
Thank you all.
Thank you.
I made one minute to stare.
Yes.
One minute to stare.
It means we have room for one question,
maybe two.
How far can it scale?
So, the question is,
what is a scalability?
Could it handle 100 billion records on?
Personally, I have no experimental data to answer that question with.
I don't know if anybody has actually done that yet.
Okay, the answer from the team is,
yes, it can do that.
Yes.
So, yeah.
What do you mean?
I've got a lot of the full cover of the open source project
for redesses.
But, you know, when you're talking about the failure,
we'll book it.
I'll think that with my soul.
So, your question was about,
having a single point of failure for the storage bucket.
Yes.
Right.
Yeah.
So, for example,
if you really want to do this to,
well, certainly you can get
as three-style storage buckets from lots of cloud providers.
And depending on which one you choose,
they may have lots and lots of redundancy already available in their network.
So, that helps you.
If you want to do this yourself,
this works very well with Minayo,
which is the open source S3-style thing,
which itself supports clustering amongst multiple nodes,
and all of that kind of stuff.
So, there are multiple ways you can approach that.
Yeah.
All right.
Yes.