Hey, where come to I talk about Suffolk Tsung?
This is not really about DNS in general, but more about orchestration, orchestrating DNS.
It's all a bit silly, that's why we're here, so be prepared.
Well, Tommy, I hope you can make it really likes bash, and you will find that in the talk again.
And I like distributed systems.
I read internet standards for fun and implement them.
It's kind of our got in the sole DNS thing.
And we both do a product surf fail, which is an authoritative DNS service.
We have everything is open source, so we run power DNS, which is open source software,
our back end is open source, and our web interfaces as well.
It's fun.
We support multi-tenancies, so when you delegate it to the main, you can choose the primary
name server, and all the important stuff, like key material and stuff, never leave the primary
name server, and it's just delegated, and the rest of the zone is just AXFR to all the secondary
name servers in the network.
We have both product-helset, product-helset name servers, which we run as a product, and we
also have when community members, when we trust them enough, and they're active for
a while, they can also start to have time, they can also run their own DNS service, which
then take a network, and also start to, well, people can also put zones there if they
want to.
We both have stable name servers, which are like production ready, they're running
since a while because of the unstable, and they're working name servers, which are different
still in the testing phase, or even temporary name server, when someone brings some
silly hardware to them, event, and we happen to have an IP address, just plug it in, and
oh yeah, we run the DNS server for a few days, so we just, we can plug it in the project,
and people could use the name server they want to do, and as mentioned before, our
tech, the quotation mark stack is built on Power DNS, as mentioned before, Bash, there's
a lot of Bash that, if I have back end as written in Bash, it's, and the API proxy is written
in Bash, and it works, surprising you well, better than some other solutions, and of course
there's auto-curl because we need, don't be right, there's auto-curl, and one and a half
hours in front of it, it will be faster to walk, yes, I will give it to you in this work,
I will need to lean in really close, because Power DNS is a lovely API, we need to
somehow access, why do we need to orchestrate, so the handful of servers you can just copy
config around, it's fine, you can get it by with that, when you get like a handful more
servers, that's becoming more tedious, more painful, start getting this configuration,
and just as a certain point, just don't even bother with manually trying to copy configuration
system, no, we have 12 servers now, just forget it, and not all of them are hosted by
us, they're hosted by volunteers who have spent time, and we don't just want to bug them
all the time, hey, can you please, can you please add this new name server, we just added
for a few days, so we really need to have something to shrink, important, important part
is that we don't have root access to all of the servers, which is by design, because
we want this to be sustainable, and we want to divide the work among as many people as possible,
instead of just, oh yeah, I will fix it, and we have 100 servers to fix, then like,
us two people, that would suck, so we want to avoid that, so what do we need, as mentioned
before the network status is assumed to change, we may have new servers which people want
to host, we may have to serve as a service, because there were just temporary or like, oh,
someone doesn't want to host a service anymore, which is fine, or like IP migrations, because
everything is also done, we don't really have a lot of own infrastructure, we grant a lot
of these servers, so sometimes we just move as well, everyone's something that is easy to configure
and run for volunteers, so when we just have push new names of us like the mod of servers,
you have to change IP addresses, and something we will get into later, we have to change IP addresses
for the primaries of the different zones, it's not really feasible to do at this scale with
volunteers and just gets really neat that automated, we need to manage something which I have
called static and dynamic configuration, the static configurations like just in the config files,
which servers are allowed to AXFR, which allows servers are allowed to notify, which servers should
you notify on zone updates, and then there's a second part, probably, and there's something called
auto primaries, when you have the right name server set and you AXFR is own, which the server
has previously not seen and just automatically creates it, we make a lot of use for that, because
we don't need to script an extra API endpoint, it just takes some work of us, it's very nice,
and that's called, that's stored in SQL, so we need to update that with PDFs,
YouTube, and everything should run as previously mentioned, with reasonably low privileges,
we don't just want to dump some, some we get scripted, we ran some we get scripted, we wrote
yeah, it run it as root done for real about it, it's not really, that's not really nice,
some existing solutions would be Ansible, which I unfortunately make a sense of use of,
I don't like it, you kind of describe the states, and just hope everything
matches that state, and there's an extra files in the directory, just don't worry about it,
it just don't worry about the extra files, they're just belong there, and Ansible just wants
to be root over time, so it doesn't really fit in the whole, yeah, yeah, we should have minimal
privileges, but it's also hard to integrate a bit, and there's nix, I know nix, I make extensive
use of that, on my own hardware, but I would never recommend it to anyone else, it's like a very
circumstances on which I would, it is missing a lot of documentation, it's slow, like a proper
nix concept, it's slower than an Ansible deployer, and you're not doing it right,
a very like multi targets, you need to build like one configuration for each target, and that's
even slower, not fun, dynamic configuration is not really something, it's designed into nix,
people have wrote some scripts which fetch the state and update the state for some data bases,
but it's mostly nix finds when managing static configuration files and not whatever the hell
we are trying to do, as also hard to integrate again because people need to run a nix demon,
and when you connect to the demon with some user to develop the config again,
not really feasible as well, so build our own, they still do a microphone and continue
it, you can still do a microphone for that, you won't be, no it's fine, you can just stay
and really close me, it's okay, hold on, I think we're good, so we have devised a solution, of course
in Bash, and the solution is two parts, it's a client server architecture, the clients are deployed
on individual power DNS servers, as in on the same machine, because they need to modify some state
that is on the same machine, so that just makes sense, and the servers or the server right now
is a centrally located control plane, and we are looking slowly into, we are thinking about
distributing this somehow, but distributed systems are not really superbomatic, so we have started
with a simple solution where there is a central authority, purely because that's easier for us,
we don't like saying that essentially this is blockchain architecture, we don't like saying that
this is blockchain, but you have a chain of events, which is to say, when you add an event to
the timeline, you can never remove that, you can just add another one that modifies the state,
this is made to ensure that all servers can know what all other servers are, like what point of
the timeline there are, so they can request changes from a specific point, from a specific height
in the timeline, in the chain, and we implement two simple instructions right now, which is
modify and delete, modify either ads or modifies an existing record, or the delete removes
records or removes the server, and surface inclined, as FanFa has already expressed,
modifies both the static configuration, so generates a conflict file, and the dynamic parts,
so calls up, pardon us, comment, saying, hey, add this out of primary, and generally,
yeah, we have deployed with duals on most systems, which is to say, it is running as a low
privilege user, and we gave that user specific commands that they can run, some of this can
probably be changed from root to pdns, I don't think we need to run pdns, but this is
roughly what we run in, and this nicely lowers the amount of surface area that we need to
work about, so in case of a security breach, potentially they will be a little bit to this
carnage, if there is anything running on the server, except for pardon us, thanks to the polling
architecture, so we have clients connecting to the central server, we can ask the clients to report
specific data back, which is to say, we report the script version, we report the pardoned
unit version, and we have some small error reporting, so in case something goes wrong, we may be able
to know just for this conflict plane, separately from this we have Grafana, and we also think about
introducing Grafana support to look for like the permitture support to set up a sync,
this is something for the future, other than that we are thinking about improving the admin
UI, because currently it's just a web page, and it is the least smallest web page I could make,
which is to say HTML forms, and very, very little error handling or anything else, it just fails
outright, because it's not supposed to be used by a user, it's supposed to be used by us, and yeah,
so we deployed it, we had our first successful server migration, everything seemed to go all right,
until like 30 minutes later, some are modifies their zone, and suddenly the eggs are not accepted,
like all of the servers are rejecting eggs, eggs are farce for zones that were mastered on that
server does it, that we migrated, what gives, so turns out the power DNS except for the
static and dynamic configuration that we have discussed previously, it also has
healed in the database that specifies a list of IPs of a master of a certain zone, so this
is populated on the first notifying, and we were unaware about this and heal that very migration,
so we needed to hold patch, and the fastest way to hold patch will be, we don't want to know
how to solve this properly, you want to solve this right now, so we just look down to all servers
manually, the thing that we were trying to prevent from having to do, and we manually updated the database
to fix this, we started the servers, everything started, appropriating all right, so that was
maybe 30, maybe 40 minutes of 45 minutes of downtime, from the beginning through, as discovering
something is wrong up to the resolutions, that was really, really fast, regardless, the proper
fix is apparently to use our dedicated power DNS comment for this, but this shows us another
problem, Serfale's think client don't really have any access to all list of domains,
because they are fully separate from all our other info, they are integrated only with each other,
and we wanted to keep them separate to maybe allow others to deploy them without having to
deploy all of our stack, and that's a problem that we have solved, we were trying to solve it through,
maybe PDN, PDN as you can also show us the list of primaries, well you can, but I wouldn't want to
expect this output to never change, so my assumption is that this is not a stable API, so to speak,
and I went looking for further solutions, unfortunately the only solution I found was
parsed the compile, extracted the API keys for the HTTP API, and tried to co-local host,
this is not yet integrated into our code base, as can be seen by the very nice mockup code do not run,
this is because I didn't have any servers to test it on our test environment, but yeah,
generally should work, hopefully, and that was one of our wildest, like the whole engineering
disaster, as I would say, want to call it, was one of the wildest things that we didn't expect
in a product scenario, it was for everything else we were able to prepare quite nicely,
and here there are just some things that even if they are documented, there is so much documentation
that's seeking through all of it is like, when you don't know that you will have a problem,
it's just not feasible, so we need this. In summary, Vachosahamer and we made
nails out of PDN as YouTube, and I hope you like the presentation, if anyone has any questions,
I would love to hear them.
We have a lot of time with questions, so.
Usually it's recommended to do XFR's over as a SQL connection, because VMS is funny,
it tends to do things at my next level that you do, saying the truth, and so on.
Why understanding is that right now it's just going directly through my idea?
Currently, yes. The question is, the general recommendation is to proxy XFR's through some
secure connection, and how are we doing this. We are currently just answering XFR's over the web
without any encryption. The thing is, we are planning to move to WireGuard, but this is
undertaking when you want to make this automatic, so we essentially will need to extend
Serpile Sync into also provisioning WireGuard, which hopefully is something in the near future.
For right now, we have not seen troubles with this. The XFRs are propagating and as far as I'm concerned,
they are signed, right? I don't think that they could be tampered with.
Plus, for users who are worried about tampering, we do offer DNS and then the way we
offer the DNS is we pre-sign the zone on the master, and all the secondary servers have
a row and it's insect free records and everything else. We're just to say that if you modify any
part of the zone, it will be noticed by the DNS second implementation on the other side.
Hope that answers it. Anyone else?