How many of you run the NS server, how many of you vote the NS server, how many of you
would not never do that again?
All right, we can start at end, so a little bit of a story by nine, it's started actually
in the last millennia.
It is a spiritual successor of bind 4 in bind 9, bind 8, sorry, and when people say,
oh, there are so many security issues with bind, they usually speak about bind 4 in bind 8.
And because bind 9 was written from a scratch, it's a different story.
Sure, we have a security issues, but a different kind of them, because there's something called
contract designed by contract, which basically makes the server crash if there's anything unusual,
which is much better than revealing a memory or writing a memory stuff like this.
It was originally single-fredded and multifredded.
You can choose that as a compound option by nine, and because of that, it's own event-based
copper of scheduling.
It uses locking for, you know, the multifredded, it has own RW lock that allows downgrades
and upgrades of the lock, which the standard POSIX RW lock doesn't allow.
So, 1998, how all this that?
So, for the same money roughly, you can get it like 4 CPU threads, 500 megahertz
clock, and you know, 4 gigab hours, and it was really, really cool machine.
Nowadays, it's like, you know, you can have a really, really be finished in for the same amount of money.
So, the technology is from 1998, a full phone, right?
So, more recent changes we did in bind 9.
So, in bind 9, the 14, those are already end of life.
We started requiring the atomics.
Here, it was not only standard atomics, but it was also, well, bind would be able to build
with the GCC atomics that's built in.
So, if you look at the source code, I wrote a, like, a shim on top of the Windows interlock
API to emulate standard atomics on top of the Windows API.
It was a horrible, horrible, like, if you ever wanted to see a horrible C macros that's the place.
So, I'm kind of a shim and proud at the same time.
By 916 already, also end of life, there's a new networking layer for, but only for the DNS server
part.
So, you know, the machine is half steam, half rocket.
And the bind 918, which will be end of life like next year.
So, it all uses the new networking layer for even the client part and the server part,
but some of the parts are still the steam machine.
It also, like, we ripped out the home cook allocator, which was really good at the time.
It was written, but nowadays, we would, like, as a DNS server, we should focus on DNS and
not brightening allocators.
So, we are relying on G, uh, malloc, uh, for, uh, like handling the memory fragmentation and
stuff, and it's really good.
You can copy it even without it, because there's, like, shim just to work with the standard, uh,
lib CL skater about, uh, the JMLoc is really, really much better option for using, uh, with bind.
So, uh, now, do you do, like, that more complicated part.
So, what makes part of programming hard is actually a name of the book by McKenney.
Uh, and this is really good book on, like, understanding the architecture of the more
computers.
It's open source.
Uh, so, term of foretling, memory barriers, atomic operations, pipeline flashes,
cash misses, IO completion, memory, uh, sort of memory references.
It all makes, like, the CPU runs slower, because it needs to throw away the cash,
and, you know, the cash is really close to the CPU, but when it's not, then it's much,
you know, the operations are much, become much slower and slower and slower.
So, every time you need to stop the CPU needs to stop, like, synchronizing memory
between each other, uh, and every time it needs to, like, fetch a memory that that's really,
or forwarded cash, everything makes the program slower, even though it's not visible on,
like, in a C or even a higher language is.
Uh, so that's, uh, for, for people, not, not programming close to hardware.
This is really, like, complicated stuff.
Uh, like, convinced me that it is not the case, but usually people, like, writing in Java
or, you know, the frameworks like Flutter, they don't really understand,
don't know if it, because they just, like, this is all hidden by the frameworks.
And, uh, I think it's, uh, it's a big shame that, like, many, sort of engineers don't understand
the hardware, they're everything, this program's they've, uh, on, because it, it's cost us,
it cost money, electricity, you know, performance, everything.
Uh, yeah, memory reclamation is also, like, the, the huge part we can talk about,
many days about memory reclamation techniques.
So, uh, by nine secretion, uh, you know, uh, still post six locks for the music.
Uh, there's a new RW lock, uh, it's not one to one implementation,
because that requires a lot of memory for the new mom, like, machines.
So, it's, let's little is simplified, but it's, you know, it gives an improve,
improvement in the, in the performance.
I'll show you, uh, flight and it uses thermodynamics.
So, that, that's what's required right now.
Um, so, of course, people using, you know, right,
had seven hours, uh, sent us seven.
That's already been, and of life are now complaining.
So, the effect of the custom RW lock.
Uh, so this is the main branch, and this is the, you know,
I created a branch just to switch to the default, uh, preferred RW lock.
So, this is, uh, so I should explain this graph.
This is logarithmic, this is reverse, logarithmic graph,
actually invented by the powerliness people here.
So, this basically means that, uh, if you look at this line,
this is, uh, response time in, uh, milliseconds,
and this is, uh, slowest percentile.
So, this basically means that something like 95 percent, uh,
uh, of queries were, uh, answered under one millisecond.
So, this looks huge, but it's actually,
the smallest part of all of it.
So, uh, so you won't, basically, this is, to be,
as small as possible, but as you can see,
this is, like, slowest percentile is, like, five,
fifth, slowest percentile goes, like here,
and something like one and a half percentile of the queries
are not answered ever, uh, which could be for various reasons.
So, this is just the effect of the RW lock.
Uh, this is, uh, this is the, this is the call cache,
this is the hold cache, the, the second picture.
Uh, so not much, much, much is the effect for the, for the hold cache,
but for the case, it's quite significant.
Um, so, and I'd like, like, complicated,
who knows what the read copy update is.
Okay, not bad.
So, uh, there's a really cool series, uh, from the altars,
uh, and there's a talk tomorrow in the kernel, uh,
uh, from one of the altars.
Uh, so basically, if you look at the RW lock,
uh, it, it stops everything when you are, you know,
for, so readers can, can work together,
but if there's a writer, it stops all the threats,
and so even the readers need to wait,
and the writer will do the work, and then they will continue.
So, it basically stops everything.
So, for the RCU, uh,
the mechanism is that the readers can, you know,
continue while this changes,
and so you start a change,
basically here, like, the pictures removal.
So, you start a change, and then you wait until, uh,
the last reader that started before,
you started a change, has finished,
and then, uh, you can, uh, free the memory.
So, this, this is a memory reclamation mechanism.
Uh, and again, you can see it, this is from the,
the, uh, the, uh, all WN article, uh,
that it takes, uh, like,
uh, it's much faster that way because it doesn't,
doesn't make the other, for it to stop.
So, this is one of the mechanisms that we are using.
We are, right now we are using just the RCU API.
This is right, the readcopy updates,
and they, they provide data structures,
our hash tables that are already, like,
lock free or, or wait free.
Um, so, um,
by nine.
Hi.
Uh, thank you.
Uh, so, uh, by nine, twenty, uh,
it uses USB-RCU.
We replace the locks that are WX in couple places,
like the logging subsystem configuration.
So, if you change something in the logging,
before there was a locking,
now it's just like,
swipe the pointer,
and when the grace period is over,
it just throws away the old configuration,
and there's no locking whatsoever.
It's not a big change, but it's like,
we are learning it.
Also, it's not so easy to, uh,
to switch the, like, the mindset,
how to do this, you know,
do the things in a new way.
Also, we had a, like, weak references.
So, the, the stronger references that, uh,
do, do, do, do, do,
do, just close the door, please.
It would come in or out.
So, the stronger references that you know,
it's there.
So, if you, if you, if you,
do reference the pointer, you know,
that you have a value or not.
For the weak one,
it there could be, uh,
it could be now, uh,
or there could be objects,
so we, you know, use,
RCU for the weak references now.
And, uh,
there's some,
smaller places where you reuse, uh,
that the lock free rate free data structures,
like,
something like called,
callbackstable that's used for catalogue zones.
And, and RPC,
it needs,
well, when you basically load,
a new version of this catalogue zone,
it needs to call back something and call,
do something to,
we use,
in the small,
small places,
at the DNS dispatch,
you use the lock free,
uh,
hash table,
and the bedcash,
uh,
so this is the place where you be stored,
that the server is unreachable,
for the name,
and stuff like this.
So, for the bedcash API,
formally,
it was a, like,
bucketed lock list,
uh,
where it was linear list,
with the DNS hash names,
as in index,
and the list that we use are,
are RU,
the list recently used list,
for cleaning,
uh,
right now,
it's completely lock free,
uh,
there's a single hash table,
and,
paraphrase,
there's a parap,
paraphrase,
are you list,
um,
so you put it in the right place,
so you don't have to,
there's no locking whatsoever,
um,
originally,
I just,
like,
fixed this in the recent release,
originally,
I did it in a wrong way,
that's,
I'm saying,
it might be somebody,
something harder to do that.
So there was, like,
opportunistic cleaning of the table,
which,
basically meant the memory just blew up.
So this is now fixed,
uh,
by using the paraphrase,
are you list,
uh,
and, uh,
a proper cleaning of the,
uh,
of the table.
Um,
so for 920,
uh,
so again,
the,
918,
uh,
and before,
it has the,
uh,
the all cooperative scheduling,
so,
and,
sometime with,
between 916 and 918 and 920,
it was, like,
a kind of hybrids.
This is one on lip UV,
that we use for networking.
So lip UV has also haven't loop,
so we just replace the old one,
to run on top of this,
and 920 just ripped all of the old code,
uh,
out, and, uh,
so there are just lip UV haven't loops for each fret,
and stuff runs on top of it.
It's a simple,
but more powerful,
uh,
simpler,
but sometimes more complicated,
but it's,
uh,
because it,
it allows more stuff.
So,
um,
the easiest API is I see job.
There's no,
no looking.
It's on,
on loop,
very fast.
It does no allocations,
because,
uh,
like the,
the part of the memory
that's needed to run the job,
has to be on the,
in the structure,
uh, already.
Uh,
and it uses the,
there's a,
like main fret pool of the,
event pools,
uh,
and it uses the,
this one,
the shared one.
So then there's,
I see a thing,
uh,
this similar,
it uses the weight free,
uh,
but it's,
uh,
but it's,
uh,
but it's,
you see the,
the very free queue,
uh,
um,
so it's for,
also for fast operations,
um,
and it uses the same fret pool as the I see jobs.
Then we had to,
there was a,
uh,
vulnerability in the NSEC,
uh,
as in the protocol,
uh,
where if you just,
like hit the server with,
like huge,
the NSEC operations that would just,
like slow down everything.
So we had to introduce a,
yet another fret pool.
So,
and this is only used for,
uh,
slow,
crypto operation.
So basically,
when there's a,
the NSEC validation,
we receive it on,
on this,
on this shared,
for pool,
and then passed the NSEC,
operation to this different one.
So,
this one can still,
uh,
dispatch everything else that's,
already being in the cache,
or,
is alternative data.
So the NSEC operations,
doesn't slow down the,
the rest of the operations.
So that's why there's so many of them.
And there's I see work,
this is using the,
lip UV,
uh,
for pool,
and this is for really,
very slow stuff,
like writing the zone on a disk.
So we just pass it there,
and just passes back when it's done,
and we don't care.
So,
um,
this uses,
the whole thing,
the whole new thing,
uses the fact that,
uh,
kernel is much better at scheduling than we are.
So,
this cooperative scheduling,
uh,
was probably okay in,
you know,
the year that the bind started,
uh,
but right now,
we basically removed everything
that had to do with the scheduling,
like assigning threats to,
uh,
to CPU cores.
Uh,
and so we are not trying to be smarter than,
than the kernel,
because the kernel is smarter than us.
Uh,
um,
the other big change we did in the,
in the 1920,
and there's this working progress.
We have a new QP3 database.
Uh,
this replaced the RBT database,
which is the,
the RBT is a bit of misleading,
because it's more like red black forest,
because it's,
uh,
RBT3 of the RBT3 is because of how the DNS structure works.
Um,
and,
is it key value storage suitable for DNS,
it uses RCU for reads,
it blocks forcing donations of rights.
Uh,
but that's had,
it still has,
uh,
so the underlying structure is much better,
but we just like bolted it to the old,
old one,
so it still uses the RWOX for storing DNS data,
and this is bucketed and stuff.
Um,
is very technical detail,
so,
uh,
it's okay.
But I want you to show you,
uh,
the difference.
Uh,
so this is the UDP response latency.
Uh,
the blue one is the 1916,
and I can't run the 1916 anymore,
even in our,
the 1918,
and the,
uh,
green one is 1920.
So I would say that's,
uh,
nice improvement in the response latency.
For the memory usage,
uh,
the same colors.
Um,
but,
uh,
this is,
older picture,
I just, like,
this week fixed,
uh,
or made up optimization that the,
the 1918 memory is much lower now.
Um,
so,
how do we measure the hotspots?
There's a Detroit system tape,
a system tap.
So,
basically you can measure,
uh,
when you start locking,
when you,
like,
actual acquired lock,
and when you release lock.
So,
uh,
we have kind of,
our own detrace,
uh,
the script looks like,
something like this,
that you,
define,
uh,
basically,
these are tables,
uh,
inside the kernel,
and then you measure,
and you have a mutex,
and free mutex,
acquired,
and it takes release,
and the output could look like something like this,
and you have a,
how, uh,
how often it was,
uh,
acquired,
and how much time it you spend.
This is, like,
sorted in,
uh,
uh, in the table.
So, these are the,
out spots where,
this is,
again,
a little bit older data,
but,
uh,
so these are the,
the spot there are more,
like most heavy content in,
in our code.
So, for 921 and beyond,
uh,
we, again,
we want to,
so it was the best thing,
memory synchronization,
because I'm not synchronized at all.
So, we want to keep all the data,
local to threats,
unless we have to.
So, we will share only what,
absolutely needs to be shared,
like a cache that's,
they make sense to share it.
Uh,
So,
caller is in typical cases to RCU,
uh,
API.
Uh,
not everything can be converted.
So, if you have a,
some long lift objects that,
uh,
like,
on one tick of the oven loop,
you acquire it,
and then it does something,
and then you restore it,
and then,
basically,
you can't use RCU,
you need to use reference,
counting,
or something.
Uh,
and,
uh,
there's a,
I'm going work to replace the configuration,
uh,
so it will also use,
uh,
key value storage,
and some callbacks,
and remove the global,
some more stuff,
but I,
you have,
some weird pictures.
Uh,
so this is,
this is recent,
so this is like,
a very heavy,
uh,
QPS.
So,
by nine,
18 versus by nine,
20.
Uh,
so again,
this is very,
very heavy,
uh,
hit server.
So,
um,
since like,
300K for cold cache.
Uh,
but,
I have,
really,
I have,
like,
this is my,
what the fuck moment.
So,
these top lines are,
like,
one and two threats.
And,
the other one,
our cleaning is much better,
if you run it with more threats.
I,
don't know why.
Uh,
and I'm probably not going to debug this into much detail,
because, you know,
uh,
this is the nine,
twenty,
and it looks much better.
So,
for the CPU,
again,
by nine,
18,
so it's like,
one threat to threats for,
eight,
and 16.
And,
you know,
if you compare this to nine,
twenty,
nine,
twenty looks like,
we,
well,
this could be just measurement artifact,
who knows.
But,
if you look at the,
so load for,
eight threats is almost,
almost,
eight hundred percent,
basically means that it used all the CPUs it can.
Uh,
So, now we need to focus how to reduce the work we actually do inside.
That's,
there's a only way for what.
Thank you.