Now that it is, you know what I'm doing, it is time for the let's talk today, Amy is
unfortunately sick, but Baker is going to take over because he worked with Remy on this
project, making the nest is actually fed on like tiny deposits with not a lot of C-F-U and
so enjoy. Yes, thank you. Thank you. As Peter said, I worked with Remy on this project, this slide
deck was written by him and focuses on the bits he did, I will just try to get through them as
well as I can. What is DNA's this? Who here knows what DNA's this? Who doesn't? Okay, a couple
of people. DNA's this is a DNA's proxy, I think there's a diagram. Right, so most people
deploy it like this. You have a bunch of clients. These might be laptops or resolves in the
internet. They speak to DNA's this, which loadbent's traffic to a bunch of backends. This is
the common DNA's this deployment model in I-Spees and Hoosters, etc. However, DNA's this, you
also sit on the client and imagine in this case, DNA's this living on your laptop or on your
home router or whatever, talking to a backend resolver somewhere out there, your I-Spees or
Quad9 or whatever. And it is that use case for which for which DNA's this only router might
make sense. So in 2021, a bunch of people sat and over to the money to sat the DNA's this would
be great because it would enable DNA's encryption from the router to the provider or Quad9,
Quad8, whatever. And DNA's this already supports all the scripted protocols unlike the
DNA's mask, which doesn't support as many. So we said, well, it's not that big, so we can
just do that. Well, we were wrong. So in 2022, open WRT supported devices as small as this,
four megabytes of flash, 32 megabytes of RAM. In 2025, four megabytes is unlikely to even fit
a useful kernel, but back then they really tried. So we bought a bunch of these TP-Link boxes,
which was a picture, but it looks like any TP-Link router you've seen, it's black 3 antennas.
It's slightly better specced. It has like four times the numbers I mentioned.
So with you be able to boot a kernel, run open WRT on it, have a weapon to face, and fit DNA's
this in with that. Turns out DNA's this is not small. The binary size in a compilation that relies on
shared libraries is like 9 megabytes today, plus the shared libraries, which may not be present
on your router. So we realized that we were underestimating or overestimating our ability to fit
inside a router, and also we didn't know what we were doing to begin with. There's
it opened WRT, open WRT, two things matter, the size of your binary in the flash file system,
with users compression, and the amount of memory you use that is only yours.
So this memory definition is called the proportional set size, and it turns out it is
super hard to measure. Like all things in memory, you would think if you have a device with
16 megabytes of memory counting a few bytes here and there would become easier, and it is easier,
but it's still not easy. Your memory is split up in a bunch of areas that all of different properties
someone just yours, someone swapped in from your compressed binary, some are
swapped in from shared library that you may or may not be the only user of, or that part of the
library may or may not be just for you. So it turns out it's quite hard to actually count these things.
But we managed to define a way of counting, and at that point, this will be found.
The NS this itself needed about two megabytes of memory, live crypto, also needed about two
megabytes of memory, and then we needed another one I have megabytes for other libraries we
needed, including lip SSL. You'll notice that open SSL is actually the biggest memory user
in this scenario, and then we needed two more megabytes just for storing our data estate, etc.
So that's quite a bit more than four megabytes. So we tried a few easy things.
The NS this defaults are for big setups, machines with four gigabytes of RAM, handling thousands of
clients, etc. Your home router is not like that. So you take all these numbers that are default
a tune them down. Only 50 outgoing queries at a single time, a single T speed threat, a single
DOH threat, very small buffers in which we keep track of recent queries, etc. This helps some.
Next up, the NS this has a shit on of features. Many of which you might not need on your home
router, like the CDB or LNDB support, those were built for doing million entry block lists.
Which you may want on your router, but not everybody would want it. So we took the open WRT
the NS this package and added a whole bunch of extra flags to it to allow you to get rid of features
completely at compile time. Then there's the compiler link of flags, those influence, binary
size and memory usage a lot. As most of you would know, you can tell GCC or Clang or whatever to
of to my zero, of to my one, of to my two. There's a funny little other one called optimized
for size, which does something between one and two, I think, except when it would make the binary
bigger. And this helps some. Hiding symbols just reduces the size of symbol tables, which is helpful.
A link time-open optimization is quite an interesting one because it removes that code,
even from libraries you might be linking statically. And I'll get it later, but linking libraries
statically might actually reduce this usage because you can get rid of code that nobody's using.
The bottom one was a bit of a pity. Disable position in dependent, I think it's execution,
because by offers more security, that it makes the binary bigger. So there was quite good.
The binary drop below two megabytes compressed by the file system choices open, W or T makes,
and memory dropped a bit as well. So that was decent.
Okay, right. So then let's figure out what is happening, where is all this memory going?
The heap is the most important bit because it can be swapped out or swapped in. Most routers do not have
swapped configured. And of course, the binary and the libraries are basically swapped in and out
when necessary, but heat memory will just sit there, if physical memory.
So, Rayme, you cell grind to investigate some of that.
I have to say I haven't seen the picture before myself, but it also might be hard to read, but the
big red reddish thing at the bottom is crypto malach. So a lot of a memory is barely going to
live crypto allocating things, keeping stayed around for whatever reason. Then there's heap track
which makes these nice flame graphs of where memory is being allocated,
which I also have to use myself so I can't tell you much about it.
But realizing that openness is always doing a lot of the allocating helped us strip some more
use the stuff from the binary. So we don't load sivers and digest, we don't need our messages,
openness, although they're not quite good. It can give quite extensive error messages,
and we figured we don't really need those. Apparently, some things would allocate big and then
be shrunk, which we could skip. And there's links to all the codes for these changes down there.
Then there's lip H2O, which at the time we use to offer the OH, that library is that,
it's not being maintained anymore and any useful capacity, but then it is what we had.
It turned out that library also contains a bunch of things that we didn't need for running on a
router. And indeed, the slide also mentions, we no longer use H2O. We now use NGH2P2, which sadly is
what everybody uses for the OH. So there's a bit of an ecosystem problem there in that if NGH2P2
has a bug, then all OPSOR's DOH, implementation is out there, we'll have that bug.
So I hope something else arrives at some point, but right now this is the state of things.
And again, we reduced a buffer because 8 kilobytes is more than enough for most DOH requests.
We tried using wolf SSL, which is a nice project, and it has an open SSL compatibility layer.
But it did not really reduce memory, and adding this extra dependency only helps, does not help the moment,
some other program, does want to open as well on the same issue, because they have both libraries.
So that's which did not do anything for us, but it was worth it shot.
We tried UPS, which is a, which compresses binaries, not in the file system layer, but on a
different layer, the problem with that is that it decompresses into memory of or if you're
unlucky even onto a temp file system, which means you actually lose the benefits of the manpaging.
All right, next step is to try to even harder. There's this tool called bloat, I think there's
output here, yeah. So we built a binary with the features we want, we copy it, we strip it,
and we run bloat on the stripped copy, because that's the one we want to measure sizes in,
but we use original binary on the side to steal the bug symbols.
And we found out that a lot of our memory was going to Lua. Lua is the program language we use for
writing the nested configurations in, so it cannot just go, but still perhaps there was some
room we could get back. So we realized that some of the structures in memory were padded
inefficiently for memory purposes, and we realized that preventing false sharing, which means
having unrelated variables not leaving close together in memory costs a lot of memory.
So if you put those variables together, you can save memory at the cost of some performance on
big multi-trade machines, which your router is not. Then there's the number of threats,
the config I showed earlier did some of this, but it turns out we could strip even more threats
from the process, which saves a lot of memory, because each threat comes with a stack,
and the things that is doing also include a lot of states, of course.
So for OpenWT, implementing a simpler threat model, where for many things, we didn't even have
multiple threats capable of doing the handling, but just one. Then we found out that we were
fragmenting memory, and as Andre said, the best way to synchronize threats, it's not synchronized
them, the best way to allocate memory is to not allocate it. So it turned out there were a bunch
of allocations, we could get rid of. The lower garbage collector is not very aggressive by default,
so we now trigger it a couple of times during startup, but just goes and cleans up all those temporary
objects that we don't need anymore. And of course, the number of threats helps a lot.
We also tried linking the C++ standard library in. This made a memory usage slightly smaller,
but it means we now are the distributor of lib s3C++, and again, if a second program wants to use
the same library, then we are causing more memory usage, which is waste. So that was roughly where
this ended up. We were slightly overtarget, but a waste smaller than before. The return on investment
of other tricks we thought we could do would be small. So we set up an OpenWT feed on our website,
and we have contributed all of this upstream to the OpenWT project, and this is the cursed status
of DNS dist for the current OpenWT stable release. The binary compressed is one and have
megabytes or five and a half uncompressed, which means five and a half maximum memory use for mapping
the binary into. That's with all the features. Yes, so we now have two builds, a full one,
and a not full one. The difference is roughly the list I showed before CDB and the B, etc.
And the memory usage total, so that binary libraries and heap is around four megabytes now,
which is still a lot on your home router, but it is a lot better than it was before.
While we were doing this, we also added UCI integration. If you ever worked with OpenWT,
you will have seen UCI or Lucy, which allows you to configure the software, the drums and your
system, and before we did that all you could do was edit the DNS dist config, and there was no integration.
So this will make things a lot better for OpenWT users with DNS dist. It's quite a big PR,
so they haven't gotten around to reviewing it yet, but I'm sure they will soon.
Here's an example of the UCI config. I'm not even sure that's being an after read here,
but you can read it later. Oh, this is a fun one. We also added DDR desicnated.
Bye-bye, help me.
Okay.
DNS desicnated network is over. It's a discovery, right? Right. Yeah.
This allows DNS dist to tell your clients, like your iPhones and your Android devices,
that they can use the RTRDOH to talk to your DNS dist, and they get a situation where
DNS encrypted from your mobile to your router, and then encrypted from your router to some upstream,
which is better than no encryption, but also a bit weird to have that re-encryption step in between.
However, this allows you to do filtering on that device.
And that is the last slide.
Yes.
Question.
Thank you. Yes.
If I could remove the lower part, how much memory would I save?
Not a lot, because of WRT already shipped Lua, although they're getting rid of that.
So it might be an interesting experiment for the next OpenWRT version.
Have we tried linking statically against lip SSL with LTO enabled?
I think we did, but given that open SSL is already installed, this will never give us benefits.
If open SSL is not installed, and we knew where the only user, this will probably be the right
cause of action, of course, of action.
If I use DNS dist on OpenWRT, I'm still using DNS mask, which is a so DNS dist,
is just between DNS mask and my provider in a summer, or do you replace DNS mask?
So the way I've been running this at home for quite some time is I, sorry, the question is, does this
replace DNS mask, or sit beside it? It could do either. You need a DHCP server.
However, you can also use ODHCPD, which is actually a better DHCP server, I found.
So I've been running at home. My office has been behind an OpenWRT box,
with no DNS mask on it for quite some time, using ODHCPD, and DNS dist.
And some of the UCI integration we did also offers host name,
what land resolving, or DNS dist pools, that information from the DHCP server.
I guess I was the question earlier, about from a matrix, the question has D to P integration,
which is indeed a DNS mask when implemented in DNS dist somehow.
Right, I did indeed just answer exactly that, very good. Anybody else?
Okay, thank you all.