So, my presentation is about DNS, but not just about DNS, not only about DNS, and actually
this idea originated as an internal company contest.
When I had the contest for every employee of our company, about this, take any data
that you like and make application out of it, and it was just for one or two days, and
apparently I was the only participant in this contest, so I have a lot of fun, and I have
this presentation.
So, I need some idea, I need a nice data set, and I need a way to visualize and analyze
this data set, and to get a data set, the easiest way is to get it from the internet, and
we will get internet scale data set, so how to get it from DNS, and just basics.
You can do forward DNS requests, and the easiest way to do it from a command line is to
just use the host command line tool, and for Google it will output IPv4 address IPv6 address
and something else, and you can do a reverse DNS request for IP address, it will output something
for example, the name, but for Google, for the same IP address, for Google it outputs something
strange.1E 100.net, question for you, what does it mean, or anE 100?
1E 100 is the certificate notation for the number they call GoGo, so the point is, regardless
of the DNS, records don't have to point to the original name, they actually cannot, because
a lot of names can point to a single address, and actually they can contain about anything.
And you can also use the more advanced tool dig, where you specify the type of record
you want to get, and you will get the answer from your DNS server.
And now, a question for you, if PTR records don't have to be precise, they can contain
anything, an arbitrary string, you can't count on these records to contain what you need,
why do they even exist?
4 for mail server, mostly, for sending emails, and also for nice infrastructure of the
probability.
So when you run something like MTR, it is like Trace Root, you will get also these names from
reverse DNS records.
And the idea is to take IPv4, which contains less than 2, less than 4 billion, something
IP addresses, slightly less, and just do reverse DNS records for all of them.
So how to do it?
There are a lot of tools, ready for that, like MES DNS, or there is a library, GNU ADNES.
They are fast, from the rhythm of MES DNS, they say, if you run this tool, you will
get all the result in less than 1 hour.
As a problem is, if you do it from home, most likely your internet will break, and your
internet provider will break your internet, and you will have to call them.
If you do it from your hosting provider from AWS, from digital ocean, whatever, it is also
not a good idea, they don't like it.
At least, even if they don't restrict it, it will be good if you communicate in advance.
If you contact them, explain your reuse case, explain what exactly you will do, they will
not, so they will not worry.
But I am introvert, I don't like communicating with anyone.
So I wanted to find another way to do it.
Yes, so from home, not a good idea, from this center, not a good idea, my friend was
blocked, because I asked him to do it, and he was blocked, not me, from the cloud, also
not a good idea.
But there, let's take a look how DNS can work.
What protocol is exist?
It can work over UDP, port 53, it is unreliable, it is not secure, it can work over this
CPU, it is not secure, but more reliable, there is DNS over TLS, reliable, secure, but
heavyweight, there is also interesting protocol, DNS over HTTPS.
It is just on HTTPS API, where you connect, and request a record and it will give you result
in one of several formats, including the original binary format, or JSON for convenience.
And one of those is cloud, where DNS over HTTPS over, there is a Google DNS over HTTPS
over, probably many more, and interesting that you can even write, you can even use it without
the domain name.
If you use something like cloud for home, you have to remove this name in advance using
a different method, like writing it in the host file.
But you can write HTTPS, just one, one, one, one, which is the address of this DNS
server, and it looks quite nice.
Another is also DNS script over TCP and over UDP, and it is actually a mess, because
even for encrypted DNS, you can use more than three different protocols.
I decided to check if I can use cloud for DNS over HTTPS.
So the API, if I use the most convenient one, with JSON, it looks like this, use that
if I, your request, the type of record you want to get, and you will get JSON with all
this information.
It can use over HTTPS 2, even HTTPS 3, and HTTPS 1.1, so everything works, and with HTTPS 2,
I can use a pipelining with multiple requests, and it also works, you can use it from
Coral.
I have read carefully the documentation about if I will just do several billion requests one
day.
Will I violate their terms of service?
And apparently I just did not find any mention of it, and I thought so cloud for such a
big infrastructure provider, if I do plus just one billion requests, no one will notice.
They suppose that the processing three lines of requests every day, if not 100, 3 lines.
So I did it, I also prepared a table for results in my database, in my favorite database,
which is clickhouse, the best one is called database, and this table is a simple day-time
and JSON with a string data type.
And then I wrote a simple shell script.
Let's read this simple shell script.
So first of all, there is a SQL query inside, and the SQL query will skip all the result
IPv4 addresses like 127, anything, link local, etc.
And it will generate a ranges of the first three numbers.
And then I will take these ranges, and also, paralyzed by the last number, the last
talk that, and I will parallelize using corral, and I will generate a comment that will
query this 256 addresses with a single HTTP as request using corral.
And I piped it to bash, because I just generated this comment, and then I piped it into
clickhouse client, finally to insert into the database, with this another SQL query,
insert into.
But it is just one line, so I will script you can see it, okay, so what happens?
I have run it, I have left it for a day, I went to sleep, and then I woke up, actually
you know, nothing actually happens.
So I did it from a single machine, I did not prioritize by many machines, and I tried to
be gentle, so, without using too much parallelism, the question is how long did it take?
A day, okay, any other two weeks, couple of hours, two months, no, actually it took about
ten days, so it was not fast due to this HTTP as request, but at least no one complains.
I was a little bit paranoid that no one complains, so yeah, around ten days, so I went
to this service, this service is named Grey Noise, and it represents an interesting kind
of observability for the internet, it is named Internet Telescope, what is Internet Telescope?
This is a bunch of sorrows across the world that have some ranges of IP addresses, the ranges
could be quite big, and they just listen to anything that comes to these sorrows.
They open every port, they lock every UDP packet, and they just listen, listen and collect
this data, and if someone does a massive scan, or the internet, this Internet Telescope
are likely to get this noise, so the sorrows is named Grey Noise, and I want to, and I found
that it detected a DNS over HTTP as a scanner, so probably me, but then I checked the
information, and apparently it is not me, this is just every time something happens, and
I don't have to worry, and even if I was detected, it does not mean that I'm a bad guy,
I'm not a bad guy, trust me, actually I did not have to do this scan, because there
are open datasets of historical DNS scans, one of them were available, was available
from the project, the scanner, from the company Rapid7, unfortunately they no longer host
this dataset, at least they no longer give it for download, so at least I did an interesting
exercise, so let's take a look at this dataset, 3.69 billion records, it is not 2 in 32,
because some addresses are resolved, the dataset size is 13 gigabytes, and this is not even
parsed, a compression ratio is 65, the raw dataset, if we take just these JSONs, it is almost
a terabyte, and to parse it I used this simple SQL query, let's not read it, regular expressions,
JSON, etc, and here is the parsed dataset, time status, flags, IP address, and domain,
IP, let's take a look inside, the columns are compressed quite good, and the parsed dataset is
even less, not 10 gigabytes, but 5, so I can look and it looks plausible, but now a question for you,
do you know what is the most popular TLD top level domain, you say dot com, how many people will
say dot com, dot arpa, maybe, how many people will say dot net, let's take a look at this easy, dot
net, dot com, and dot arpa, either I just removed it from, but probably it is not that popular,
I'm not sure why it is not here, it should be here, we will take a look, so dot net is the most popular,
now the question for you, what is the most popular second level subdomain, or first level depends
on how do you count, go, what, go you k, go gp, any other options, net, what, maybe, let's take a look,
I think you all, you are all wrong, and the second for some reason, comcast dot net, and the
self is bb tech, what is it you know about, but probably you know, okay, okay, now the next thing I did,
is I tried to find all the reverse DNS records, containing a couple of files, the name of my favorite
database, and I found a lot of them, a lot of them also with company names, and what I did,
I sent this list to our sales team, actually I could use different tool, what is it, what is it,
so then it shows 23,000 clickhouse servers are visible on the internet, probably because they are
not security, like it was with deep seek, but probably for some other reason, like they are security,
but handshake goes, to them currently it is 33,000, and I want to get more from this data,
that's how I need visualization, and the most obvious example is to make this map, this picture is
from 2006 from XKCD, about representing the whole range of IPv4 using a space feeling curve,
so the curve does not go like the scan line, but the curve that goes like this, feeling all
the space, it is difficult to say, but better to visualize, and also another example also not from
me, it is from 2018, almost exactly what I want, but I want it bigger and better, so I have
made another one line shell script, it is this, I will draw a picture with SQL, and this SQL query
will calculate first significant subdomain, like ABC.co.uk, ABC will be the first significant subdomain,
then calculate hash, Cb hash 64, then from this hash extract 3 bytes, RGB,
and then use another function, more than decode, that represents a space feeling curve,
and it will generate these pixels, and I will make it like a text in the format name of
that portable net map, PNM, and after all of this, I converted to PNG, and it looks nice,
here is the picture like I wanted, and we will dive deep into this picture in a moment, but now,
yeah, this is even more, it is zoomed, but I want anyone zoomed out, so I generated it with 4k resolution
using that script, actually I want a full picture, and the full picture will be 4 billion pixels,
65k by 65k, so not 4k, but 65k, and 65k displays don't exist as of this moment, but I can do it
interactively, yes we can generate 4 giga pixel PNG, but actually I tried it, but when I tried to
open it then, most of the viewers were failing, actually, some were not failing about that slow,
but most were failing, and then I decided to generate an interactive page, a HTML page,
like Google Maps, allowing to zoom inside a lot of tiles, and there are different tools for that,
like OpenSeedRagon, and LiftLag, LiftLag is more used for maps, and OpenSeedRagon is more used for
something like scans of museum, museum pictures, drawings, and it appeared not too difficult, so I
have written another script to generate tiles in a different zoom levels, and here is
several lines of a loop for zoom levels, a loop for x coordinate of a tile, for y coordinate of a
tile, fine name will be like here, and here is the same SQL query to generate a picture and
convert to PNG, done, done, done, I left it for a day, and it finished, now I have a bunch of tiles,
and I want to arrange them into a source, and the source is here, let's take a look, so here it is,
here is our map, we can zoom inside, we can zoom even more, let me try, what will happen if
to this, why let's, something, do you know what it is, could it be, could it be tried to guess,
come cast, let's take a look, no it's Amazon, but what is interesting if I zoom even more
inside this violet thing, I can see a lot of different dots, and the question is why inside
Amazon, I have this different dots with a reverse name, defined it specifically, and the answer is,
yeah for sending emails, for sending emails, okay let's zoom back, and it is actually so beautiful,
I spent a whole day just looking at this picture, now let's go back, and see, so every click also
generates a SQL query to this table and from JavaScript it is easy to do, just use this fetch
request with select query inside the post, HTTP post, and it works because I have the primary
king's table, it is a pointer request, it is fast, and to make a source I used, I created a user,
I set up limits, Quattas, read on the access, and this HTML page, from JavaScript directly queries
this service, take a race, please don't be afraid, you can use clickhouse for data analytics,
you can use clickhouse for DNS analytics, and you will find something new, you will have a lot of fun,
maybe you will find creative way to test and break something, the source code is available, it is
really easy, HTML page to JavaScript, and that's it, ah, by the way we have a dinner today,
but unfortunately it is full, so please don't look at this one, thank you