One before last speaker of the day, Michele will be talking about how to figure out if
you're so far as updates by sorting this data in DLS and getting it from there.
Yes.
So great.
So many people are here.
It's mentioned my name is Michele Kin, and from the Netherlands, I mostly go developer
and most of today's talk relates mostly to the Go Software ecosystem.
I've an open source project called Mox.
It's a modern, all-in-one email server that the aim is to make it really easy to install
and maintain your mail server, including getting it updated.
So that's how I got into this topic.
So today's agenda, two parts.
Part one is actually about this using DLS to find, to discover that updates are available.
So first, doing it manually, and then I found a way to automate this for all the go software
out there, and then part two is taking it to a next level, so that only finding the updates
will also actually upgrading your services to run with the latest version of the application
and also automatically compiling binaries, so you can actually get those latest software
deployed.
So let's start with the manual updates through DLS, so in Mox, there's an option, and
I urge always admins to enable it, that will check every 24 hours to see if there is
an update available.
So that's this endpoint, underscore updates at XMox.nl, if you create it, you get essentially
the latest version back, and if there's a new version available, and Mox compares it, if there's
a new version available, Mox creates an enomestate, and delivers it to the postmaster
mailbox, and then the postmaster, the admin, can update the installation.
So that means when I do an release of Mox, I update the DNS record.
So that'll work.
Now I think the DNS is a pretty good fit to do these kinds of checks, because it's very
low overhead, it's just a little bit packet, and that is a funds back, and of course DNS
provides the caching redundancy with another service, and what I like especially is that
it's not so easy to track, which installations are out there, because the Mox instances
will likely go through a network recursive resolver, and that can be many machines behind
one resolver.
You can do other stuff, it's the PS request, but then it's more overhead, and it's more
centralized, you get more options for tracking.
I don't like being tracked, I don't like making software, and that makes it easy to track.
Now then I figured out, there's a way to automate this for all the goes software ecosystems.
So there's a lot of text on this slide, I want to have this with point nine point, but
I just tried to explain this, so all of those software is in a transparency log.
So we know that transparency log is a few people, we've heard about certificates transparency,
much more people, so certificates transparency is essentially a transparency log with all
the T-less certificates that have been issued.
So you can monitor that and see if your domain is in there and you didn't expect it.
The transparency logs are append-only logs, and you can cryptographically verify the
entries that are in there, and you can verify that it's actually append-only, that it's
working correctly.
So for go, the go ecosystem has a transparency log called a go-sum database, and all the
go software, both applications and libraries are in the transparency log.
Okay, something I want to mention, go software is identified by a go module path, so
they're packaged in so-called modules and they have a path, and you can have multiple versions
and there's semantic versions, and the transparency log also contains the hash of the source code.
So it's used for supply change security.
If you want to release your new go module version, you make a get tag and you push it or another
version console system, but it's the most popular one, and you push it and then it gets
discovered and added to the transparency log.
Now the thing was, I already had the surface code go for watch.org, it was already monitoring
the transparency log, so it's a pent-only ledger, and you can just follow it, and I was already
putting all those entries in a database for easy lookup.
So to automate the DNS records for all these go applications, I just needed to add the
DNS server to go for watch.
Now here's an example of creating the same application of mine, through DNS, now you
have a bit of a longer name, DNS name to look up, so essentially it's go for watch
log at the website, then L is the delegation to go for watch application instance, V0 is
the versioning ID, so if I want to change the responses and the format of the left part,
I can just make a V1, so that thing, that's a go module path, we call it, but it's
actually more like a URL with a host name and a path, so I just put the original host of
the URL in there, then a bare underscore to the limit, the host part from the path, and
then there's the split path elements with the special dash there at the end, which is
kind of a special encoded with an underscore in the access, that's more value of the
S key value, and that is more, so this whole thing, if you look that up, you get the same
information back, the latest version is 0.015, and there's also timestamp, access, the time
it was discovered in the transparency log, so when it was published, you can use that to
delay your updates a bit, we'll get to that later.
There's also the go tool change, so new compilers and standard libraries are also in this
transparency log, so you can also query that, you've got multiple semi-colon separated
values, there's the current latest release, then there's the previous release, the go
has released every six months, and two releases are essentially kept up to date with
patch versions, and there's also a third one that's the release kind of that will be the
next version, so you can also query that.
Then, this go for what's DNS server, it's built with this great library, mini g slash
DNS, it makes it easy, just keeps me DNS superpowers and go, adding the DNS server
to go for what was less than 1000 lines of code, and one third of that was parsing configuration
file in cryptographic keys, because it also includes DNS sec with online signing and
compact the null of existence, so there's like 600 code handling, a UDP request, TCP connections,
and handling the right messaging, ED and SDR, all that stuff, so this is, I think it
really nice about these libraries and programming nowadays, with the go ecosystem, it's
not easy to pull in these libraries, and so be safe, feel safe, and doing that, and just adding
all this functionality to an existing application, now you may wonder how, how actually
does this transparency log, because some database get filled with all these modules, with
all these applications, well the go tool change, so the go compiler, that all the developers
use, they essentially go through, they ensure that these entries get added, when you use
them in your application, so when you install a go application, you'll usually install
the latest version, and you'll go, if a compiler will go for the go module proxy, that's
something the go team runs, and that will resolve the latest once in a while, by going
to get, seeing if there's new get tags, and if it finds new ones, it downloads the code,
and it adds this hash to the transparency log, and the same happens where you use an external
library for the first time, by the way, you get the latest, that's resolved, and the new
entries automatically get added to transparency log, so it's pretty much like always up to date.
Then, go for what, so the DNS there's not that, not yet done, so actually I only have
one instance, so I want to get this redundancy, need to run another go for what instance,
it's very easy, just a static go binary, and it'll run, but haven't gone around to it yet,
I also want to test or first somehow this DNS message handling that I've implemented, so
I'm sure that I didn't get everything 100% right, so if anyone knows, test suite, that
I can run against this, let me know, and then I may also want to change these responses,
so currently they're just very small text responses that you get from the DNS server, but
I could actually include this transparency log proofs in there, so that you can, you can
apply into doing these requests for itself, verify that these entries are actually in the transparency
log instead of having to trust the go for what's a service, but there's, it gets a bit complicated,
there's also new stuff coming out within the transparency log scene called something called
verifiable indexes, and that allows you to actually check that this is indeed the latest entry,
not just I'm entry that's in the transparency log, but actually the latest one, but that gets
complicated quickly. All right, that was the DNS screening part, now we're taking it one step further
with automating upgrades of the software, so I've got this separate tool called iSCO,
that allows you that will upgrade to your software for you, you'll go services,
and it uses this other service that the builds go builds.org, that will create binaries on
demands from the source code. So, automating upgrades with iSCO, so instead of running your
those service directly yourself in your system D file or your RST script or whatnot,
you make iSCO start that, so it's like a supervisor demon kind of thing, so it just starts the
application, but it also at the same time starts monitoring the go for what's DNS endpoint for updates,
and when it finds an update, so that's the querying every 24 hours, it also queries
for new go tool chains, so in go binders are statically linked, so if there's a security fixed
to the go compiler and instead of library, you want to update your application to include that fix,
and the go standard library is large, it has its piece server, et cetera, so many applications
use that, so you do want to get updates into your application. So once it's found an update,
it can upgrade by downloading a new binary, terminating the existing process, and restarting
the application with a new binary, it's just simple as that. Now, iSCO can do that just with one click,
so as the admin is the one click to do the update, iSCO is a small web interface, we'll look at
the screenshot in a moment, so you can there's also Prometheus metrics in iSCO, so you can
get an alerts when there's an update available, but you can also just have iSCO completely automatically
do the upgrade. Usually, i delayed a little bit, so if there's a release, i'll wait two days
to do the automatic upgrade, because you have to maybe a quick release after to fix some last
minute bugs that were found, they can set a policy of what to update, so maybe only one to update
patch versions automatically, and do the other ones, the minor version updates manually,
there's some more policies, but it gets complicated, or i don't have time to explain it,
and you can also set a schedule that you only want updates, automatic upgrades during working
hours or outside working hours, so just to make sure you don't get broken stuff where you're not
available, so the great thing about this approach is that no modifications to go applications
necessary, this works for all the most go applications, there's a little caveat about
which go applications it wouldn't work for, but again, bit too much detail.
This is screenshot of the web interface, at the top you can check for updates now,
so after i do a git push of a new version of my own code, and i want to upgrade my
other services, i click that, and i got a sometimes update scheduled, you can select a different
version and different go tool chain to build with and just hit the button, and it will
get a new binary in update at all, that long version is like a pre release version,
essentially just for a specific git commit tag, so when i do commit due development,
i can also easily update my development environment for a specific git tag.
At the bottom, you can see that ice code itself is also managed this way, it has a version,
and it goes tools in that it's compiled with, and it can also update itself, get a new binary,
re-execate itself, so it doesn't have any downtime. So everything stays up to date,
automatically with this, then the go build to talk service, it compiles source code on demand,
so when there's a new git tag pushed in the ecosystem and discovered, you want to binary with
that for that source code, what you get to binary from, some projects that publish binaries, but
usually don't really release them when there's no new go tool chain, so you've got binaries,
maybe provided by a project, but only with old tool chains build, so now you can always get
a fresh binary, so the source code that's download is fairified with this, and it goes some database,
the transparency log, so let's how you know that go build is using the correct source code that
everyone else is seeing, the service also automatically downloads and fairifies go new go tool chains
when they release, the cool thing about go is that it allows you to cross compile for any operating system
architecture, so that makes it much easier to provide this service for the entire go ecosystem.
In Go, your builds are reproducible, so you get byte for byte, the same output, the same binary,
so you can verify that this is working as intended and you're not getting some different binary,
and effects go builds has two machines, different architectures are different set up,
the build same thing, and go build checks if the output is indeed the same, and when that's the case,
it adds the hash of the binary to its transparency log of its own, so for transparency,
so we can verify that the binary you're getting is sort of publicly vouched for,
and you could monitor this transparency log and also verify that these builds are indeed
correct, that you get the correct binary, so this adds trust, and this, just otherwise,
I would be hesitant to download binaries from the internet and start running them on my service,
but this way you can verify that it's the correct binary as expected.
So another cool part that next to me at ease in doing this is that the Go tool chain,
when you compile code, it only runs its own compiler code, it doesn't run any code of the project
that it's compiling, so there's quite a bit less danger in compiling random codes that you find
on the internet. Now, how are we doing in time? I was just about to put up the tenant.
Okay, oh, well, then you've time, probably if we got half of the stuff I wanted to say,
do we have a browser? Yeah, we do. Now we can look at this Go build service for a moment,
so it's just a website. Can I make it a bit smaller, you need a bigger? Yeah, I can.
Yeah, there you go. No, no, that's not it.
Which green thing? Green thing, then I'm going. Yeah. All right, so here you can see some recent builds.
So one of the problems I'm having at the moment, AI scrapers, they scrape the website, they follow
all the links, and they build all kinds of stuff. They're hard to block. I don't like them.
My phone is lighting up too much now, but let's see, I'll just click one of these,
which is all all stuff. Anyway, you can you can see from the URL also,
can see from the URL what what this is, right? This is a specific application. This is the
Go build service itself, specific version created for highlighters, created for a freebiesd,
AMD 64 with a certain tool chain, and this thing here at the end is the hash of the binary,
so you can send links to people. They can download the binary from here, then you don't get
the advantage of having yourself verifying that this binary is in the transparency log,
but it's still useful service. So for my Go project, I don't provide binaries at all,
anymore, it just points to this service, and people can just automatically download the latest Go
binaries at the time. So that's there. You can, you can, in fact, download it with this
Go build command line tool and verify that's the correct sum. You can also reproduce this
build locally, so this is what Go build is running. If you do that yourself, and you verify,
you can verify that it's the correct. See, there's a snippet here that it's
this Unreal will. It has a Python snippet, so you can verify that the hash is correct.
And here at the bottom, so these are the links that these AI scrapes are following,
because you can get any module version for any supported operating system architecture, for any
Go compiler, and the default build or strips build with the debugging signals. Let me see if this works
if I know. No, no space left on device. Yeah. This is, this is something I have to fix soon.
Let me get back to slides. Yeah, so this is already getting to the end. The summary is,
I can get for any Go application, one click, or fully automated upgrades.
Without changes to the application, and DNS is helping me discover these at a very low overhead
cost. Isco is the supervisor service that does that, and Go build is giving me the binaries.
Then, one more slide with all the source code. Of course, this is open source. You can run it yourself.
It's pretty easy. And here is a blockbout with some more details about ice going, what it means.
You know, if it's a good idea to do this at all, it raises some questions.
So if you're interested in this, please read that and contact me by email.
So that's it for now. Yeah, well, this is was quick. Thanks.
Thank you.
Any questions from us here?
There's a question.
So questions, how do I handle low names? You mean DNS, like a packet names?
Yeah, so how do I handle low module names? I don't. So if you have, if you think, it's a wise
idea to have a really long project name, then you cannot query it's over DNS, and it's
just, to be honest, yeah, the query will just never come in, right? Because if the query
would come in, I would answer it, but DNS will not allow it. So yeah, that's, so do you have, by any
chance, long, if we just say, like, more, not a question.
All right, let me add, actually. So I've recently built something like this, and on top of that,
you're just not leaking what's installed on the machine. So I do have a solution for that,
you're using lip-soluble and encoding it with several public key. So I'll, all right, yeah.
Oh, interesting.
Do you do the origin one? Because it's very lightweight. Yeah. I also need to add, like,
replay production, and stuff like this. So if I understand correctly, you say, I'm leaking that,
by doing this DNS requests, I'm leaking what I'm, I had installed, because I'm asking for a certain
version. Yeah, so the thing is, the solution, and I have the solutions, though. Yeah, yeah, yeah,
yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah, yeah,
it's so much to do list for having a, for, for the redundancy. And it's very, it's very simple.
This go, this go for what's thing, it's just a static binary can run it. It doesn't,
it doesn't need much, it just needs incoming GDP requests. But yeah, I need to set up a machine,
and run it, and time is limited. Yeah. One more question?
Is it possible to distinguish between the security and the structure of things?
Is it possible, all the questions? Is it possible to distinguish between security,
feature updates and security updates? And no, I don't think, not really,
but you can assume that patch updates of go applications are security updates, maybe.
I can have, there's no meta information about a release of whether it is a security
release or a feature release. Except that go applications are supposed to, I mean,
so, be semantic recursion, but do developers actually do that and practice not always,
so there's no way to really know for sure. So if, if I'm, I'm using high school to deploy a specific
application that I'm using, I can check, I kind of know what the developers are doing. So maybe
I know that some applications are safe to do patch updates and assume that they will do a patch
update of their security release, but there's no guarantee that that's the case.
So yeah, the answer to the question? Yes, so I understand now, because the meta data is not there,
you cannot provide that. Indeed, so there's in the go binary, there's the go module path,
so this identified as a version, and they go to the version that it's built with,
but there's no information, and also I kind of, I don't think there's a public place,
and maybe there's a maybe, maybe I could go to some place and see if there's an automated
way to get security vulnerability. Probably there is, maybe, maybe I should look into that. Good question.
Yeah, so is anyone, would anyone, like, happily run this, or do you think it's a bit scary?
Yeah, as a bit scary. Yeah, a bit scary, yeah, why is it scary?
But you're, so you're afraid, so you're afraid, essentially, that I'm leaking with applications,
I'm running on a network, even though it's behind the recursive resolver.
Yeah, and what are your fate of then? What would, what would it attack me?
Yeah, but you're running on an untrusted network.
Yeah, so the fake answer you could send is, well, this data's exciting, right?
So you could, no, no, so I'm sure, no indeed, so you have to trust the resolver,
but I don't, many people would trust, I think, the resolver, but maybe not.
But then with this, that's why I had this idea of adding this transparency of proofs to the responses,
but then still the attack could be just to drop all those DNS requests and then you never update.
So if there's a security update that you would get, one to get, you're not getting it.
Of course, if you're so security conscious, maybe you also want to monitor and manually deploy your
applications anyway. For me, I'm using this more as a convenience way, because I noticed that I'm running
all by myself, I currently running 35 internal services, not very important, although also my
mill service updated this way and go watch and go build their all updated this way.
It's just very convenient to stay on the latest go build, because otherwise I have to
set up, I guess this is what you do for your serious applications. You do set up a CICD pipeline,
lots of YAML, build everything, deploy everything, blah, blah, blah, and so I noticed I'm not doing
that. Of course, I'm using my own software, but this is for me a way to stay at least up to date with
the latest security fixes also in the standard library of go. So when I would be afraid of more,
I would suggest, I would expect someone to say that you're running, I'm running application,
that I've not been running the test suite, because I just built the code as published.
I'm not, but that's why I wait two days, so, before I deploy it, it's like, if something is wrong,
I would probably hear it by, see it somewhere by then, and then maybe there's any release by then.
There's not a question behind the back. Come in.
You're the ask queries, but then to create the fingerprint. It makes me, makes it possible to be
to track the behavior of that system. So, yeah. And this is for all the software applications,
if they're timely, you typically have a number of applications that ask CICD to leave,
or update information, and that will provide a fingerprint from which I can track you.
Yeah, track, you can use this, these Dnesia requests to track what is installed and what's happening.
I call the other genus, so I can correlate your fingerprint.
Yeah, all the other genus queries are in the lab system.
Yeah, yeah, that's correct, yeah.
No, I would have not expected that to be sort of a concern for people that's tracking.
Yeah, yeah, okay, interesting. Thanks.
All right, thank you, Miguel.