Around of a glass, please, for Paul Meier, who's going to talk about, go in the next ecosystem
and some interesting work he did in the area.
Welcome to my talk. I'm Paul. My guitar panel is cut X or HIN.
I'm also a master-done, I'm a security software engineer.
The executors maintainer and I'm also part of the next package of the Scouting.
So if you're wondering why you didn't receive the latest GoVinability update,
I would tell you that it was currently in staging next and you have to wait or apply an overlay.
So I want to quickly give another view of the GoVinability Penance Management in general,
outside of NICS, go and manage dependencies as modules, which is the collection of packages,
that is released versions and distributed together. So this is usually a repository scope.
We have to go that not file that declares minimum required versions of module dependencies.
Let's also go that some file that contains cryptographic hashes for these dependencies,
but to go that some file is not an exact log file, and also the hashes in there are incompatible with NICS,
so we cannot use import from the abbreviation here.
And then you go use this minimum version selection to, which is a deterministic algorithm to select the actual versions that I used in the build list.
And only at the one version of a dependency is selected in the end.
And use a quick look at the GoVinability Mod file.
Here just to get an impression, the required statements declare or direct,
and at the bottom indirect, so transitive dependencies.
So now come to the NICS part, how do we build our package, go, stuff in NICS.
In NICS packages, we have a single build up of this called Big Go module.
And yeah, for most of this might be straightforward, we have a source that we fetch from GitHub,
and we have a package name and version, and what I really want to point out here is the vendor hash at the bottom,
which pins our dependencies.
So we will take a closer look at this, and also at the delay, the relations structure here.
So we have one derivation of which is Big Go module, the thing we want to build, then we have the source,
which is a fixed operating relation that pulls our source from GitHub.
And then we have a third derivation here called GoModules.
And this depends on the source, and use the GoMod file and the Go command to download the dependencies of the model we want to build,
interface output derivation, and then this is pinned with a by the vendor hash.
So if you now look at two different Go packages, pet and pin act, both will have GoModules, derivation they depend on,
and yeah both will have some Go dependencies in there.
And as we see here, now these two packages depend on the same GoModules.
We have the same dependencies, but in next, we do not have any introspection in each of this.
So we will download the same dependency again, and yeah, we do not know that these two fixed operating relations actually have the same content.
So what does this mean for the current status and next message from GoModule, we have to download things multiple times.
We are storing these dependencies over and over again.
Very important, we don't have any form of caching.
So we have to rebuild the dependencies over and over again, both between two different packages,
and also in case we change something about our actual package, we want to build.
There's no caching for the build of the dependencies, so we also have to rebuild our dependencies in this case.
Then we do not have any introspection into the dependencies in nix.
It is really hard to update dependencies of the package, for this we would actually need to create the patch for the GoMod file,
and we would have to do this for each GoPacket individually.
And following, we do not have an easy way to patch vulnerabilities.
So first I want to look at the vulnerability aspect, how bad is this.
I've created a small project, GoModule check nix packages.
GoModule check is the official vulnerability scanning tool of GoModule.
And it has, go has a separate vulnerability database containing CVEs, but also GoModule.
So the vulnerabilities that are not assigned to CVE, GoModule check is quite cool because it checks if the vulnerable code is actually reachable.
So you get less false positive in case you would just import a vulnerable package, but the isn't a core path that will make your code reach the vulnerability.
And then GoModule check nix packages runs GoModule check on the source of all Go packages and nix packages that are built with GoModule.
So what I have the results here, and black we see the packages that are discovered by GoModule check nix packages.
The discovery is currently based on simple attributes of the output.
Then in blue we have the packages that are actually scan tool successfully.
This is just like a really simple project, and for example for packages that you see go, we do not provide the CD dependencies.
So the vulnerability check I will just fade because the dependencies aren't there, so we are missing out on some of those.
And then in red we have the number of vulnerable packages which is quite stable around 30% of the scan packages.
And then in yellow we have the number of vulnerabilities in total.
We have like around three vulnerabilities per vulnerable package.
And yeah just remember we have we don't have like an easy way to fix vulnerabilities, especially not at this scale.
So now I want to to make a jump to the catching problem again and just yeah.
How the cache usually works with go is we have like an environment variable GoCash which points to a five system path and there's our cache located and go will read and write to this path and manage the cache itself.
Really cool feature in go124 and also in earlier versions as experimental feature that lets us provide binary called goCash Proc.
And yeah this gives us a lot more flexibility.
Go will just invoke the binary during the bit presses and talk a JSON based protocol.
He has done that in standard out with two basic operations get to request a cache entry via an external idea and then in case we got the cache miss go where built the package and then called put.
So the go cache port can can write the cache result somewhere.
Now at this point I want to introduce goboot.next which is next generation build up for go.
I use this goCash Proc and just so you get an idea the goboot.next tries to provide the package sets of go dependencies.
Where each go module has its own derivation so we will try to module the go dependency graph index.
And first I want to show how goboot.next uses goCash Proc.
So we have written a goboot.next cache binary that implements goCash Proc.
In the way this works is we have an environment variable where we read a number of paths from.
And then go the cache over will serve the cache entry is found at these paths and the pass will point to the next door.
And in case we have a cache miss the cache address writes the built result into a defined output.
And yeah we can add the stop paths we set up hook so we put the go module dependencies into our built input.
The set up hook will add the stop path to the environment and then the go cache the cache can serve these.
Now looking at this from a derivation level so if we want to build getup.com full slash a.
We have a source in green then we have the source of our dependencies which is required for the build process.
And we also provide the cache so the build output of when we build the dependencies to the cache.
And now the idea is that we will not have to rebuild that the dependencies because they are already cached.
The cache will serve the build cache for for the dependencies and we will only need to rebuild the packages that belong to full slash a.
And for these packages the output of the build process is written to out.
And then the final bit result will contain the cache entries for full slash a.
And looking at this at the package set perspective.
These packages will construct the package set and mainly consisting of the build results.
The cache entries that the cache are stored in the output.
And also quite important is that we have to have the source of our transitive dependencies available for build.
And if we now want to build a go binary we would do that outside of the package set and just add all the built inputs.
As usual so the go modules as built and put and we would have all the dependencies already available and also built.
And yeah every dependencies only available in one single version in this package set.
So we are doing like some overwriting of the of the version where where package depends on.
So what we get here is we get increments as far as fetching we can fetch every dependency on it's own we get rid of this vendor hash.
Also we get rid of the of the bad development workflow that comes with that.
We get increments to build.
For a build process because the dependencies are already cursed.
We get great composability because we're using set up hooks.
We get the possibility to patch vulnerabilities because we only have dependency once in our set and we can then bump that.
And it will propagate to to all the dependence.
That's really cool.
This project targets next packages.
The status is highly experimental it's not usable yet.
I did some research and I figured we will have to have like 12,000 go dependencies or so next packages.
We've got packages to build all the other packages that are currently in next packages.
But now we are like about 30 packages that we are building.
So there's let's do a lot of work.
And we still are figuring out like how can this this work at all.
Yeah and it's a lot of work.
I'm looking for people to collaborate on this.
Both for the the builder the new builder but also for the vulnerability research thingy.
Please talk to me.
You can reach me on networks or here and person.
And then I want to give shout out to all go related next projects.
And I want to point out especially the one at the bottom which is quite new.
And which uses the same go cash product mechanism to serve the go cash from outside the sandbox.
So this introduces some impurity but it relies on go to ensure the correctness of the build.
So yeah that's something that might be usable today.
Yeah thanks a lot.
Thank you very much for the great talk.
We have a lot of time available for questions.
Are there any questions?
I see your hand back there.
So the question is if this still uses the go.mod file.
And the answer is now it does not.
You have to add the build dependencies as built inputs in nx in your next derivation.
And we will also rewrite the go.mod file.
So because if you as I said there's only like one version available.
And this means that when there's like a diff in the version you have in your go.mod file and the next packages.
There could be like 250 dependencies that have changed.
So yeah we rewriting the go.mod file completely at this point.
There is another question here in the file.
I think a format of the question.
Is there a kind of equivalent log file that you already generated?
And next.
Because you have to manually add all the dependencies to your build file or do your query.
Generate a log file like a log file with the idea is that you do not.
And the question is if you create some kind of log file for this.
And then so it's now that here is that first you have your dependencies available in nx packages.
So you just have to add them at this build input.
And then they are pinned as we fetch the source.
And at this point they are pinned.
So quite a lot of work initially.
Yes this card is.
This is a lot of work.
No.
I don't have better value.
But you may automatically respond to a go.mod format.
You can kind of have a tool that takes the load file.
That gives you the next array of the dependencies that you have done.
You can make it.
Yes.
So the ideas that this could be.
There's like a lot of room for automation.
And sure there is.
I don't think we will have to.
Yeah package all these things by hand.
Especially as there are a lot of libraries that are.
We do not have like any.
Any build input that all or we can use the inputs.
The cloud in the go.
Not file.
Yeah.
But we're not there right now.
Quick in between question is Josh.
Can you hear who's going to be the speaker for the next talk?
Okay.
No.
Okay.
Questions about the action.
No.
No questions.
Well, then thank you very much.
Thank you for finishing on time.
Please.
Thank you.