All right, everyone, I'm Matt Swazo, I work on the Google open source security team.
We're speaking about build auditing as part of the OSS rebuild project.
So let's just rebuild aims to reproduce open source packages.
The focus on language package ecosystems.
We currently support PIPI, NPM, crates, Ruby, gyms.
Thanks to one of our organizers.
And Debian as well.
So we provide evidence of our reproduction,
the form of a salsa at a station, and include our build setup in the at a station itself.
So the public users can run that themselves.
This process happens out of band from the actual main team or upload.
And the sort of first-party Providence publication pathway.
He's another P word, which is nice, because maintainers then don't need to modify their build release process at all.
They can even release from that laptop if we can rebuild it.
They can have an attestation from a trusted builder.
And I guess first, while I'm up here now, it was just rebuild is not just me.
I have a whole team of engineers, both.
On our team and external collaborators, and would not be up here.
The project would not exist without them.
So to get more into sort of what it looks like.
And I have made this as hard as possible to see.
Imagine you can see the boxes and arrows.
But the architecture is a complaint.
So we have our upstream source going into our build.
We get the upstream package.
We compare them.
The stabilise comparison.
And publish them as an attestation.
The whole slide show will be like this.
So I guess, yes, let's get started.
That would be incredibly helpful, I think.
Could have checked this somehow.
Oh, geez.
Oh, boy.
OK.
Yeah, I mean, yeah, hopefully you can see the boxes.
Just want to look at exactly.
Please use your imagination.
So with the sort of attestations we're publishing, again,
this can be in addition to or instead of a maintainer publishing
their own first party attestation.
So when an existing one is present when the publisher has
provided one, we can corroborate it.
And as we'll be this object of the talk,
we can also augment it.
So adding detail that maybe was not present during the original build.
Maybe going to an even higher level.
So what is the trust sort of relationship we're talking about here?
So we have this source code here, hopefully understandable,
we have our build environment that we trust.
We think is trustworthy.
And with some effort, obviously, the equal sign is doing a lot of work.
But we can get a reproducible artifact out the other side.
But like all models, it's not accurate.
It's just convenient, maybe.
Neither the upstream source nor our build environments or inherently trust
worthy.
So while a convenient assumption, it's certainly not a safe one.
And as you actually show of hands, who has seen a presentation in the last
two years with XE, the XE attack in it,
who has seen three, five.
OK, so I'm going to similar about, but for all of you who
have only heard it once, let this be another opportunity.
So suffice it to say, three years of social engineering
sort of culminates with this nice, elegant, effective,
scary attack on ultimately Debian's SSH package offering sort of preoff
remote code execution, the maintainer gained control of the project,
and was able to backdoor XE utils.
So the backdoor itself is split between the published source code
underhanded, again, a lot of elements of it were well-offuscated,
and the additional trigger, ultimately,
was published out of tree, so in the source tarball,
malicious, that enables the build in the Debian build environment,
Debian, and Ubuntu, maybe, probably quite similar.
And that results in the artifact being compromised,
and fully reproducible, can be a project reproduced it.
Great, looks good, obviously not.
So there was maybe an animating moment for our project,
and I assume many others in the space.
And so we sort of restructured, maybe,
OSS rebuilds approach to not just verification by repetition,
but verification by observation, in addition.
And sort of what we found has been, we think,
generalizable across, you know, package ecosystems, not just Debian.
So our instrumented environment sort of comprises maybe three pieces,
our execution sandbox, which is distrusted by the rest of the system,
just does the build, the network proxy, which records
fetched resources without needing to modify the internal build,
which is important for us, because given the scope,
we're not really in the business of being able to run a distro,
basically. And finally, our fiscal analyzer or graph construction,
which comprises the ABPA, EBPF based monitor,
and a data flow graph builder library.
We'll talk about right now.
No, well, network proxy first.
So we terminate TLS between the build,
this proxy component, record the trace log.
We have the ability to modify, in line, the results we get back upstream.
Yeah, but more importantly, we are able to, like, say,
record everything coming out of, you know, NPM install.
And another, I guess, a bit, nice, is that even if the build is launching
separate containers, we actually hook the container creation process
in the proxy itself, and we can add in the trust route
and the network configuration to that container launch,
so that it passively sort of adopts the path we have built through the proxy.
Just, you know, nice and necessary when you need it,
but for Debian, for instance, you don't.
The CISC graph SDK recently opensource, but is our library for turning
the events generated by our EBPF monitor into the process and data flow
graph will be talking about, yeah, pretty straightforward.
The graph compresses efficiently, it's very important, so we're
instead of talking maybe gigabytes, talking, megabytes still,
but we get for it is a process and file system trace that's
tremendously useful as a basis for analysis.
With this sort of exhaustive view of the build, like, who cares?
But even beyond, say, detecting XE, what we found was that it has proven really
valuable for actually working towards reproducibility and achieving reproducibility.
So, start there.
Much ink has been spilled on dependency management and lock files,
some sure everyone in this room is acquainted with and our experiences have been
no different dependency manifests are convenient
fictions and really, when we want to know what's going on in a build,
especially portably across ecosystems, there's really no better place than
network trace itself.
Base, not best there in precise at worst.
Let's take a look.
Okay, what version of XE, do you think that is going to fetch?
Well, it really looks like that doesn't it?
These are trick. It is 0.8.8.
Not only does it upgrade the patch version, it also upgrades the minor version,
just fun. Let's pick on someone else. This is great.
What version of Guava do you think we're going to be using?
31.1, obviously. Just like we asked.
Okay, so what version of tools are we going to use?
Yeah, trick question. It's the same trick as all the rest, but it is still a trick question.
Yeah, it is. Just the record of the last person to have built it.
So 0.3.8.
Another fun tidbit we can get from this sort of privileged position we have in the build is
that many package ecosystems will do creative things say during insulation time.
It's a varying degrees and one end of that spectrum is NPM and maybe the most
permissive ecosystem. So our network instrumentation will tell you that on this call,
we'll see all of these. Okay, how many people can read it? All right, whatever. It's more than just
registry NPMJS. There's this big GitHub release and this long URL asset.
That is dynamically accessed based on the local installation environment at runtime.
Which is nice because we get, you know, platform specific.
Minorities, but this is out of sort of the normal NPM model, right? If I have an
attestation say for my NPM package, that doesn't include these, right? What are they? Who knows?
Are they guaranteed to resolve, continue to resolve to the same artifact? No, hands. They don't
count. Yeah, and this is the snippet in installed JS in the NPMJS pre-built. That does the
30 work. So shifting gears to the CIS graph for reproducibility, there's not a lot of tribal
knowledge in debugging reproducibility issues. And detailed execution tracing found allows us to
encode more of that knowledge than we could normally into something that maybe resembles a
winter or a set of rules that might be able to guide you through debugging and issue like this.
This is generally easier than source code auditing because the problematic cause of a reproducibility
issue often far removed from its effect. So you may be configuring the environment variables
sort of centrally and exporting them through your cost deck, maybe not obvious. So at the site,
what's happening? Maybe to go through a couple of these, reading CPU info is not wrong,
it's not bad, but it is often an indication that your build is doing dynamic things based off of
the core count or scaling parallelism of the build in response to the machine that's running on
if you want to add a different machine, that's where you get into problems. So this is a frequent
thing where you can't often just set these parameters and rarely build, we'll just try to do it
themselves. And therapy dragons. Going to this next one, locals, huge sources of issues, especially
with make where things like the sort order of arguments into a linker depend on the sorting
your local supports or permits. So seeing things like LCLC should give you the warm fuzzies.
The file prefix map similarly, that's just saying don't include my local build path in
the debug symbols or in the file macro. So you'd like to see that on these command lines,
those without it, that could be a cause of issues. The bottom one, I guess, I guess on the flip side
of the parallelism question, J being set anywhere could be an issue, then put it red or yellow,
but it's just something to consider. And instability graphs work great for this. You are
recording what's happening in the build, comparing them across multiple rounds, the same thing,
very easy to identify non-determinism, either in file accesses, argument order, that sort of thing.
This is, yeah, it's been tremendously useful. Okay, and now tracing for detection, so
consider the XE case. Now that sort of instrumenting the build, self-test, reproduce it,
looks good, but how do we prevent XE? Why is XE's build reading a test file at all?
You'd never did this before. Why has it started doing that during the build process? They
with tests disabled. So this is now really fixed out like a sore thumb. We can pretty effectively
find these test files. And again, you find a really sharp increase. Well, from 0 to 3,
but that's as good an increase. You love to see it. Very obvious. Similarly,
I can't really see this, but why is head writing out this .0 file? That is also sore thumb territory.
Glaringly apparent, maybe with the benefit of hindsight, but an easy rule,
.0 files come from compilers. Let's keep it that way. Declaring invariance like this for build behavior
can really point us towards this anomalous or buggy, but in any case, suspect build behavior.
Right, so what I was alluding to before, we can generalize to sort of comparing the two adjacent
graphs of a build and trying to figure out sort of what is interesting about it. This is one of the
more complex changes in version over version, not major version over version changes in
deviant packages that I could find. It does change the build process, but in terms of summarization,
mostly path and version changes that are easily sort of structured. There's no fanciness here,
really, that's going on. It is just comparing the two graphs. Yeah, so the key insight here,
being that the flux between versions is often low enough that we can manageably identify
variants when it happens. It's not shown here, but file accesses can similarly be
dipped between builds and, well, if you add a file, you expect that file to show up in the build.
That's not surprising, but you're dealing with a similar ability to sort of model and
establish expectations for how some change in the repo affects the change in the build.
Unnecessary behavior. So, may have been radicalized on this point, but we just don't do anything
in a build, except build. We don't need tests. So, the Phantom JS dependency downloaded, you know,
thing from the internet is never going to run in a build, but NPM doesn't, you know, differentiate
dev and test dependencies. So, we want the build, you know, which requires dev dependencies,
we're going to pull the test dependencies down. With CIS graph, it's kind of a double edge
stored, it is easier to identify these cases of Linter's test triggering, but the noise causes
more difficulty for analysis. Imagine running ahead of this browser and seeing how the graph
lights up and how to reason about that change over version changes. So, please leave CIS to CIS
hard enough already. And thankfully, Debian has recently last six months made a push for packages
to actually observe the, please don't run tests flag. A lot of packages just chose not to. So,
thank you if you're doing that. And looking forward, runtime interpreters are a big challenge.
What you saw was actually with a bash, a shell hook that we added into our EDPF monitor. So,
it's not impossible to do this. We are sort of trying to scale down our, you know, the granularity
of our ability to get at the behavior of the build. Run time interpreters like Python, the JVM,
make that challenging or opaque. We've had some initial look, but this is a, I think, a very interesting
future direction for build instrumentation. I think there's a lot that can be done there.
Build complexity builds are software that have to do a lot of things, right? You have to build
portably allow configuration parameterization and it needs to be usable to the end user.
But really, we don't need that. We just need the build to do like the, you know, hundred things
that we tell it to. So, if we maybe project that down onto a concrete build, think of it maybe
as like unrolling or materializing a specific build, I think that's an interesting area to go
where we might not have the constraints of it needing to be approachable or easily manipulable
by humans. And maybe a last direction that we find interesting is having a sort of golden
graph, a golden image of the build that we can use to guide sort of porting or packaging,
say, into a distro, a piece of software. So, we have this re-implementation, right, or, you know,
a wrapper at best that we need to do to get a build into a distro or into some new context,
having a detailed description of what, you know, a good build looks like for that thing
is really useful as a sort of mechanical way of evaluating maybe your progress towards that
endpoint. And with execution traces, we really have that ability.
Wrap up. Excited by these, you know, enhancements across all the ecosystems we support,
no, OSS rebuild largely right without any involvement of maintainers, they don't need to,
they can just sort of get these detailed data about their build without, you know,
them needing to modify their CI, their release process, anything. Prospect of this sort of
zero cost improvement is really what interested me in starting the project and we'd love for
anyone who shares this to get involved repo, docs. We don't and can't make all of our data public,
but if you are interested, we are very willing to collaborate and share that,
much easier at a more human scale to share it than just to stick it on the internet as we do
at a stations. We're also interested in how you all can apply these tools and methods to your
own areas of interest, everything is open source on OSS rebuild, please check it out and
I believe there's potential here and we're eager to see what you can build and rebuild.
Questions?
Yeah, sorry.
It's difficult at sort of a large scale to differentiate between things that are
objectionable and things that are like bad. We've definitely found lots of objective things
the last sort of illusion there to future directions and it would be really interesting to try to
remediate that badness with what we know. Please stop having this effect happen and try to
try to set that as an endpoint, but yeah.