Hello, everyone.
We're going to start the second talk for the package manager
Devroom. My name is Exigner, and we're going to talk about the
current state of attestations in programming language ecosystems.
So first, we're going to talk about the two attestation types that
these ecosystems have been adopting over the past three years or so.
We're going to go through a tour of the different package repositories and
how you can get this information from them to use in your own
security incident response.
Most of these implementations are very similar, but there's also a few
differences to call out.
And then we're going to talk a little bit about sort of like the
near-term future and the longer-term future with the attestations.
What are the sort of like next steps and what sort of future
direction are we looking at here?
Okay, so first of all, we're going to talk about build
provenance.
As far as I know, this is a net new capability, security
capability across ecosystems, and it provides a very easy way for
us to connect a package with a specific hash to the exact
source code, build instructions, and build logs in a
non-falsifiable way.
So this is really a tremendous tool that we can use for
instant response.
You have some binary on disk, and you're like, where the
heck did this come from?
What's the source code?
How is it built?
Can I go and look in that for malicious behavior?
We almost all of the ecosystems are using salsa build
provenance, so the specification for it can be found on
the salsa.dev website.
And then the other attestation we're going to be talking about
today is release attestations.
This is more recent, and so it goes by a couple of different
names.
Several ecosystems call these publish attestations, but
this is what we traditionally think of as the job of a
package repository.
We're mapping a specific package name, a specific
version string, and we want to know exactly what
packages we should be expecting with the hash, so that
we know that they weren't tampered with.
I do a lot of work in the NPM ecosystem.
There's a one-to-one mapping.
If you have a package name and a version string, it's just a
single package, but many other ecosystems provide the
ability to have several packages.
Maybe you're building for the very popular doors of
11 ecosystem.
And so build the provenance or sort of release
attestations doesn't require a one-to-one mapping.
It can be a one-to-many mapping, as many different
artifacts as you want.
This release attestation is defined by the
Intodo project.
We'll see that some ecosystems have their own
definition for publisher release attestations, but if you
are looking to use this in the future, I would really
recommend using the Intodo attestation
predicate type here.
And if you follow that URL, there are sort of like
details on the specification and
invitation, that sort of thing.
All the must-shoulds and nice to have.
Okay, so these two attestation types, they're like
different, but they're also very linked.
So if you put them together, you can follow this sort of
really interesting chain of information.
First, you could say, okay, I want to know this package
name, this version string, what should contain?
I get back a list of hashes.
You can take those hashes and feed them into
build provenance, and then get back the exact source
code and build logs for them.
So, you know, these are like very similar, sometimes
confusingly similar, but really the two of these
capabilities are meant to link together and sort of
provide you this trail of evidence that you can follow to
figure out how a given package was built.
So, I thought it was really funny that Adam was talking about
an attack on PIPI.
I'm also going to talk about an attack on PIPI.
This is not at all to throw shade at PIPI.
The reason why we keep bringing it up is because
they do a really excellent job of writing blog posts
with instant response and talking about not only the
attack, but the mitigations that took place.
So, Adam was talking about an attack in 2025.
This attack was in late 2024 against a package called
ultra-littics.
Version 8.3.40 was a normal release, and it had this
build provenance information, and then the community
noticed that on the source control platform, someone
made a pull request and the name of the branch for the
pull request looked like it was an injection attack.
And sure enough, when they went to PIPI, there were two
new versions of this package published.
Version 41.42.
It still had build provenance, but it also included
malicious code.
And the community was able to determine that quickly because
of the build provenance information.
They were able to look at the source code.
They were able to look at the build logs and the build
instructions and find that malicious behavior.
So, then the community, this package was able to
regain access to the package in the source control.
But during the attack, the attackers also
exfiltrated the PIPI API keys.
So, then versions 43 and 44 were published directly
with those exfiltrated API keys.
So, did not have a build provenance.
And so, it's important to note here that just because
the package has a build provenance doesn't mean it's
not secure or if it doesn't have build provenance doesn't
mean it's not secure.
All we're providing here are links to source code and build
instructions.
You still have to follow those links and make your own
sort of like security decision.
However, it is very suspicious when a package is using
build provenance and then stops.
I would call that a yellow flag.
So, again, I'm not saying it's, you know,
bad or malicious, but that's definitely a signal that
is worth following up.
And so, then after the maintainers, we established
control both of their source management and also
in PIPI, they published version 45, which
should not contain that malicious behavior.
So, this is an example of why build provenance is useful
in an attack to do incident response and why we've seen it
be adopted in so many different ecosystems.
Okay, so let's talk a little bit about build provenance
works and that will make the examples of how to get this
information in ecosystems make a lot more sense.
So, this is really winding back the clock, at least
from a technology perspective.
Five whole years ago, many build systems started adding
this concept of a signed workload identity so that during
your build process, you can request this, usually it's in
like an OIDC token and then that OIDC token contains
all of the build provenance properties we care about
the source code, build instructions, etc.
So, our goal is to transform the signed OIDC token
and we want to have that information available
when we sign our package.
And then the signature is in an X519 certificate and so
we'll go in the extensions and we have a couple of OIDs
for build provenance and we just sort of map the information
from the ODC token and we put it into the signing certificate.
So, then we have this information available with the signature
that travels along with the package.
There's an open source project called Sixdoor.
It contains both implementation of a number of clients
in different programming languages as well as a
public good service, which has multiple port parts.
There's Fulcio and that will exchange the OIDC token
and create for you the signed certificate.
There's also a transparency log, we'll touch on briefly.
But the main component we're going to focus on here is Fulcio,
which is sort of doing this translation.
So, of course, it needs to contact the workload identity provider,
make sure that the signature verifies correctly,
and then it can return to your build this signing certificate,
which then the build can for the distribute.
And so, you send your package along with your signing certificate
to the package manager just like you would normally.
One last slide, before we start typing into what this looks like
in different ecosystems.
So, like I mentioned before, there's a couple of different components
to Sixdoor.
Mostly we're going to be dealing with the signing certificate
because that's where the build problem lives.
But there's a couple of other items in here.
There's a transparency log that's to allow folks to monitor
both that their identity is being used for signing
when they expect and also to ensure the trust
with the operation of the Sixdoor public good instance.
And then, at the bottom of the bundle,
you can either have a plain signature over your build,
or you can have an adaptation.
So, this is a JSON document with some metadata
that you want to communicate along with the signature.
And we'll see what that looks like in a second.
And then, it's also really important to note that
you should not just download a Sixdoor bundle and trust it.
You need to verify that the bundle hasn't been tampered with.
You need to verify that the bundle refers to the build that you're expecting.
So, we're going to come back to the Mavent Central ecosystem.
But here we're downloading a specific jar file.
We're downloading the associated Sixdoor bundle.
And then, cosine is the Sixdoor CLI that folks use to verify content.
And so, in all the slides, I'm not going to show the verification process.
Just how to access the province information.
We should make sure that you're using cosine to verify any Sixdoor bundles
before you start trusting the content that's inside of them.
There's a couple of really great blogs from Sixdoor about what this process looks like.
To verify bundles specifically, and then also, like, in general,
what you should think about if you're going to build a system that relies on Sixdoor bundles.
Okay, we are finally ready to dive into the various ecosystems.
I do a little bit of work with NPM, so that's why I started.
NPM shows to implement a new API endpoint, the attestations API.
And so, here I'm creating that for the SEMVER package at a specific version.
There's going to be a lot of JQ on the next slide. Sorry, not sorry.
But, you know, this data is sort of like nested deeply inside of it.
As we'll see in a second, NPM has support both for salsa province,
as well as publish attestations.
And so, the first JQ is saying, we want to take all the attestations NPM returns,
and we specifically want to select prominence.
And then the second JQ is diving into that Sixdoor bundle,
looking at the signing certificate, getting the raw bites,
and passing that to open this to SEL.
And then, of course, you can see here, we have information about exactly what platform this package was built on,
what repository the specific source commit.
Also, there would be build information if I had more room on this slide.
But this is an example of how to get build province information from the NPM ecosystem.
Instead of using release attestations, NPM uses publish attestations.
This is for the very good reason that it was implemented before release attestations existed.
It was sort of a precursor to release attestations.
This is almost the exact same query,
except instead of looking for the build province this time,
I'm looking for the NPM publish attestation.
And it has the exact same information as the release attestation,
the pearl of the package, and the exact hash that uses.
As we're going to talk about more publish attestations and release attestations
are kind of in a weird state currently.
NPM, I think, is the only ecosystem that exposes information.
And part of the reason is because we're not exactly sure how people are going to consume this.
It's really worth noting here that the history of security,
especially software security is filled with people proposing like, you know, ways to sign things.
But they don't always talk about, like, how is it verified.
And so, this information is being distributed.
But, for example, the NPM CLI doesn't currently consume it.
And so, you can certainly go on the transparency log of six store and see this information being published.
You can get it directly from the registry, but it's not yet being consumed,
which is something that would be a good thing for us as ecosystems to figure out in the near future.
Next up is a pie pie.
I'm not going to go through this as much detail, just kind of note the similarities and the differences.
One of the main similarities is that it's using a separate API to expose this information.
If you have the problems URL, otherwise this is exactly the same.
You grab this certificate, pass it to the open to the cell.
You can see exactly where this package came from.
And then pie pie collects information for a published adaptation.
And I think it actually submits it to the public transparency log.
I'm 95% sure, but it doesn't serve it out.
And again, this is because that ecosystem wants to wait to understand who and when this information would be used.
And so, currently, the CLIs that are frequently used to access information from pie pie don't use this information.
But if you're curious as to the format of that adaptation, even though it's not currently being served via an API,
you can see what the format of it looks like there.
Ruby gems has some pluses and minuses.
Again, like the other two ecosystems, it has its own attestation API.
There is a little bit of a speed bump, though, in that the information that it serves out isn't formatted correctly.
The good news here is that it could be.
But there's a bug currently.
Marty, I think I said you this before, so hopefully you're not learning about this just now.
But essentially, instead of the certificate robites, it's taking the payment coding of the certificate,
based 64 and coding it again, and that's what it's serving out.
So if this is an interesting topic to you, if you're looking for a sort of like a small project to maybe dip your toe in,
this would be an interesting issue to follow up on.
And if this was changed, then you could verify it with a six or two link.
Last but not least, there are a tour of ecosystems is Maven Central.
They made a different invitation decision, which I think is interesting.
It's totally fine.
And so instead of having a separate attestation API, they're just serving the six tour bundle alongside the content of the package.
This really illustrates that in these conversations that we're having across ecosystems,
the goal here isn't like 100% alignment lockstep everyone implements it the same way.
We're all seeking to serve the same goals, right?
We want transparency and our things that are built.
We want the registry to be consistent about what packages it's serving.
The specific implementation details don't necessarily matter here.
But once you pull it down, it's just the same pattern as before, pass it through open to SL,
and you can cross out the build properties properties.
So, where do we go from here?
I think that probably the next step that needs the most clarification is like,
what are we doing with these release adaptations?
Or maybe that's even the wrong way to face the problem.
Taking a further step back, how do we want to protect the index of package repositories?
It's really, really bad if an attacker accesses the data store for these package repositories
and modifies them and says, oh no, the package didn't have hash 1, 2, 3, it had hash 4, 5, 6.
Whatever my malicious version is that I'm serving out.
So, somehow we know that we need to better protect these indices.
There's a couple of different approaches that folks are taking.
For example, the way that NPM is signing content, it actually provisioned sort of a durable public private key pair.
And got the public key as a trusted delegate inside of six source verification material.
Is it interesting experiment?
I would not encourage anyone else to follow that path because then if you need to rotate your key material,
you have to coordinate with this like third party open source project.
It was a little clunky.
The key I did this in a more elegant way with a sister capability called trusted publishing.
So, here again, they used that signed OIDC token, the workload identity,
but they use it to make it so you don't need a API key in order to publish to IPI.
Instead, you set up a trust relationship and you say, if my build is coming from this platform,
this repository and this known build workflow, you can trust it.
I don't even need an API key to authenticate.
That's a really powerful property and it allows them to construct public addestations using six source when they receive that signed OIDC token.
And then in the open SSF, there's another project that's taking a completely different approach.
So, instead of publishing addestation per package, they are signing the entire package index periodically and then publishing that information.
I think like multiple times per day on the order of like several hours.
The only sort of like downside or limitation here is that no one has deployed this to production yet.
I believe it's being used in dev in some of the packed repository ecosystems.
So, you know, in summary, there's like this really cool new security capability build profits.
So, I hope people go and check out.
We still don't know exactly how to protect the indices of packed repositories.
There's a couple of interesting experiments that are ongoing, but we don't yet have a definitive answer of what we think this should look like.
Thanks.
We have time for some questions.
So, wonderful.
Yes, happy to.
Yeah, go ahead.
What's your presentation?
I would have the one question that we have of opinion.
One thing I noticed is if you have the build blocks linked in, for example, the vegetation in the NPM.
And the build blocks are on data.
You type of own the best system for roughly any year.
Do you see that as a problem?
Or do you have anything in the bottom of this building better because probably,
I don't think we're going to have to solve that.
Let's talk about the best thing that you have.
Yeah.
I guess.
Yes.
So, I mentioned a couple of times that, oh, sorry.
Yes, yes.
The question was build profits includes build logs, but many platforms do not process those build logs more than like 14 days.
And it seems like a security incident later, you might want to know what was in those build logs for a critical build.
And full disclosure, I work at GitHub.
I think that you are right that we need a way to store these build logs more accurately.
I'm not prepared to commit publicly on GitHub's behalf.
When this might be addressed, but it is something that we are aware of and talking of internally.
And I hope that soon there might be more information.
Thank you.
Any more questions?
Great.
Thank you.