Hello, my name is Marco Dacos. I'm a software engineer at Google. I was going to give
this presentation with Brandon Lum. Unfortunately, he fell sick. We had originally planned
for, well, Brandon loves S-bomb so much that he really wanted to give the talk. So, he did
a recording of the first part, but we didn't check, we didn't realize that that's
not supported here at Fossum. So, I'm just going to give the whole thing. So, bear with
me for the first part. Let's begin. Today, we're going to talk about Google's journey
with S-bomb. We'll start out with how we implemented various aspects of this and then some
lessons learned through this process. We will go through the S-bomb lifecycle of a generation
storage retrieval and then various applications. So, before we start off with this lifecycle,
I guess first, to give some context, our initial motivation to push dive deep into S-bomb
was in mix of responding to EO 14028 and security and license use cases as well. Before we
get into that lifecycle, let's talk about some design principles that we developed. The first
question we asked is, what are we looking for in S-bomb? We rallied around two properties,
first accuracy and trustworthiness, is the dependency information in the S-bomb correct.
And then trustworthiness. Can we trust this S-bomb and use it for important security and
compliance decisions? Based on these properties, we developed a series of best practices that
will go through in this talk. There's a link here as well to the document that goes through these
design principles that we recently made public. So, feel free to check that out. Another
question, clearly, that we are in into, is S-p-d-x or cyclone-d-x or both. But more generically,
the question is, how opinionated do we want to be throughout this process? The scope for us
was very large. It's scanned, it's spanned, organizations, different products, different
text acts. There were a lot of moving pieces as a result. We decided that less is more. That
for all of the moving pieces to come together in a better way, being more opinionated would
be helpful. So, we decided to only use one S-bomb standard based on the experience we had
at the time. We went with S-p-d-x. So, let's get into S-bomb generation and how we approached
to this. A question that we faced here is, when and where to generate S-bombs, we can do
it at the source phase during the build or analysis. After the fact, after the artifact has been
generated. On one hand, if we look at the source, if we generate S-bombs at the source, we
found that things like tests and plug-in dependencies would end up in the S-bomb. We found
that there is perhaps ambiguous dependency resolution. And on the other side, for analysis S-bombs,
builds our lossy. There could be missing context in 2022. We did some work and showed that
if we, for example, import a binary into a container image. As we mentioned, builds our lossy.
So, that information won't. The dependencies of that binary won't show up in the S-bomb.
This is a Goldilocks situation at the source S-bombs are inaccurate and in the analysis of
the artifact stage, they can be incomplete. We found that the best approach is to meet in the
middle at the build time where there is a good trade-off between accuracy and completeness. We
found that to know when or to really know what goes into the artifact that's being produced,
the builder and the build process knows best. So, we recommend that only the builders can
generate S-bombs. But this is again also a nuanced question because during the build time,
there can be an S-bomb generated by the build tooling itself or an S-bomb generated by an SCA tool.
We recommend that wherever possible, the build tool should generate the S-bomb.
And in some cases, this lists some cases where we do that. For Android, we created a
Gradle plugin that generates an S-pdx S-bomb during the build. We have a similar approach for
Google 3 which is our monar repo. But outside of that, we use SCA tools such as SIFT and OSV
to generate the S-bomb during the build process. So, now we are faced with S-bomb storage. We
have the S-bombs. So, let's just put them into a database and be done. Well, it's a little bit
more complicated than that. And the main, one of the important questions here was to be able
to use S-bombs for important security, compliance decision, to be able to get them to the
government, for example, how can we create an S-bomb database that we can trust?
So, for this, we developed a system called SILO, the Supply Chain Integrity Log,
contrary to its name. It's supposed to break up metadata silos. For those familiar with Gwak,
it serves a similar purpose. Gwak in the open as a project in the open SSF. It serves a similar
purpose, and it's under the same team. Or it's worked on by members of SILO as well.
So, SILO, for some context, collects metadata from software supply chain events that occur
in Google. When builders produce artifacts, they produce build provenance, they sign it,
and then they send it over to SILO. And then SILO can verify the signature,
can verify that it was produced by a trusted builder, and then can ingest it, store it,
and use it, and make that information usable. For S-bombs, we took a similar approach.
We used something called, we used in Toto, and in Toto, a predicate type called the reference
at a station, which we have now upstreamed to the at a station repository that's a SILO-C8s
and S-bomb. Or in this case, S-bombs with a software artifact. And so here, when builders produce
S-bombs, they generate and sign an Intoto-autostation and send it over to SILO and SILO
verifies this to ensure the integrity of S-bomb. And this also supports only accepting S-bombs
from trusted builders. So now, in theory, we have a table mapping artifacts to S-bombs and
retrieval should be easy, but famous last words. So what we want is to have a simple input
output of looking up an S-bomb by an artifact, such as a URI or a digest, but what we got was
nothing when we did this operation. Why is that? So the context is that we're working in a
supply chain. It's a graph and it has to be reasoned about as such. When we search for the S-bomb
of GCR.io slash ABCD, we didn't get anything because no S-bomb was produced for that artifact,
because that last stage when that artifact was produced was, say, a promotion process.
And that's not a build. So there is no S-bomb generated there. So what we can do is go back
in the graph. We can look at one lower lower. And here, in this case, we did not find the S-bomb.
And then we go even further back and turns out that this artifact in the staging is a multi-ark
image that was assembled from different container images itself. And here, we found S-bombs.
Great. But we can go even further back. All the way back through the chain to collect all the
S-bombs that were in some way associated in the production of this final artifact.
So now, when we look up for the S-bombs of an artifact, we get a collection of S-bombs.
Awesome. Here are some edge cases that we ran into throughout this process.
Well, going back a second, what we did was generate a, do this, implement a transitive
search through the software supply chain graph whenever looking for S-bombs to collect all of these.
And here's some edge cases that we ran into through this process. I won't go into all of them here.
But in general, the practice here, the principle here is that to get good quality S-bombs,
that are accurate and complete, we try to compose them to arrive at completeness.
So we've left out a little bit of detail here. Where does the scoff come from?
This is an idealized state or is it real? So let's go back to the build, what we mentioned earlier.
That when the build there produces an artifact, it creates a build problem, it's not only describing
how the build was conducted, but also the dependencies and the materials that went into the build.
So this problem is going to be used to construct the graph.
And it can be used to glue together these lost pieces of S-bombs, creating more complete S-bombs
by composing them together. And this is a blog post, partly by Brandon, that goes into this in
more detail about how salsa and S-bombs work together.
Another challenge we ran into here is that often a request for S-bombs are the artifact granularity.
Somebody will ask our S-bombs for a product say pixel OS. And translating this to
to something that can be interpreted with the mechanisms that we developed
is challenging, maintaining self-to-inventories hard. And in these cases we relied on the expertise of
product teams. This is an example of why, and part of this is challenging because different
products can be composed of artifacts in different, in varied ways.
Okay, so we've responded to the EO, but compliance isn't fun, so let's talk about some other
things that we've done with S-bombs. S-bomb blobs are cool, but what they contain is even better.
We used them to develop dependency inventory across the organization.
This involved parsing and storing S-bomb data at scale. For now, we only store
flat lists of dependencies that are in the S-bombs. We found that the graph structure that can be
encoded in an S-bomb relationships was useful, but that it wasn't mature enough to really be
dependent on. And developing such a dependency inventory with S-bombs in a large,
softer producing machine like Google is difficult, where it's spanned different products,
tech stacks, different CSED flows, different S-bomb generators. But there are some properties of S-bombs
that really helped to succeed. This included, the fact that S-bombs are common format across
ecosystems, that's enabled us to reason about otherwise siloed systems through a single pane of glass.
And another thing that really helped here was that S-bombs are flexible both in the generation
and in the concepts of them. We did, as we talked about, put forth recommendations on how they
should be producing Google, but within that each ecosystem could choose to generate S-bombs
in a way that produced the best S-bombs for them. But this flexibility also led to some challenges
that we'll talk about in a bit. So we have this dependency inventory. We can combine it with
things like threat intelligence and organizational metadata on using systems like silo and guac
and use that for various use cases such as internet response, measuring risk from upstream,
open source dependencies, and also for general business and housekeeping purposes.
This inventory enables fast identification of where packages are used. This is critical for
internet response. I think this is like a log for a shell or XE type capability, a quote from
a product team is that they were able to figure out that they weren't affected by an incident
within 10 minutes using S-bombs. What's shown on the screen here is a sample dashboard that
can search across this inventory. As you can see, the searching for a single dependency
searches through various siloed systems. That's part of the value that the S-bombs provide.
This slide also calls out the importance of point of contact information or
attribution of artifacts to artifacts and artifacts to products and to teams. This organizational
metadata is important because otherwise the actionability of this inventory is limited and
it creates toil when responding to incidents. Five minutes. It's not until 1030.
Yeah, we'll have the questions. Okay, so another thing, another use case that we
applied it to is to measure the risk from our dependency usage. This graph shows
the subset of the fleet, the dependencies of subset of the fleet,
mapped to the open SSF score card scores and this identifies a danger zone here.
But this really highlights the need for additional metadata such as criticality and
attribution to make this more actionable. Now, run through some lessons that we learned through
this process of operationalizing a dependency inventory using S-bombs. As you might have noticed,
we decided to use perils for the identification scheme for this inventory. We decided to use perils
because they're common in S-bombs. They are other identification schemes and cut it and
we didn't want to develop a new one. However, we quickly noticed that many S-bombs or
packages in S-bombs didn't have perils. In these cases, we generated fake perils with the
information that was available in S-bomb, but this still had limited utility.
And often this happened due to things like build-time generation of S-bombs from artifacts
that contain third-party vendard code, where the proper identification metadata hadn't been
propagated through. Other shortcomings that we ran into related to S-ca tools that we used.
And these are really a fundamental result of how they operate. But S-ca tools attempt to
read files on disk and match them to packages in a registry based only on what the file
on disk says and not and without access to the registry. Understandably, this can lead to some
issues. The problem is that the same package can have, sorry, the problem is that different
packages can have the same names or a very similar manifest files within or outside of the
same ecosystem. And the scanner has to choose one. For example, Unity, which is a game engine,
has the same package that JSON file as NPM. And in this case NPM perils were generated for
something that wasn't NPM at all. Other cases include private packages. This is an example.
This is shown on the screen. This is a meta-host file of a private package in a repository that
was not published to NPM. But an NPM pro was generated for it. And for the by the pro spec,
this pro indicates that this package is on the registry. But in reality, it's not.
So why is this important? Why do I flag this? It's because some other people not with the best
intentions also realize this. For a lot of these packages, such as the private packages, for example,
malware was submitted to the registry under that package name. The name was squatted. And then
this creates a lot of toil because whenever somebody uses a project that contains one of these
say private dependent, private packages and runs a scanner on it, it will flag it as malware
and reality. This is an example of a thread of a project that contains such a package and it
receives a lot of complaints. Understandably, the maintainers didn't sympathize.
So we also ran into some challenges with identifiers going to go through this section quickly
because I don't have much time left. But the SCA problems that I mentioned are
archipounded by lack of expressivity of the identifiers that we used. General identification is hard.
Even if we could identify that a package is private, for example, pearlism support, creating an
identifier for it. SCA tools could extract more information such as hashes, but how to do this
isn't standardized. And how to compare packages with that say supplemental information isn't
standardized either. So I'd like to also just call out a reference that I think a project that
I think is making a lot of progress in the space guac, which is guac that we mentioned earlier.
guac is looking at a strategy of disambulating and correlating identifiers to
to help solve this issue. So these issues are all fall under
S-bomb quality. Other issues have also highlighted this and the importance of that application.
The importance of S-bombs have meant that this year we're really going to focus on S-bomb quality
both on syntactic elements, but also semantics such as completeness and accuracy.
This is an example of S-bomb quality that's a lot of accuracy, but I don't think I have to
go that into here. So what now? We went from almost zero S-bomb to four million S-bombs a week
totaling over 200 million S-bombs. Security and compliance teams are using S-bombs to triage
security and compliance issues. S-bombs are part of the security posture of several organizations
and we've been through a few S-bomb tools along the way. And we use S-bomb tools along the way.
This is just a list of some things that we're going to, that we're looking forward to,
that we're focusing on this year. And yeah, that's just on the screen.
Yeah, so I'm ready to take any questions. If anybody has any.
With the Wich school do you use for recognising vulnerability for example?
Is that completely different? Is that from the S-bombs?
Oh, yes, sorry. So the question is if we use, what do we use to detect around abilities in
S-bombs and do we do that as part of, is that in scope of S-bombs or is it out of scope? Is that
done separately? And in this case, it's out of scope.
I also have a question. Yes. Since you mentioned the supply chain, do you reuse parts of the S-bombs
or which one results in the end S-bombs that is being used at a stand-alone unit?
And as it forms to, either media and S-bombs that are being.
If I understand your question correctly, we use provenance to, sorry.
The question is, do we, I mean, just very,
explicitly do we merge S-bombs together or do we compose them? We compose them with, with a mechanism that we mentioned?
Yes. You've got to change about a date set. Really good.
How are you contributing the date so you're finding upstream to make the matter data better for everybody else?
Well, a good question. Yes. We have a big data set. How are we contributing this back to
upstream to tell the community? We, the sheer we're looking at generating error developing,
and S-bomb quality library. This is something that we may be able to open source,
and O-S-V-Scaliber will be a part of the S-bomb quality that we look at, and this is open source as well.
So you're not, you know, thank you very much.
Actually, the Copeland and Brandon was online on the Matrix, and was answering questions there.