Hello, everyone. Welcome to our next talk and security room. Next talk is from Tunis
at a streamlining signing artifacts in container ecosystem. Welcome Tunis.
Thank you. So, yeah, I'm Tunis. I work at open source built-in, a talker. And
today I'm going to talk about sign artifacts and some of the advancements we made in this
area. So, to get started, like, why should you sign? And I think there's simple answer
is that because there's many threats there to end many things that want to attack your software
and compromise it. So, when the first example I just picked this actually, you
tells hack that was came, like, lots of attention. I'm sure many of you remember this
one that lots of coverage, mostly the coverage was about this bizarre situation, how there
was this maintainer who built up trust for a long time. And then, also, like, the patches
were really, really complicated. What maybe got less coverage was that if you actually look
at, actually, exploit, then a part of it was never even like committed to get. So, basically,
some artifacts were just built on a random machine. They were shared with the rest of the
internet and people just trusted it and started to use it. So, like, the question is, why do
we think this is normal? Like, like, isn't there like a technology solution that we can,
we can use to prevent this? And yeah, come from Docker. We work with containers and there's
been, like, many different attempts to do some kind of signing in the best. But if you, like,
pick some kind of, like, a random popular container at the moment or a container image,
then, most likely, it's not signed. So, this is what I'd like to change. And then,
just when we're looking at some of the solutions, I've listed here, like, some of the things
that I'm looking for in this possible solution. So, I want the solution to actually
be effective against some of the practical attacks. If you're doing the signing, you should
really understand, like, why are you signing this? Like, what value is the signature providing
because, like, if you're just signing it for the sake of signing, then we're just
adding, like, another failure point. Another thing, I just want to emphasize is that I do
think it needs to be simple. If you want to get, like, a really big adoption. Like, I could
come here and talk to you that, like, you need to get, like, three of your big keys, and
you might go on with a, like, a ton of another, and then you put the third one into a safe
time, then you go and share it with whatever ways. Like, I don't think this is, like,
way how we can get all our images signed, because, like, most of us need to spend our
time, like, actually building things, like, we can't, like, worry about the security
problem, also, like, the whole time. So, yeah, I'm looking for something that's, like, it's
very tight and integrated with existing tools. And, yeah, it also doesn't make it, I
think it's thing tools super slow once you enable it, because, like, if this happens, then,
like, people will just not turn it on and, and nobody will, will care anyway. So, let's see,
if you can find something like that. I first want to talk to a little bit about the
circuit. So, the builder framework that we use at Docker, the powers at Docker build command
we use it to build our Docker files, but it's not really, like, anyway, restricted to just
Docker files, it can build, like, you kind of build the conditions. There's many, very
popular use cases, even I saw the Docker. It's hyper formats, it can build very complex
builds, it does automatic casting, it's about the, like, the built kit container image,
just past one billion pulls this week. So, it's, like, built it is running, like, millions
of builds every day.
Uh, a little bit to keep the understanding of how built kit works internally. So, your built
definition is to need to a graph of build steps, and all the, this graph starts from
something called source nodes. So, these are, like, things like images, heat repository,
HTTP URLs. These are all immutable. They're guaranteed to be immutable. You can get
a checks on for them. And then, while you build this running, then, those sources can
transform them. And there's, like, very, you can do, like, any kind of, like, dependency change
between them. The important thing is that, uh, everyone of those build steps is sandboxed,
and it's, like, a side effect tree. So, it's not, uh, the, the step is not influenced
by another step running as part of the build, and it's, there's never influenced by the
system where it's running out as well. Um, then, once you, once you build completes, you
get a build result. So, of course, like, you want to get images, but build kit can give
you any kind of hard effect you want. So, we build our all-ever release binary, as well,
like, build kit because my wouldn't, yeah. Uh, in addition to the build result, you automatically
also get the provenance at a station. And, uh, like, if you're seeing the provenance
at a station, maybe from some other tools, like, or sometimes they, like, seem like there's
just, like, a checksum, and you add, like, some kind of parenthesis around it to make
it chase, and then, like, this is not like that. Uh, this provenance that the station
has, uh, your, like, complete the trace of your build. So, like, it's, as the complete
information of, uh, what exactly happened during your build. So, it has all the, uh, all
the build sources, uh, our capture there with a checksum, all your build configuration, your
build step-to-step initial. So, every client, uh, like, this, uh, changed during your
build flow, photographer's source, timing information, like, you know, like, if you're, like,
build access network or, or not, uh, if it's, like, binary reproducible things like that.
So, this is very good if you're, like, have an artifact and you want to figure out, like, how
this artifact was built. But it can, it can give, like, really, like, complete story from the,
like, the initial ingredients to, to how, how the final binary was made. But, uh, I still
could make this bonus statistician, it's not signed. So, you can't, like, really use it for,
some kind of a smaller detection. So, you need to trust it the same way as your trusting
the image. Um, so, let's see if it can, as some kind of trust it, with the signature.
So, I told you that, uh, toolkit provenance is, is not signed by default, but let's look at some
other example, in here, where, where there is an attestation and it is signed. So, this is something
that I copied out of from, documentation from, uh, GitHub, well, they, they kind of, like, have,
like, documentation for our, like, how to use our actions in, in, in the, in the, in the,
in the GitHub actions development. So, in the first section in here, here are, like, our actions
for, for actually running to build the toolkit. And then, in the end, they write that one extra
step, uh, that, like, generate artifact of the station and if they look at the, like, documentation
for the step, then they say that this thing is an unfortunate statement. So, like, at least,
somebody is very confident. So, so, let's, let's look at, like, how, unfortunate, all this, this
release them. So, let's first, uh, see what's good about this. Like, like, this is a signed
attestation. There's no key in here. Uh, so, the way it works is that temporary key is generated every
time, or you run that this workload is, every time there's a new key, and based on the scope of your
GitHub token, you get a certificate from a fixed order that matches this scope and matches your,
your key. Like, once you're, once you're done with your job, then you throw away the key.
So, that, that's, that's all simple. And if you look at some of the tech vectors, so, like,
if our tech vector is that our register credentials can compromise, then this is a very good signature.
Like, if you have, have a way to change stuff in the registry, you can never, like,
create a signature like this that matches the, the both as well. But, what if our GitHub credentials
get compromised, not register credentials? What if our, like, our, like, GitHub token, for example,
leaks, like, this is what we saw, like, many times last year with all the, like, like, a childhood,
a text, right? So, in that case, you're, you're kind of, like, screwed, because,
then, like, with your GitHub credentials, first of all, you can, the attacker can change anything
that happens before in this, in this, in this build flow. But, you don't even need to do that.
You can just use the GitHub token directly, go to sixth or register the token in the sixth
or, and then, like, you, you can do, you can save, save that, that employee key forever and
use it forever and, and, and take as many things as you want. So, to avoid that, what we really
need to do is we need to put all this build flow away from the user repository. So, that, when
you have access to the user repository, then, you still need to, like, you still can't,
can't mess with this build flow, and you can't mess with the signature. And the way to do this
is some, called, our usable workflow in GitHub Actions. And this is what we've done with our,
like, newer repository that we've created, that's called, GitHub Builder. So, this, this provides
this trusted, usable workflow for builder for doing this build flow. If you use this, builder,
then, all the attestations are signed or, like, get a proper six or signature. And those
attestations, how do we, like, the build key statisticians, not, not some other, like, simple
attestations. Yeah, we make sure that we only allow secure build configurations. We make sure that
everything that's part of the build pipeline is also verified before your build starts. If you
want to cache, then we sign the cache as well. So, like, you can only import cache that's already,
also created by this builder, and, and, and properly signed, and yet, and many other features,
as well, like, the laundry security related. So, just to illustrate, like, what's happening in here,
like, you're, like, a simple file flow of the, getting to six towards signature. This is how we
was in the, in the, like, first example. And now, if we add the trusted builder there, it comes in
the middle of your upper repository, and the sixth or, like, the certificate provider service. So,
now, all the, all the signing is contained in this trusted builder, not with your upper repository.
Uh, you can, as a user, you can't change anything that's happening in this trusted builder.
You can't, you can't inject anything into that flow. Only thing to adapt repository can do,
is provide itself as a source for the trusted builder. And then, once the build is complete,
then it can get the result back. So, just to make it very clear, like, what does the signature
prove? Like, and the answer is that when, when you have an artifact that was built with this builder,
it has a certain checksum. Then there's a very strong relationship that this binary was built
exactly from this, keep commit in the repository using exactly those steps that are covered in
the provenance attestation. And, uh, and, like, some, some of the, like, things, like, that can't
fake this relationship is, like, you know, when you have a registered financial leak,
you can't fake this relationship. If you have, uh, kita credential leak, you can't do any of that as well.
Uh, yeah, like, if you're, kita token, for example, leaks in CI, you still can't, even if you're,
like, a rogue maintainer of the source repository. Like, whatever, you, you want to do, like,
you have access to their repositories to change anything in there, but you can't make, like, a
fixing ensure, like, this from the, uh, paid based on the source code. And obviously, like,
just to make you sure, like, like, talker can't, like, create two signatures as well. Like, like,
like, it's not that you need to trust, talker and, and hope that talker doesn't create, like, a
fixing ensure for your project. And, technically, you don't need to trust six store as well,
because, like, the transparency locks. So, it's, uh, the transparency locks of the, of the six
store would, would then show, like, right away, like, if anything, even, like, happens on the six
store stack. And, yeah, we will also protect against the built-up end of CI leaks. So, like, we
make sure that, uh, the workflow that's actually, like, running when this transformation happens,
doesn't have any, uh, any, any, any possibility to do, like, a tool come there and, and
check this off. And we also make sure that, yet, you have no side effects from the host at my
moment. So, when GitHub changes their VM image, and there's some kind of arbitrary state that's
subtracting their VM image, like, this doesn't ever add any effect for your build.
So, yeah, as a summary, this is, uh, like, like, the, the trust of this, of this model is provided
as a combination of the two tools. So, one is, uh, this GitHub builder that provides the trust
assigning context, and that it moves the, like, the correct version of our, of our build stack.
And then build kit itself is doing the, the, the trust model of the actual build when it's running,
making sure that they're bi, build is properly isolated and making sure that, uh, all the,
all the steps that run as part of your build are properly captured.
Okay. Now, this was about signing. Let's talk about very fine, as well, because, like,
a lot of the time, like, very fine, it's kind of ignored in this story, like, there's so much
talk about signing and then point away, like, I guess we can somehow do the very, very fine,
as well. But if you do not verify, then there's really no point of signing at all. So,
because, like, the whole point is that we create this very strong signature that attacker can't fit,
but if you, if that, if you don't verify that, uh, all the fact, for example, that attacker has replaced,
doesn't have this strong signature, then, like, there's, there's no point of doing any of that,
the dog. Uh, so, some manual ways to do the verification, uh, is, uh, we've had that,
and you think you're inductor inspect. So now, when you're, like, inspecting a signed image,
signed this way, then it would show the signature there already in the verified form.
You can manually sign with, well, verify the cosine, of course, because it's, uh,
completely, like, a legitimate six-store signature. Uh, there's a thing for Kubernetes, like,
if you're, like, writing policies in YAML, uh, what we really want to do, though, is something
more built in, so something that's more built into the tools and, uh, more, like, automatically enabled.
So what we've added is, uh, a re-godd that's from the open, uh, policy agent, uh, like, uh,
policies written in the re-god language are not, not supported directly in our build stack.
So what you can do, which you can write the policy, like, the matches you're, like,
talker file, for example, name it, talker file.trigo. And then if this file exists in your project,
we will automatically load it anytime you invoke, talker build. So, and what's happened then,
is that when built, it will run your build for every source in your build, it will first go back
to the policy and check with the policy. Like, if, if this source value, they're not, and if it's
not valid by the policy, then your build will just not complete, like, it will, it will just
throw it out right away. And yeah, it's very flexible language to write policies, and we will
give you all the source properties, like, to, to check with. So you can do, like, very, very complex
stuff in, in, in those policies. So this is just some of the examples that, of course,
like, I'm going to show the signature case, but you can do the check, verify the check sums,
verify, or like, pin the check sums, like, you can do the kitchen at yours, whatever,
the image properties, like, anything you pretty much want. So, here I have the example,
for, like, how do, how you would use this regal policy to verify,
this trusted pillar signature that I just showed previously. And for that case, like, we've added
and, help her in here to make this easier. So pretty much, what to say is that, if you have a source
as part of your build, so that's, what a image called, or app, then it's only allowed to be used.
If it has this value trusted pillar signature for this same kit up repository name, and
extra condition is also that the kit up tag and the image tag need to be exactly the same.
So you can switch the versions as well, in this case. And this is the same exact thing,
it's just, like, written out what would happen internally, like, or, like, if you work,
example, like, write to verify manual six-store signature. Important thing to notice here is that
we need to verify both the, like, the source repository and the pillar repository separately,
because, like, if you only verify one, then it's not secure. It's only secure if you verify both,
like, the combination of them. And another question might be, like, okay, like,
come, using those dependencies in my project, like, how do I know which policy rules are
correct for those projects? And I think, like, at least what something that we can start with,
is that if you use this builder to build a release binary, then what you can do is,
you can just write in a read me that, like, hey, like, we are using this, in our release flow,
we are signing images in this way. We have these rules of, like, these tags,
I'm remutable, these tags are not, for example, and put just an example there, and, like,
that users of your project could, could just put their into their policy to make sure that
your dependencies, like, validate that in their most correct way. And this isn't, like,
like, the point of that is that then you, like, commit this, commit this policy into
it, and then it kind of like comes as you are, like, security thoughts for your project.
So now, like any time you work on your project, you are automatically protected by the policy
when you, like, your CI runs for your project. It also protects your CI when you release
what runs, it also protects your release, but when other people clone your project,
they also get this default behavior. And, of course, there's a way to how they can, like, disable,
if they don't want it, they can tweak it. But, as a project owner, you can kind of, like, define,
like, what is the secured default for your project? So, to recap, we just talked about,
like, we sign your images, you know, reason not to sign, like, we have, we have the technology
to add this very, very strong, like, security parameters, need to have all the effect,
and it's very easy to do, I think. Not all signatures are equal. So, when you're, when you're
doing their signing, like, make sure you understand what you're actually protecting with it,
be to signature. And, software that pulls those sign packages, like, it needs to do the
verification. If it doesn't do the verification, then they kind of like, the signature is useless.
So, the trusted builder repository, it's available today. You can use it right away.
If you're, like, using our existing kit-up actions, for example, at moment, like,
like, I think something, like, 800,000 repository still or something like that, then it's very easy
to, for you to upgrade, like, it's pretty much just a little bit of, like, syntactic changes,
but it's built by the same people. So, it's, like, the UX will be very, very familiar with, yeah.
If you're not building, like, container image is at all, then I think this could be still,
like, super useful for you, because, like, whatever way you're building your, like, release
planner is, for example, like, it's very likely that whatever tool they're using, it's
run is fine inside containers as well. So, you can just, like, pretty much, like, take your existing
thing, but add a little bit of, like, a container wrapper around it, and you get this very strong
signature guarantee, like, almost like, out of the box. Yeah, and all are, like, latest releases
for, for a built-in, we've already created new versions of them, and they're all already signed with
the same builder. So, we're, yeah. And, when this tool's run inside the GitHub,
later, we also get a signature verify properly. And, yeah, the vehicle policy support as well,
we check it out. It doesn't, like, think that we will continue working on, so, very much
interested in any feedback if you have there. And if you use the GitHub builder, then, please, like,
keep, like, an example, policy as well, for their finance that you've created, so that people know,
like, what's the best way to validate it? So, I think I'm, like, pretty much, out of time, so,
I'm not sure if we have time for any questions, but maybe one question. You're a closest.
Thank you so much for the talk. I was wondering two things. If this workflow, it's strictly
tied to GitHub, or I can be exported to other systems, I guess, yes, possible. And, the second one,
I was wondering, what about Rigo? If the policy that it supports are integrated with the package
manager, that we use to build the artifacts, like cargo, that downloads a lot of stuff, so,
I want to actually check. Yeah. So, the current, like, the GitHub builder is very much like a
GitHub specific collector. It's based on the GitHub workflow. And you also, like, yeah, you need to
run it in the GitHub stack as well, because, like, if you run it in a third-party runner,
then we can't really trust what was going on there, right? So, but of course, like, the whole stack
is open source, and, like, some, a lot of systems are very similar to GitHub, so, like, it's probably,
like, very, really portable. If there's, if there's interest or any other system. And the,
the Rigo's question, like, at moment, the things that you can write to policies against are those,
those sources that tilt it support natively. So, so, we have, like, a big data type that we
expose to the policies that has all the properties of the, of the built-in sources, and the
elected, to, like, package manager things are not, like, natively supported by a, by a built-in.
The, the some ways to actually, like, make them kind of, like, built-in sources, the way you do that,
is you kind of, like, like, turn them into URLs. It's kind of, like, a complicated to explain in here.
But, for example, if you, if you, if you look at the promise of the station for something called,
the Docker Harder Dimitris, then we have, like, kind of, like, a special built-kit front end in there.
And that one actually does, for example, like, a package is for Debbie, and an out-by-natively
in built-kit. And so, so, in the case, they would show up as URLs in the policy, and you could write,
like, like, a policy rule, specifically, for the, the game package.
Okay, thank you. We are out of the time. So, if you have some further questions, please ask them on the
metrics or catch up with Tony after the talk. Thank you.