OK, OK.
OK.
Okay, we're going to go ahead and get started here then.
It's going to be done by Robyn and his team is relying on more transparent trustworthy
sources for arts-linic packages.
So let's give him our new attention.
Hello everyone, thanks for attending this talk.
So yeah, today we're going to talk about the decision that we took at Archenix to rely on
what we'd qualify as more transparent and trustworthy sources for our packages.
This is basically our response to the supply chain attack that's on Linux distributions,
suffered from recently.
Before we dive into it, allow me to introduce myself.
My name is Robyn.
I'm also known as Antis on the Internet.
I am a Linux system and DevOps engineer by day and an open source contributor by night.
I am primarily involved in Archenix in case it wasn't obvious yet.
But I also contribute to reproducible builds and a bit to Alpine Linux as well.
So let's start with a few definition.
First of all, what's the supply chain?
Of course, the supply chain is a very generic term that can apply to a lot of things.
But I came up with a simple definition for the Linux packaging ecosystem.
This is what we're talking about right now.
So the Linux packaging supply chain is composed of all the elements that are involved in building
and delivering packages to users.
Whether it is, you know, technical components such as our build script, et cetera,
but also the infrastructure the package is built on.
The infrastructure that is used to deliver the package to users.
And more importantly, also the individuals like included into this whole process.
So what's a supply chain attack then?
Well, a supply chain attack is when one or more of those components is compromised,
leading in like undesired and unexpected behavior and malicious code being introduced
in otherwise trusted packages.
Now, do we have protections against supply chain attack?
Yes.
Fortunately, we have some mechanism to prevent or at least detect such supply chain attack.
But monitoring and securing the entire supply chains remains a challenge,
specifically the specificity of very early part of the supply chain.
But we're going to see that later.
So let's go through a quick list of the few of the mechanisms that we have to protect
ourselves against supply chain attacks.
First of all, we have checksums.
So when we download the source for a package, we generate like a hash of the source.
And then this hash is locked into our build script as the hash that we expect
the next time we're going to download or use this source.
So that ensures that the sources have not been returned in between multiple downloads
or rebuilds of the package.
But, unmatched checksum could indicate a supply chain attack,
but it may also have different reasons.
We have a few upstream that have this bad habits of recreating a release,
like they say they made a typo in the man page.
And instead of cutting a new point one release,
they will discrificly like delete this one and recreate it.
Please don't do that, guys.
Or it can be get hub that decides to change.
There are algorithm and get archive that will change every text sum of every get-up
total that happened.
They reverted it, but once that happened.
And as I said, this is technically more of a locking mechanism rather than an actual integrity check.
Once we generate the hash, we expect this to never change.
And we lock the source that way.
But it's not like it guarantees anything in the content of the source in the first place.
Then we have the reproducible bills.
Reproducible bills defines that from the same source with the same build option and the same build environment.
Anyone should be able to reproduce the same artifact bit for bit identical.
Cool thing with the reproducible bills is that it allows to independently verify the path from source to
binary code, entering that no supply chain attack occurred in between.
So for instance, I has a package maintainer.
I could technically inject the malicious patch into my package before the delivery to user.
Of course, all of this is very theoretical, right?
I will not do that, but I technically could.
And reproducible bills is a way to verify that I did not abuse my position because you could try to reproduce the package.
And you should arrive at the same exact result as the package I am delivering to you.
And if you don't, then I may have do something that's thing.
The other cool thing with reproducible bills is that it gives you a technical verification process.
So you don't have to trust me as an individual.
You don't have to put your trust in me.
You can actually verify it yourself with a technical process, which is obviously more difficult to compromise than an individual.
While it's always a desired outcome for distribution reproducible bills,
doesn't always cover the whole supply chain, though.
So that's a bit of the limit of reproducible bills.
Specifically the very early part, as I said, that I'm going to demonstrate that in a minute.
And the last mechanism we can go over is cryptographical signatures with like open PGP keys, like new PG signatures.
This hack does an additional integrity check by guaranteeing that an artifact has not been tempered with.
Since it was signed, I think this is an important part here.
So when you sign an artifact, if it gets tempered with, then the signature verification will fail.
So this is an additional integrity check.
And the other cool thing with signatures is that it allows for the authentication of an identity tied to the signer.
So it's basically someone to blame if something that happens.
So that's cool to have someone that is somewhat responsible for whatever they are delivering.
But set identity may or may not relate to an actual person.
Like an artifact could be signed by an identity that relates to the release team.
Like the arsenics release team.
And you don't really know who that is, right?
So it will not always tie to an actual person.
And it may or may not correlate with the creator of the artifact.
The person that signed the artifact is not necessarily the person that created it in the first place.
And there's no way to establish a natural connection between the creator of the artifact and the signature.
We just assume that's the case, but it may not be.
So I'm not going to spend too much time on this, but realistically signature are only relevant.
If they are accompanied by a trust path, basically if you maintain a chain of trust between the different keys you trust.
If there are multiple keys involved in signing artifact and you just blindly had new keys to your hearing, then you're doing it wrong.
It basically defeats the purpose.
You want to make sure that if you trust that key and a new key is involved, you have a path between the first key you trust and the second one.
Creating the chain of trust that I'm talking about.
If you don't have this bridge, then it defeats the purpose and you might as well just not verify signature, basically.
So all of this to say, we have mechanism.
And yet, you, and yet, this happened.
So just a quick context here. So XE is a library to to compress archive, basically.
And it was backdoor targeting SSH, which would theoretically allow any untrusted third party to access to your mission remotely.
And this happened, this by the upstream package source been locked behind checksum.
Being cryptographically signed as well.
And this by the infected package being reproducible as well.
So the three mechanism I showed you were correctly applied here, and yet the package got backdoor.
So how did that happen?
Well, there are a few things to note here.
First of all, the malicious code was introduced in the upstream source by one of the upstream maintainer themselves.
Which, as I said, is a bit difficult for us to verify because such early part of the supply chain are out of our control.
It's very difficult for us as downstream or distributor to have some verification mechanism on the very upstream part.
So this makes it very difficult for us to detect.
And another thing that is important to note here is that part of the malicious code, actually the trigger for it,
was obfuscated by only being present in the custom-made release table.
So what I talk about as custom-made table is those distribution tables that are distributed as an extra artifact of the release,
all the table that you download from upstream website.
And a lot of packages build their packages against that source.
And yeah, it can be abused.
This is what happened here.
So if you look at the raw sources and the git repository, there was no trace of that trigger for the malicious code.
But if you download that extra tarble, then the malicious code was there.
So that implies that there are different types of sources that we can use for our packages.
I'm going to name a few, not all of them.
But the first obvious one is plain vcs object.
So just check out of the git commit or a git tag, basically a clone of the repo.
You also have auto-generated table, but git force.
So that is the GitHub, the auto-generated table, or the git lab one, or the code, but one, whatever you use.
And you have those in-famous custom-made release table.
So, oh, well, jumped a bit too far here.
Why do we need custom-made release table in the first place?
Why do they exist?
Well, they are generally easier and most straightforward to work with on the packaging point of view.
Because there are some upstream sources that are based on technologies that require
preliminary step before you're able to work with it on the packaging side.
Now, if there are packages in the room, I may raise some PTSD for you guys, but stuff like auto tools.
Hit sub-modules, translation file, manpages.
So, yeah, auto tools, for instance, needs this very opaque auto-gen bootstrap script stuff that you have to run before you're able to work with sources.
Git sub-modules are used to pull third-party dependencies that you may or may not need.
And you would need to do that before you're able to package your sources.
Translation files may need to be generated, manpages as well.
And usually what upstream does is that it gives you this custom-made table.
Well, all of these things are already done.
So you don't have to take care of that on your side.
There are also historically signed more often than other source type, probably as a way for upstream to promote this type of sources against other types.
But there are also very opaque by nature and more difficult and involved to audit, as compared to a Git repo that is publicly readable.
So, in my opinion, historically we have put way too much trust on signature over other mechanisms.
What will come back to that later?
And this is, as opposed to, quote, unquote, transparent sources.
So for us, this is like plain VCS objects or just a check out of a specific Git tag or specific commit.
Or assuming that you trust those platform, we also decided that Git Hub Auto Generatorable or Git Lab Auto Generatorable are okay.
Given that we trust those platform, but you know, all our code is hosted on GitLab, so hopefully we trust GitLab.
And those sources are publicly readable, easily accessible, and therefore generally more audited.
There are way more pair of eyes on a Git repository rather than on the content of those custom made toggle, right?
It gives you this feeling of what you see is what you get.
So, this is an oversimplified image, but I usually like to see transparent sources as a transparent box where I can see the content without having to open that box.
And the custom made release toggle would be in the pack box.
And the signature on top of the custom made toggle would basically just be a sticker on it that says trust me it's fine, you know.
And I don't want to trust you, actually, I don't want to trust an individual.
So, and you kind of have to do that with, would like custom made toggle.
Our transparent sources, I can just look at it and say, okay, this is what I'm going to get.
Now, I've said it may involve some eventual cost in terms of complexity into packaging because stuff like auto gen fraud at all,
gets some modules, et cetera.
Those will be on our side, we will need to do that.
But we now think that it is worth, it is worth the effort, because distributing packages based on more transparent sources is generally a good move both for us and for our users.
And as a matter of fact, you probably cannot see it clearly here, but as a matter of fact, this is the fix that we applied on the XE infected package and the fix is mostly just to switch to get sources rather than this custom made toggle.
And you can see that we had to, whereas it's here, we had to have this auto gen step that we did not have to do beforehand, thanks to the custom made toggle.
Now we just do it, and that's fine, honestly, and we have a, it's not like a bulletproof source, but at least it is more difficult to obfuscate malicious stuff in those source.
So this is what we conclude from that study.
Basically, so there is a link here, you have a QR code at the end to get the slide, so you can follow the link.
The link is basically a package I switched from custom made toggle to get toggle and it implies like, I don't know, 15 gets submodules dependency, so the package went from this to this, basically, in terms of a build script.
But it's still, it's still with the effort, because even if the cost of more complexity, we now think that those, quote unquote, transparent sources and more trustworthy sources should generally be considered a preferred source for our packages.
We're got in cryptographical signatures, so like GPG signatures, basically.
We only think that they are really relevant if accompanied by a trust path, so basically maintaining a chain of trust.
And if you are not able to maintain that chain of trust, you might as well drop key verification.
And if a choice has to be made between the two, so if you have to choose between transparent source or opax source but signed, we now think that transverse sources are equating signatures.
Of course, in an ideal well, we would have both, both transparent sources and sign sources.
So we encourage now our package maintainer to establish communication with upstream, basically telling us our expectation on that front.
And it should be very easy to do, basically signing get tags is just what we need.
So we can base our package against your get tags and have that signature as well.
Maintaining a trust path for us should be very easy as well, it could be a signed commit, it could be a signed message.
One thing that upstream usually does is having a maintainer.md file into the repo, and you would have like the key fingerprint of every key that are expected to sign releases.
And just having signed commits, it's time you had a new key or remove a key from there is enough for us to have a chain of trust.
So yeah, that should be an easy step for upstream to do and we are, we are now establishing communication with them to hopefully being an ideal well of transparent sources and sign sources.
There are a bit extra resources about all of what I just said, so you have the XC, Archenex Security Advisory, which says again that part of the malicious code was confiscated into the custom made to our bowl and that the fix was for us to switch to the get tags instead.
You have this RFC that I can jointly wrote with David Runger, which is another Archenex maintainer, and this is basically what I just said in that talk, but in a very very long document with a lot of technical thing.
So if you want to, if you want to get further on this subject, there's a link to that RFC.
And you have also what source.org, that is a website that was created by KPCYRD, he is an also Archenex, also an Archenex maintainer.
And this website is very cool because you go to what source.org and then you tap for instance bash, and it will show you what sources every major list distribution are basing their bash package on.
So you can see that, for instance, arch is basing as package against a custom made to our bowl and fedora does it against the get tag, etc.
So that allows you to see what distribution are generally using for a package in terms of source.
A few contacts information, and this is Archenex.org, if you want to talk to me, I have a website as well, I am on the Fedivers.
And if you are going to get the slides, if you want to follow the links, and that's about it.
Thank you.
Any questions for all?
We've got a couple over here.
Yes?
Do you prefer to get tag over a get tag?
I mean, yeah.
So do we prefer a get tag instead of what do you mean get ash?
We have both actually.
Our package tooling is able to generate a a shot to 56 ash for VCS object, so we commit a tag etc.
So we actually have both.
Question?
Yes?
Would it help if the.
If the step from the sources to off the gen or the.
If we pay the title, would it help if the steps from going from the raw sources to those custom made to our bowl?
Basically, so out of general, where reproducible or at least more transparency.
Of course, that would help.
This is one thing we are trying to do with upstream and and discussing with them is to give give us the information on what's actually done from going to this source to the custom made to our bowl.
That allows us either to replicate it on our side.
So we grab directly the transparent source or even better if it is reproducible.
We could technically be based on the custom made source and with reproducible bills, it would be fine.
Yeah.
So that would definitely help us.
Okay.
One comment on that.
For dead end.
It's always.
That's the rule.
One to have source code in the prepared form of modification.
Which means all the generated files are not prepared form.
So we will always want to have the transparent form that have all our.
Through the way the generator stuff and generated on our own again.
Okay.
I mean that that is obviously a good.
Oh, I sure.
So Debian does it differently.
Basically what they do is that they grab the trouble.
Remove the remove the custom stuff and redo the custom stuff itself.
Basically.
Yes.
Sort of.
That's.
The expectation.
Okay.
That's expectation.
Okay.
That's also a lot of work.
I'm not sure if we are willing to do this.
I mean Debian is a volunteer base as well.
True.
But yeah, of course that would work.
But for this to work, we would also need to be like a policy wise and everyone doing that,
which is also a bit difficult for us to do.
But we have a lot of like more out of truth.
We have a half while that doesn't generation.
True.
What could be good, or as well as just not using all of that would be even better.
Other questions?
Yes.
There's some extreme cases through the work.
What do language what we're going to work for?
Where to generate a little smile.
I don't think actually you need to see maybe even five and maybe some jazz stuff.
Would you say it isn't always worth it?
And sometimes absolutely doesn't want to cooperate.
Because it works for them.
That happens like.
As so question is there.
There are some upstream build system that are complex to work around.
And sometimes upstream doesn't really want to cooperate with us.
How do we handle that?
Well, we actually decided to not rely on PiPiTarble for Python packages, mostly because of the same issue.
And some upstream would actually tell you that they only expect PiPiTarble to be built.
Yes.
And if it's the PiPiTarble created by some data population.
What if the steps to build this custom table are documented or inside the CI that we can load it?
Of course, that's better.
But it's also more work.
Ideally, we should find a good middle ground that would work both for us and for them.
So, yes, that would work.
And if we have to do that, we will do it.
This is not about reducing our number of packages because upstream don't want to cooperate.
We want to keep our the same repository state that we have.
But, yeah, that would be okay, but not the most desirable, basically.
Yes?
Is there a systematic way to check upstream change in the industry?
We now have one.
We did not have one before.
Oh, yeah, repeat the question.
How do we do if upstream recreated attack, basically?
How can we identify that and eventually confront them on that front?
Well, with what source.org, the website I said,
WebSource.org is now registering every of the commit ash and commit ID of a tag.
And if it changes, you can see that way.
And it also gather a list of files from the previous tag and the new one.
So, we can also compare the list.
We're usually relying on what source.org.
Or if our bill service, till have the old sources, then we can use that as well.
Yes?
So, that would be some view that you could actually use by two or all of the developers.
And if they were to confirm, you shouldn't waste them.
So, we do have a, yes.
Our packages reproduce by third party people, or just by us, basically.
Actually, both.
So, we have a report of our reproducibility of packages.
So, we have about 90% of our packages in the report that are reproducible.
And this, this rebuilder is maintained by us, but on a different machine and the bill machine.
Okay, I think that'll be the last question.
And, and we also have like third party user that run their rebuilder instance and that also shows the result.
So, you can compare our result with also third party result.
And they are usually on par.
Excellent.
All right, I think that's it.
Thank you.
Good.