We will be talking about enabling architectural features in Debian, PAC and BTI on 64.
Thank you so much.
Hello, everyone. Good afternoon.
Thanks, please.
Tafit, you're always standing up.
So I would like to begin by telling you who I am.
Lots of people in this room know me already, but for those who don't, my nickname is Emma,
on IRC and with the match everywhere.
I've been a dead end developer for a long time now, about 20 years,
and since 2023, a couple of years I joined Arm as a dead end developer.
And when I did, when I got the job friend asked the great question,
and he said, why did they hire you actually?
Because Debian works already on 64,
but I don't see any point for you to be working there.
Which I found, you know, is a very interesting question,
even Debian works in my laptop, probably works in your laptop as well,
and it's not that we're closing the distros level and we're saying,
okay, work is done.
Distros development is a bit of an ongoing process,
and it involves a lot of different moving parts,
which is kind of the topic of this presentation.
And also, my friend, when you ask the question,
you know, I request a very long explanation about what
Linux distros is, according to me,
and you're getting the distilled version of that.
So you can see a Linux distra just as a bunch of packages put together,
maybe source or binary depending on the type of distro.
But there is more to that.
Of course, these packages are nice and low,
but you need to install them on a system otherwise,
you know, they're not much use.
On top of that, you could see it differently,
and actually consider the distra as the release.
So maybe the release process,
we had talks today about different sort of rolling distros.
Maybe that's what the distribution agreed upon.
Or you have stable releases that are cut at a certain point
in time with a certain process.
And this process is what also makes the distribution in a way.
The group of people that then come up with agreements
and discussions and then come up with rules and procedures
and try to follow them.
I want to talk about two talks,
someone said that the hardest thing is
of development is convincing people to do things.
And that's basically what this talk is about.
I would use an architectural feature,
as an example of enabling a change in the distribution,
which is, you know, touching different parts.
And how this process is really mostly about talking to people at the end of the day.
But given that the architectural features have two names,
it seems proven to tell you what this name is about,
Park and VTI.
That's the first thing we'll talk about.
Then I will tell you how we went about to enable the features,
how we were checking the status of enablement,
and what we're going to do next.
Park and VTI, with our disk to create the acronyms about.
So these are two security features.
The idea is to make a line of this draw better
by mitigating control flow attacks.
So the basic idea would be, you know,
in the past you had a very simple stack overflow attacks,
whereby you could have a buffer of a flow vulnerability,
right past the end of the buffer,
and then inject some shell code,
and you were ready to go.
Computer systems became better,
and now mostly, you know, not all buffer,
but actually the buffers are not executable in general.
So you cannot just do that,
but what you can do, you can mount more sophisticated attacks,
whereby you hijack the control flow,
hence the name control flow attacks,
and you essentially redirect the execution flow
to routines that would be legitimate
and they are remembered for a reason, for example,
G-lip C-routines,
but if you're clever enough,
you can point to areas called gadgets in this area,
and change them in mountain attack.
There are two main categories of such attacks,
one is called return-oriented programming attacks,
and the other one is jump-oriented programming attacks,
similar but a bit different,
and a pack is the first mitigation,
called pointer authentication,
and it's for ROP attacks,
and the other mitigation is called BTI,
branch target identification,
and it's for jump-oriented programming attacks.
I will tell you a little bit about both,
but not a lot,
because we don't have three hours,
and the pack would be essentially,
the idea is that in a return-oriented programming attack,
you're exploiting the buffer overflow
by overwriting the return address and the function,
so that instead of getting back to whatever it was called,
you begin that attack of chain-ing gadget together.
And the way pack mitigates this is by signing the return address,
so at the very beginning of the function,
there is a assembly instruction that signs the return address,
which is in our registry and on-arm,
and then just before returning,
this address is authenticated,
this signed address is authenticated,
the authentication fails the whole thing,
knows a runtime if it's successful,
everything continues working.
BTI is a bit similar,
but different,
the idea with BTI is that the goal is to mitigate
jump-oriented programming attacks,
where the idea is not to work with the return address,
but rather with jumps,
and in a jump instruction you can pass an arbitrary address,
so we cannot have something to sign at the beginning of execution,
we need to instead validate where we are jumping to.
So the idea is to introduce new instructions called landing pods,
BTI, branch target identifiers,
that we put wherever expected jumps are valid.
So if you jump anywhere else, you get an help.
So that's it.
There's a lot more, of course, to say about all this,
the end of the talk, I'll give you links to reference material,
if you want to read more.
How did we enable all of this in the distro?
Very simple, there is one flag to pass to either GCC on LVM,
and you can say dash and branch protection equal standard,
you enable both end of the talk.
Well, you can also enable either Park or BTI,
but the work we did was,
yes, to set this flag.
But the deployment has a little, like a few concerns,
there are a few things that we need to look for.
In the case of Park, it's fairly alright,
in the sense that we just add new instructions.
If the instructions are there,
there will be executed, of course.
All the CPUs that were maybe designed before all of this came out,
are not going to have problems,
because the new instructions were defined in the knob space.
So when all CPUs get knobs, instead of this stuff,
which is nice, we don't have to provide multiple packages,
you know, one package for more modern CPUs,
for example, and another one for the old ones.
So that's okay.
We just have to build a program and we're happy.
For BTI, it's a bit more complicated.
I mentioned that we're adding this new landing paths,
so you can only jump into specific places,
which means that when you're compiling, for example,
a C program, you may have multiple C files that become object files,
and then the linker puts them together.
Now, it would be really sad if one of those object files
would not have the landing paths,
and then we put everything together,
and if you jump in the wrong place, then the program crashes,
there would be very bad.
So to avoid that, the linker,
when it's linking, checks whether actually the old object files
have BTI enabled, and if they do,
it says, okay, this thing can have BTI,
and it does so by adding a note to the old section.
So the L file will have a note that says, okay,
this thing can do BTI,
which means that then the loader at runtime will check for it,
and we say, okay, this thing is BTI enabled,
we can go ahead and enable it.
There's a little trick here, a little thing.
So even if you build a simplest hell of word program,
you know, one line of code that says,
print hello in NC, that is going to use a couple of object files
from GCC and G ellipse.
And given that, as I said, all object files need to have BTI on,
it means that even the simplest hell of word in order to have BTI,
needs to be built with a G ellipse and GCC
that will have previously built with BTI on, right?
So this means that to enable all of this,
we first have to put it in GCC and G ellipse,
which means that we have to convince the GCC maintainer to do it,
and G ellipse maintainer to do it,
and then the package maintainer to also enable this flag
for all packages in the name.
So this was mostly work of communication,
talking to people asking nicely and so forth.
We got all of this in July, 22, 2024.
So all packages uploaded today,
so they began after this date, started getting BTI on.
How do we track all of this?
So how do we check which packages have it on,
which don't, and what has to be done?
So as I said, the whole idea is that you get a new note in the F,
so if you just write type redf dashnet notes,
and the executable, you will get the notes,
and then you can check if BTI is on or not,
which is pretty simple and nice.
So one of the things I did is,
because of course I don't just want to analyze LS,
I would like to know whether the whole debut in archive has BTI on or not,
and which packages still have to be enabled and was listed.
And possibly also regressions,
so if something has BTI and then suddenly doesn't have it anymore,
because somebody dropped the flag,
who knows why.
It would be nice to know that.
So I wrote a few scripts that use snapshot BbNOR,
I can make sure many of you know what that is,
but it's essentially a very cool service.
It's a bit like a time machine for the BbN archive.
You can hit as a very nice JSON API,
it gives you the list of all snapshots available,
and then you can point your source is list,
for example, at a certain snapshot,
and then you can install the B&C precisely,
as it was two weeks ago, for instance.
So this seemed like a very cool way to find out
what was the BTI enablement state,
to set a given point in time in the past,
and how it's progressing.
So the idea is to get the list of all,
well, to download all the architecture dependent packages,
from snapshot BbNOR get a second day,
for all of them extract the package,
go through all the elf files,
see which ones have the feature on,
and which ones don't,
and then print three graphs.
Downloading the archive doesn't take a lot,
it's about 20 minutes,
while being nice to the service,
and having only seven or eight parallel downloads,
and so forth.
And then running the analysis on our VM,
which eight cores takes about half an hour,
so this is perfectly fine.
It could be optimized,
but 50 minutes to analyze the whole day,
and archive is all right,
we don't do it too often.
And this is essentially the output of that set of scripts.
The red line on top is the number of binary packages
in the BbNOR chip,
at least one elf file,
a bit less than 25,000.
You can see that the red line does a bit up and down,
that depends on whether new packages have been introduced,
or some packages have been dropped.
But it stays fairly constant.
And then on the bottom you'll see the blue line,
and starting from when we got to Glypsy and GCC rebuild,
and we added the default flags,
then slowly all packages being uploaded to the archive,
started getting bit the eye.
So the blue curve follows precisely,
you know, uploads to the VM.
That's the only thing it is.
And it goes up,
but it seemed to go fairly slowly,
I mean there's probably a lot of packages that don't get uploaded,
so often.
So during that chat with Paul Heather's in Cambridge,
we from the release team,
we came up with the obvious idea of trying to rebuild everything,
and see which packages in the BbNOR would get to be
the eye if we were just to rebuild them with the right flags.
Yeah, so here, this is a great idea,
and it's like what we should do.
I made a list of packages that have at least one elf file,
we don't want to rebuild packages that only should pdf,
for example, because they don't get bit the eye.
And then I rebuild them all,
and I check whether they do get bit the eye or not,
with the scripts you've seen before,
and it turned out to my delight that many of them did,
about 7,000 source packages did get bit the eye on.
So then we said, okay, let's must rebuild them all,
right, let's rebuild them all,
so that we basically close that gap, you've seen earlier.
Before doing so, I double checked that all those packages
were actually uploaded before July 22nd,
otherwise it seemed like something strange,
they should have gotten bit the eye already,
if they were built already with a proper tool chain,
that was the case, so I didn't embarrass myself by sending a list of wrong packages.
And yeah, so I asked the release team to please rebuild
7,000 packages, which in the beginning you have to say,
schedule, being enemies, please.
And Sebasti Arama here did all the work of scheduling the rebuild,
so thank you very much to both Sebasti and Paul, if you see this.
Yeah, and the graph looks much better after that.
Those are all the rebuilds that were staggered across a couple of days.
And then after that, the line keeps on following pretty much precisely,
packages being added or removed, doesn't grow too much.
As expected, because we already got everything that would get bit the eye
with a simple rebuild.
So what can we do next to try and further shrink that gap in the next weeks?
So in that graph, you've seen that there are about 19,000 packages that have bit the eye,
and there's 5,000 that don't.
Those binary packages come from 3500 source packages.
And so the realization is that we can probably categorize them,
so we can see how comes those packages.
There are so many packages that don't get bit the eye, if you rebuild them.
And the realization is that GCC and LLVM are not the only compilers in the
produce L5's, Fabian is reading, because it is laughing because it's in the Rust team.
And yeah, so we started looking at this list and categorizing these packages by language.
So now a bit of trivia.
What do you think is the language, which is the biggest of all in all of this?
Haskell.
Yes.
There are a ton of packages in the Avian that are written in Haskell,
and they have L5's.
So not interpreted Haskell, but you actually produce an L5.
And what is the next one then?
Hello.
There's a big language.
There's a devroom about it somewhere.
Hello.
How's it going?
It's going.
Go.
Go is the next one.
And then somebody said objective common, and that's indeed in the list as well.
And then we finally have Rust.
The reason why Rust is, I mean, of course there's a lot of software in the Avian that uses Rust in different ways.
But Firefox is not part of all of this, because Firefox already ships binaries that are not, you know,
built with the Rust compiler.
So those are, that's a package that we don't consider in this.
These are pure Rust packages built with Rust, see and everything.
Okay.
So the obvious idea now is to fix these compilers.
For Haskell, there's already work on going at ARM, colleague of mine called Bill is working on this.
And for Go and Rust, there are two upstream issues.
And they, again, seem to be mostly about communication.
So there's a long discussion when people are getting about things.
And so I really hope this moves a bit further, so that we can get, you know,
they'd be a more secure essentially.
And then once that is all done, there will be a long tail of another about 1,300 packages that don't fall in those categories.
So there we can see our C++, but they just ignore C flags completely.
So they don't use the C flags you've set, they overrides C flags for various reasons.
I'm sure some are very good reasons and some are very bad reasons.
And so we will have to look into all of those.
And then there's probably also going to be a very long tail of strange cases like stuff with an in assembly.
God knows.
Okay.
So, yes.
To conclude, Puck and BTI are useful features.
I think in ARM64, and they will be available in Trixie.
All the packages you've seen before will be fully protected by both Puck and BTI.
And there's a Wikipedia page there with all the details about what I've talked today.
And much more, of course.
There's also towards the end of the Wikipedia reference section with a bunch of PDFs that really explain in detail.
Puck and BTI and control flow attacks and how they come to measures work.
So if you're interested, go ahead and take a look.
And yeah, I also wanted to make the point that distribution work is not only hacking and from to the keyboard, but it's also talking from to the keyboard, perhaps with other people.
And coordinating and tracking status and trying to convince folks that something is worth doing and so on.
Additionally, another point I wanted to make is that this reward is also about integration.
So if the integration is not done, the future is not there.
You can come up with the best possible feature in the work, very clever, supported in the kernel, supported in the C library, and the bootloader and whatnot.
But then if you don't do the integration work and put everything together, then the future is essentially not there.
And that's all I had.
Any questions?
Thank you for the great talk.
I'm mostly interested in those packages that I mentioned that North Sea Flex, because of the actually for my own personal reasons for, I run the debugging for the service in Devon.
And that's also impacted by that decision in packages.
So I was only if you have a list of the packages that like maybe we can work on and help with BTI and debug it for these.
Oh, this would be lovely. So I do have a full list of old packages that are not into those categories.
So packages that did not get the feature enabled and are not written in, sorry, Haskell, Rast and so on.
So all of those are good candidates to look at, about 1,300, so the list is long.
Yeah.
But some are immediately obvious. There are a few where you open the make file and it just sent C Flex equals.
Yeah.
That's it.
Yeah.
So that's a pretty simple one.
And others are trickier because maybe they use Arcane build systems and it's not that easy to figure out where things are at.
But yeah, I would love to work with you on that.
That's great.
There's also, I forgot the name of it, but there's a system to grab build logs essentially and see if certain flags are set or not.
Right.
That's, that's good.
Maybe we could use that as well.
Yeah.
That's good one.
I've been using that system for, but I don't have a full list of them.
So I just, if you do, that's, that's great. Thank you.
Thanks.
So you mentioned the sampler.
What if in line assemblers missing these landing paths or signing instructions?
Right.
So there is, there is the option of writing the landing path.
Right. So you can write your assembly in a way that you put the landing path in the right place.
And things work.
And then you manually have to mark the object as BTI enabled.
So then the linker will see, okay, this is, like, somebody claims this is BTI enabled.
Probably is.
And then when food swings together it will turn the feature on.
So that's essentially the idea.
There's an article also referenced in the sections that explains how to do all this.
So there is the way to, well, the best idea would be to not use this.
It's impossible.
But there are cases where you can't, of course.
And there's a way to do that.
Thank you.