I'm a bit close.
Okay, you're rising.
You're so wet here.
Hello?
Okay.
All right.
Hi.
I'm Amy.
I'm going to talk about ABI stability in the next kernel.
How stable it is and how stable it could be.
I work at Chain Guard as a software engineer.
Chain Guard is a software supply chain company, so we build a lot of open source software, including Linux kernel.
For some of those builds, we want to reuse pre-compiled object files from another kernel build.
And when you do this, you quickly run into problems with the kernel's ABI.
So a lot of this talk is from that perspective of trying to reuse these objects and work around the instability that there is as much as possible.
So this is what I'll cover.
We'll talk about exactly what we're reusing, and we'll talk about the issues that one encounter is when we're using the objects.
How breakages start to appear and then how we could work around them and how the kernel could prevent some of these issues in the first place.
So what I say when I mean object reuse is that I've compiled something.0 during a kernel build at some point, and I keep it around somewhere.
Then at some point in a future kernel build, which might be a different kernel config or a different version, I pull something.0 into my new source tree, and I re-link it into VM Linux without re-compiling it.
This is kind of hacky. Why would you do this? For a chain guard it is fips. If you are familiar, you might be familiar with fips.
If you're not, it's a series of standards for processing government data and doing cryptographic operations on them.
If you want to comply with fips, you need certain things such as crypto modules and entropy sources to be certified, and the certification can only be applied to a binary, not to the source code.
We could certify the whole kernel binary, but that would prevent us from updating it ever until we build a new kernel and certify that, which takes time and money is a big lift.
So how do we actually do this? First, we want to enable config WR and config object to old WR.
When you're doing something unusual with the build system, pretty much any warning is indicative of a failure that you're going to see later on.
These are usually indicative of deeper problems.
Then we need to take our pre-built objects, copy them into the source tree, and write a make file rule to avoid re-compiling the object.
Just touch something.0 is fine.
So when you do this, how long does it actually remain compatible? It depends on how tightly you couple with the surrounding kernel code.
All you do is call print k. You can keep rebuilding for a very long time. If you do actual work and you call actual APIs, you break a little more often.
But generally, you can expect this to break around once per major kernel version.
So for example, if we build an object from 6.6.1, and we try to build that object with 6.6.2, things generally work fine, and we can continue with pretty much the whole 6.6 series.
The APIs are pretty stable within a major version, but as soon as we move to 6.7.1, things break pretty much immediately.
So what is actually breaking? Most of the time, it's not the source code compatibility.
We can take our source code for 6.6.1, put it in the 6.17, 6.7.1 tree, and recompile, and everything pretty much works.
The internal APIs and the functions that you're calling are pretty stable and don't change a lot.
And when our builds do break, it looks something like this. We either get undefined symbols, unreachable instructions, or BTFID mismatches.
Undefined symbols generally means that you're calling a function or accessing a constant somewhere that doesn't exist anymore.
Unreachable instructions is usually because someone was supposed to call you, and their code was refactor, and they stopped calling you.
And BTFID mismatches are due to the kernel assigning different IDs during BTFID generation, and your object disagrees about the ID of a symbol.
So if a compile does it actually work? Well, there are really only two outcomes. You either boot or you page fault pretty much immediately.
If you booted, you probably reached user space, and your module is either loaded or it's built in, things are great.
If you did not boot, it's probably a page fault from your functions accessing something they weren't supposed to, either while reading from the stack or while returning.
Generally, it's because your functions in your pre-compiled object and functions in the rest of the kernel that you freshly compiled disagree about what registers things are supposed to be in.
Generally, either works or it doesn't, and if it works, you'll know right away.
So how do we fix these issues and make it easier for us to keep reusing objects?
There's a couple things that are pretty easy to fix quickly. One of them is your tool chain. Generally, this isn't a big deal because a valid L object is a valid L object, but if you really need to, you can just pick a major version of your compiler and move on.
The second is BTF mismatch issues. When generating the BTF information, some of this gets embedded into the object itself in the data sections.
And the workaround for this is to just move all of your module underscore something macros to a different source file.
All of the BTF information lives with those. So you just stick them somewhere else. Then you don't have any BTF information to worry about.
So this is the first of the two difficult parts is making function calls. If you call a function in another object, the binary interface there needs to be pretty much the same as the function, your pre compiled object was expecting to call.
If the interface for the function is stable, then everything is okay. But even if the API for a function is stable, it's still possible to break the API by changing the type of an argument in the function signature or other minor details that change the way you're compiler decides to set up the stack.
What matters is the prologue of the function, we want the stack to be exactly the same way as we were originally expecting or at least in a way that looks reasonable to the linker and behaves the same at runtime.
The simplest way to deal with this is just to write a shim layer and only call functions that you control.
The universal print k, you call shim print k. The API and API of your function never change because you're in control of them and the real functions that you're actually calling can change as much as they want.
There are some downsides to this, namely there's a performance penalty for calling two functions every time you want to call one function. And of course you can't inline anything because that defeats the purpose of the shim.
The kernel also has these great validation tools for memory access, undefined behavior and code coverage.
Unfortunately, all of them work via compiler instrumentation and sometimes they inject function calls into your code, which have all of the same downsides as function calls that you make on purpose, except that it's not in the source code.
So I can't run any shims for these. So we need to disable them, but fortunately we can disable them just for our pre compiled objects, not for everything.
And sometimes the kernel changes something very low level about the way objects or functions are laid out.
They have two specific commits up here, which are renaming a data section in the object files and changing the register of the stack protector guard on x86.
These kinds of low level changes are inlined into every function and every object layout and everything the build system touches has to agree on these things.
When these kinds of things change, there is no getting around this. If you want to build with a kernel that has these commits from an object that was built without these commits, you need to revert them or patch around them, which you really don't want to do.
The whole point of trying to get a stable ABI is to pull new changes without having to patch things to recompile.
So could we have a more stable ABI without large changes to the way the kernel is developed? It can't be stable stable.
Part of the problem is just how expansive an ABI really is. It's more than compiler plus architecture plus source code equals ABI.
Anytime you embed information into a binary and other binaries and to agree on that information, it's now part of the ABI.
Even the IDs for the VPF type information, for example, need to be compatible.
There's a lot of good reasons for the kernel ABI to remain unstable.
And if you want to read more, there's a file in the kernel source tree called stableAPI nonsense.rst, which talks more about why having a stable ABI is a bad idea.
However, there are a few changes which could be introduced to kernel development which could make a stableish ABI.
It wouldn't actually be stable, but it would be stable enough that Linux distributions which can control tool chain kernel conflicts, target architectures, and kernel conflicts very tightly can build on top of this stableish base to make kernel packages which have a nearly stable ABI.
How do we accomplish that? First, everything I'm saying is relevant to LTS kernels. If you want stability, you're going to pick an LTS kernel anyway, because fewer patches mean that it's already more stable, even though there are no promises.
The problem is that an LTS version would have a stableish ABI would make an official pathway to something like this kind of project.
Even incremental improvements for resolving some of these issues would improve ABI stability a lot. If you did convince greKH that this was a good idea, what this would look like is these two changes to how patches are accepted into LTS kernels.
The first is freezing the function signatures for anything with export symbol. If you want to stable ABI, but you don't want to enforce a stable ABI, what you can do is enforce restrictions on function signature changes.
You change the entire content of a function at any time. You do not have a stable ABI. But if you're restricted in when you can change the signature, then you have a stable enough base to build a stable ABI on top of it.
Your function might do something completely different from version diversion, but you don't have to recompile the call it.
The second change is refusing patches that make changes to the low level build system primitives. These are pretty much showstoppers for ABI stability. They can't be worked around. You have to patch around them, and even just restricting these kinds of changes would make the ABI much more stable.
The result of this is not a stable ABI. It's a stable ABI. To take advantage of that stable ABI would still require some patches to the pre-built object, requires you to write code in a certain way, a disabled compiler instrumentation.
But it avoids the need to patch new code from the kernel when you pull updates and recompile with your pre-built object.
It's an ABI that's stable enough that the code that you're compiling to make your pre-built object doesn't need major patches to take advantage of the stability.
So why even do this? Who would benefit? The fifth case that we have is making a certified binary, and it means that if you use the least reusable approach as it is now, certifying your whole kernel, you are missing out on security updates to keep your certification.
This is the way many organizations manage their fifth kernels today. They certify a single binary, and they stay on that kernel for maybe a year, maybe two years.
The whole point of fifths is to improve security, but to achieve compliance with it, you need to forego updates and stay on this kernel for one or two years as vulnerabilities are discovered and CPDs accumulate.
That's all I have. Thank you for listening.
Questions?
So again, isn't this a case of the tailwagging the dog?
I think this fifths restriction to compile objects or certification only for binary objects is something that was related to worries about the compiler being corrupted or something like that.
Wouldn't it make more sense to lobby fifths to sort of update its outdated criteria?
Yeah, absolutely. Certifying binaries is how valuable is it really that I know that my crypto module works exactly the way I expect within the down to the assembly.
I also know that that module has like five or six CVEs that I'm not allowed to update to patch. I agree.
Yeah, the better solution is to stop certifying binaries, but personally I have no access to lobbyists who could do such a thing.
So I'm left with this.
I'm still a bit confused on what you actually then do instead. Do you certify specific object files from a previous kernel or is this about the specific kernel module that you're certifying?
Yes, we certify specific objects from a previous kernel build right and then it does can be built in those can be linked into a module.
I have the question, how do you certify based on the code that is internally used on the function calling and the branch predictions or just regularly on the functionality and how it behaves?
What is the perspective of certifying such a binary?
The process for certifying a binary is functional testing and code review.
So we build the object based on the object over, we send a kernel built from the object and the source for use to build it and the review process involves functional testing of what's in the object.
In this case, in our case it was an entropy source, so the functional testing involves entropy sampling and ensuring that it's ensuring that it's random enough and then also code review of the C source code.
I'm not sure if I heard correctly, did you say you hoped that the LTS maintainer's suit agree with this or you talked to them already?
No, I have not talked them about this.
I don't know if I agree that this is a good idea.
I think this is the path I'm left with for now, is patching my copy of the kernel source code to be capable of doing this.
This is a path that the kernel could take a better option, would be if FIP certification stopped requiring binaries.
So I'm not sure whether Greg actually wrote this table, I've been on since document, but I've seen him reference it a lot of times, so I think he would not be amenable to this.
I don't think so either.
But you can adjust the stable backport sloker to work around if they change something.
For the most part, we can implement most of these work around ourselves, which is occasionally have to patch new changes to avoid object layout or stack protector changes.
Does that answer your question?
Yeah, thank you.
When it comes to the changes that you had vice-poor, would you advise like a gradual adoption across the subsystems or it's just all or nothing?
It's all or it's all or the obsystem, right, for example, there's our there is subsystem that tend to fuck up the situation like more of them than others or like
I'm not sure if there are subsystems which would already be more or less stable, which could afford to adopt this more easily.
I think there are subsystems where this is more valuable to adopt than others.
The crypto subsystem specifically.
Obviously.
That's my use case here.
Thank you so much.
With the addition of Rust, even the compiler doesn't.
Even if you use the same compiler, you cannot guarantee a stable API for us to work.
So how can we handle that?
Sorry, can you repeat the question?
And it's quite here.
Yeah.
Even if you use the same Rusty compiler, you cannot guarantee the API.
If you stick to the same version of the compiler?
Yeah.
So what are the plans to handle that?
You can guarantee aBI stability if you stick to the same version of the compiler.
But even if things like the kernel config can change the way the API is laid out.
Yeah.
And when you're pulling in new changes from the kernel, these might include updates to the defaults or anything like that.
The compiler is just one variable and easily the most easily controlled, I think.
Anything else?
Oh.
Do you have any ID?
How many researchifications that this saves you?
Or what is like, how long does this certification of an object generally take?
Yeah, like how long in calendar time?
Yeah.
Without any of these changes, it lasts like two months.
With all of these changes, roughly one to two years before you have to build a new object.
Before you hit some change that you can't patch around and you can't deal with.
If you want to certify, if you go to FIPs to certify it.
Depends on what kind of certification you ask for.
If you ask for certification of your entropy source, you can get it back in three to six months.
You ask for certification of your whole kernel acting as a crypto module.
You get it back in one to two years.
Maybe three if you're on lucky?
No, I understand why you do this.
So there is a way of doing this just for FIPs compliance.
You're interested in and we've discussed it at the kernel maintainer level.
It's not actually making the ABI stable.
What we've actually discussed is working out the crypto modules so that we can compile them separately
and then feed the binaries into the kernel build.
So this is not a stable ABI, but we would be able to keep the crypto module stable across
rebuilds of the kernel, which as far as we think is sufficient to certify FIPs.
Or at least as far as my FIPs are to regulate as tell me.
That's sufficient to certify FIPs.
Would that also work for you?
Are you reusing the same elf objects?
Yes.
So basically it's a hack to the kernel build.
So you get the build of the object modules that are the crypto ones.
You take them out and certify that.
And then the next time you do a kernel build, instead of compiling crypto modules from source,
you just put the binary object files back and we build it.
So it doesn't give you the exact binary stability you're looking for, but it,
it, to us it seems to be sufficient to satisfy the FIPs criteria of not offering a crypto.
All right.
Well, I didn't know.
I was unfamiliar.
So David Woodhouse is the one at Amazon who's actually looking at that.
Microsoft will probably follow whatever Amazon does.
Okay.
Yeah.
I'm unfamiliar.
Anything else?
I think one's twice done.
Thank you.