All right, so the next step is going to be a presentation about
Thursday by Jason and Chris.
Thank you, Stefan.
Thank you, good afternoon.
Thank you for joining us.
We're going to talk about Tuesday, but do it yourself.
Linux, kernel, live patching tool.
My name's Chris Townsend.
This is Grace and Burino.
Today, we're going to talk, introduce ourselves who and why, who we are,
while we're doing this, then I'm going to hand it over to Grace and for the cool stuff.
What is TuxTake?
He's going to talk about the architecture overview, do it demo, and then we'll open up for questions.
So who are we?
Again, my name's Chris Townsend.
I'm a senior engineering manager.
Grace is a software engineer, the lead developer on TuxTake, or both from GICO.
If you don't know what GICO, who GICO is, where a large US insurance company, one
largest in the US, we are from a team that's called Containers OS and language
run times within GICO.
Our mission is to deliver a secure, compliant, and efficient containers OS and language
run time solutions.
So why are we building this?
So we obviously have a need for live patching the kernel.
So obviously there are distro-specific solutions out there.
There are paid solutions, but none of them really fit the need that we have internally.
So then we went to see, are there any open source solutions out there?
There is one from the Ginto Gen2 project called E Live Patch, but it's been pretty much
a band in last commit was almost six years ago, so it's pretty much dead.
So that's not really good solution.
So there's another one we found from the Debian project.
I think it was in DebConf, 2023, kind of trying to get interest from the community to do this,
but they're just a little, they have just kind of a sample patch, not a whole lot going
on there.
So why not do it ourselves?
We want to try to get a community involved, see, get the experts to help us out, and do
this out there.
It was not really a good open source solution.
So with that, I'm going to hand it over to Grace.
All right, so in summary, so what is talks tape, right?
So talks tape, it's a tool chain for creating, building, and deploying Linux kernel live
patches.
And it's not just one program, it's a whole ecosystem made up of several components.
This is kind of the component that we've decided will be once we hit the MVP phase,
and once we actually release, I'm going to be demoing a proof of concept, so things are
a little bit different than this slide here in its current state.
But just a little bit on how Linux kernel live patch works, there's a tool within the
kernel called K patch.
And what this does is it allows you to redirect vulnerable function calls over to past ones.
So you can fix a vulnerable function in the kernel, and then using F trace in the kernel,
you're able to then redirect that function call over to a compiled kernel module, and it
will run that instead of the vulnerable function.
But the thing with K patch is you can't just look at what they've got in the patch that
they submitted to the upstream Linux kernel.
The patches need to meet some certain requirements for them to actually be compatible with
K patch.
One of the good examples is you can't directly modify statically allocated data, like the
stack that the function takes cannot change in size, so there are some work around available
in order to get around that.
So what is the proof of concept seek to explore?
The automatic generation of raw patches was one of them, so we watched the Linux CNA, the
CB naming authority, see the patches that they submit, and we try to get a get dip of the
changes that they made to fix a particular CBE.
We also wanted to determine which CBEs actually affect a mock server fleet, so just because
a CBE affects the kernel version you're running doesn't necessarily mean that that vulnerability
is in the kernel configuration that you're running on your server.
Let's say a file didn't get included that the CBE affects in your particular kernel build,
then you're not affected, so we also wanted to kind of explore what the maintenance work
flow would be like for the developers who are creating these patches and reviewing the
K patch patches before they get sent out.
We wanted to build a text user interface to see kind of how that would all go down and
with the process would be like.
We wanted to see if a viable language for this, and if it was a preferable language for
us to build this in, for the whole ecosystem, parts of it, or none of it, and we wanted
to see if sufficient creates existed to do some of the heavy lifting that we needed to do.
Some of the crates we've used in this proof of concepts, using Tonic for GRPC, get to which
is like a lip-get wrapper, Tokyo standard, read it to a really great, to a library.
There was a wonderful talk on that in the kernel room, or in the Rustroom yesterday, I highly
recommend looking at that.
You rack for a simple HTTP client, and we're currently using SQLite, we're thinking about
moving the Postgres once we actually move into the next, the MVP phase of this, so all
of this is subject to change, it's all kind of fluid.
What is this proof of concept, not explore?
There's a lot of bits that we still need to take the time to continue flushing out.
One of those is the ability to do the conversion from the raw patch and convert it into something
that's K patch compatible, so any of the funding that you need to do in order to get it
actually to be run-able by a K patch, we want to get the ability to have that done automatically.
We're not there.
Similarly, since we're not actually building these K patch modules yet, we're not compiling
them, and since we're not compiling them, we're not deploying them, so there's no fleet
client and plash deployment yet, this is all upcoming, and then we wanted the ability
to submit the K patch patches for review and approval, all via this text user interface dashboard,
that's coming, there's a bunch more advanced features for patch creation that need to be
done, and then additionally, so we really want to support as many non-mainline kernels as
we can.
We've got some ideas on how we want to do that, but that is still in development, and this includes
Ubuntu kernels, which is the first thing we really want to target after we get
in mainline working.
All of this is to say, some of the functionality and this proof of concept is not representative
of the future behavior of how this is all going to work.
So let's get into the architecture of the proof of concept.
It looks a little something like this, and I'm going to break this down by components,
but this is the overall scheme of how it works in its current state.
So let's start from the top, so this is a toxtape CVE parser.
This is the thing that actually builds our database that we're able to store our patches
in and access by the dashboard.
So this will fetch the Linux CVE naming authority repo that they publish on Git, and
publish by again, I mean, and it, they give us die-ad files, which states, and I'll break
this down in the future, but give us information on what a patch was introduced in and
what commit the vulnerability itself was introduced in.
From this, we're able to generate Git patches from the stable kernel repo.
So when I've been saying the term raw patch, that's what I mean, it's just a pure Git
diff of the changes that were made.
So this is what their die-ad files look like.
It's in the format of the introduced version, the introduced commit, so this is when
the vulnerability itself was introduced, and then they state the version that the vulnerability
was fixed in and the commit where they submitted that patch.
So this is an example of a raw patch, so this is for CVE 2024, 4256 on the 6.10 train.
So this was a 9.8 base score CVE, and this would be the lines of code that need to get
changed in order to patch out that vulnerability.
So after we've gotten the raw patch itself, we need to fetch all of the other sort of CVE
metadata from the NVD API, the national vulnerability database, so we'll also contact their
API fetch things like the base score, the description, and all that stuff.
So the CVE parser is responsible for on an initial run, building our database, and then
on future runs, we'll update the database with any new CVEs or patches that have been
discovered since the last run, and you could run this as a cron job and just have it
have it go in the background once every day or so, however frequently you need.
This is a quick aside, we discovered a bug in the CNA's diagenerator, and they almost
immediately had patches within hours, which is really cool, we're very grateful for that.
So let's get into the touch-tap server.
This is our API for interfacing with that database, we're doing this with GRPC.
It's got a TLS, it's got a GRPC reflection, and with this we're able to fulfill requests
from the dashboard, the dashboards, how you're going to manage everything, what you're
actually managing is all stored in the database.
So this allows you to right now in the proof of concept, the main service functions we have is
you're able to fetch all of the CVEs that affect your mock server fleet, and you have
the ability to add a new kernel config that you then want to compile into a kernel, and
we have a kernel builder which I'll talk about next, which we'll compile the kernel and
use a thing called remake to profile the build and keep track of what files actually
got included.
So this is how the kernel builder works.
Once it spins up, it'll register itself to touch-tap server, and after it's registered
any time a build kernel request is needed to be sent from the TuxTap server, it'll find
an available kernel builder, and dispatch that build job.
It'll profile it, remake gives us a nice JSON explaining what files got included, and then
we're able to return a build kernel response which then gives us a big list of all of the
file paths of the files that got included in this particular build.
So we've built a dashboard as well, so this is the management interface for viewing and
editing those raw patches.
It will prioritize the CVEs by the base score and just straight up ignore any CVEs that
don't actually affect our fleet, and then it also has a second pane that allows you to
configure kernels via menu config, and then request that it get built and profile.
So let's get into the demos.
So we've got here, we're going to spin up the CVE parser, it's going to pull down any changes
that have been detected from the CNAs repo, and it'll generate a bunch of CVEs after
this it will contact nist to infect all of that metadata that we need from them.
And then we'll spin up TuxTap server, not much to see here, the TuxTap server is running,
and then we'll set up one instance of the kernel builder.
This will register itself over to the TuxTap server, and we can see that the server's got
a connection back to the kernel builder.
And that's the fun part, so we'll launch the dashboard itself.
So this is our two-week here, this lists all of the CVEs that affect our mock fleet, we've
got a 6, 6, and a 6, 11 in there.
You can see this is all the metadata that we get from NIST, the description, all that you
can scroll through.
And whenever you click on a CVE, we have different instances, depending on which train the
patch came from, if it was from 6, 10, or 6, 11 in this case.
This is our, I'm sorry, 6, or 6, or 6, 11.
This is our 6, 6 patch for this vulnerability, I'll close it out, and open up the 6, 11 one,
looks pretty much exactly the same, very few differences here.
And then let's get into the configs pane.
So this is how you're able to configure a kernel, and profile it's build, this is very
kind of primitive right now, but it gets the point across and has been helpful for us
to test this.
So we will fetch the 6, 10, 13 kernel, which is what I'm building, then you can fill building
it with defaults, and if we switch back to our kernel builder, we can see that it is now
building and profiling that kernel as it goes.
So we're keeping track of every file that actually gets included.
So as a recap, for the current capabilities, we are able to generate these raw patches,
and we can view and edit the raw patches, and because we're able to profile these kernel
builds, we're able to determine which CVE's affect the fleet.
Future capabilities is the ability to submit, review, and compile these manually formatted
patches.
The ability to auto-generate K-patch compatible patches, to compile those, then dispatch
them to the server fleet, and support for non-mainline kernels is also in the pipeline.
On top of that scaling, testing all of the fun groundskeeping that needs to be done.
So that's it for the demo.
Thank you all.
I'll just add that we don't have a public repo open just yet, but we're like weeks away
from having that.
So I'll make an announcement on LinkedIn, and you can find me, we'll try to message that
very well, but look for that to be done very soon.
Any questions?
Questions?
What is the always on the top?
So thanks for the talk.
I have a question regarding the generation of the live patches.
So as I've said, you are trying to turn it right from the GitHub, a K-batch, and then
apply it.
But many of the kernel patches that fix bugs are not just changing code, they also change
data structures.
That's why all live patches from Zoosa are right at require a lot of manual injection.
And how are you going to use to solve this?
Yes, so that's all stuff that we are discussing for the next phase.
We've got some ideas.
I don't want to get too indefinite because I don't want to lock us into an approach.
But I don't want to say it.
Sorry?
No, no, not immediately.
We haven't got any specific plans for AI in this.
Hello.
Can you elaborate a bit more about the term in specific CVs relevant to a survey?
So it's just some tracing, if effective line is called, or in the modern world?
Yes, so right now, all that we have in mind is just checking whether or not the file that
was affected by the CV is actually included in the kernel config that you then are in the kernel
that you have built.
In the future, we've talked about kind of doing some euristics to see whether or not functions
are actually being hit, how frequently they are to determine something like that, but that's
something we haven't got to explore yet.
We will probably be approaching that at some point.
Hello, the questions?
Yeah, I'm just curious you've been filming this in terms of live patching, but it seems like
a lot of this work here just in terms of determining the relevance of the CVs and the
whole config stuff.
It is useful on its own without even thinking about getting to the live patching bit.
So if you thought about just having that part usable separately for what you're planning
to do with live patching?
Yeah, absolutely.
It definitely could be useful for the ability to, because the general plan that we want
to do is the ability to then build a kernel that has the patches in it.
Live patching should be kept up as short of a time as possible.
You don't want to leave the kernel in unpredictable state with a live patch.
This is our, like we got to break the glass, we need to fix this vulnerability.
Yeah, that it is kind of the last resort to roll out a live patch.
Once we've confirmed that a patch works, we can then rebuild a new kernel with it in there.
Other questions?
Thank you.
How do you manage to stay on top of every CVs that is exposed in the kernel?
Do you manage to date on it?
I'm sorry, could you repeat the question?
Do you manage to stay on top of every CVs?
Check them, everything, and make sure that it is actually something that you need to patch?
Do you want to take that?
Yeah.
Right now, we will have, like, every enterprise is going to have, like, there are different ideas about what they need to do on this, right?
We have our own ideas on this and what we feel like we need to patch within our fleet, right?
But it's going to be, you know, a human manual process for us to do that, right?
And we will have to determine what we feel is that we need to live patch or not, right?
But I won't get into, like, what our criteria is and what that determine what we do on that.
Yeah, so, like, you know, every, I, that would be a difficult question to answer for everybody, right?
Like, like, everybody's going to have, like, different need, they're going to have their own idea of what is critical.
Well, critical, obviously, probably everybody's going to be the same.
But if we talk about, like, medium CVs or something like that, right?
Like, we will, we may feel like we have to do it, but other, by not, you know, we just have to do that.
So, we will provide the tooling to do this, and then you will have to decide what's important for you, right?
All right.
Okay?
Thank you.
Thank you all.