Hello everyone, hi, so this is my, this is our talk with Lorena and we are going to
describe how we can investigate screen-considence with forensic katana track point in
Kubernetes environments and this cooperation with my supervisors, a professor of
Dr. Bruno and professor of Sarmor at University, PhD student in scientific
computing and Lorena is also a master student in focusing on trade detection and
Adrian and Victoria are also here. This work is in part with done with the Kubernetes
track-minter store working group and if you have any questions, we will be
happy to answer again. Okay, yeah, I'll try to speak up. Thanks Lorena, do you want to
introduce yourself and we can start. Good morning. So in Kubernetes, most
successful attacks are most invisible and that's because instead of using
common malwares, attackers are relying on native tools or API, like Cubsityals, Bash or
Python. In addition, they offer masquerade as legitimate workloads. For example,
deploying normal looking pods and also the persistence is maintained in an
native Kubernetes way, like demon sets or cronjobs. So long story short, the problem
is that attackers behave like normal common Kubernetes users. On the other side,
traditional forensics systems assumes that systems are stable, long-lived, and they
could be observable after the fact. But Kubernetes breaks all these aspects because
accessing nodes could be disruptive and difficult. So another aspect of classical
forensics systems is that they are based on these artifacts. But in Kubernetes
attacks, usually they use live memory malware and runtime state. So we said that
data are a funeral and the forensics traditional ways doesn't work correctly. And also
attackers knows that. So they designed cloud native and runtime-focused malware,
lived in memory and they use a living of the land techniques. Abusing the APIs,
native Kubernetes, and native tools like Bash or Python, that they are trusted by the system.
So they use also the RBAC misconfigurations and tokens abuse. We said that classical forensics
doesn't work, but how can we detect these attacks? Detection is still available, of course.
But the detection is just the first part of the security chain. So detection said that
is something bad happen. Yes, probably or no. But we have to know what happened by whom
why and what's the state of the system after the talk. So detection is just the initial
part of the security team, but it's not enough because it's based on signals. So even
based on if you're all data and we need something persistence. That's because we need
a way to do a complete investigation plan based on the state and durable artifacts. So
we need snapshots. Thank you, Loren. So as Loren mentioned, one of the problems
in Kubernetes' clusters is that container environments are FML. So workloads start very quickly
and disappear. And it's very difficult to preserve evidence during cyber security, during
cyber attacks. So one of the solutions that we are exploring is how to create a snapshot
of the container that can be used as digital evidence. Forensic readiness is essentially the
approach where cluster administrators want to prepare, practically prepare the environment
so that it can capture evidence during cyber attacks. So that when a security incident
is detected, the collection of snapshots of the containers can be fully automated. This
is essentially complementary to the existing audit walking mechanisms that would, for example,
capture the different events happening in Kubernetes and the different, for example, actions
performed by the APA server. So to understand the type of information, we can, we can essentially
extract during essentially cyber attack. We have to understand what the different attackers
can be. And there's a wide paper published by the quote, quote, native foundation. Essentially
describing the threat model that is commonly used in Kubernetes' clusters. So in this case,
we can have, for example, malicious outsider. This is an attacker that would be using exposed
APIs, for example, for a web application and trying to exploit, for example, non-vonar abilities.
This type of attacker can gain, for example, access to essentially be able to run
actions within the container. We can have also an informed insider, for example,
if you inject malicious code in a container image and someone deploys this in the Kubernetes
cluster. And then the attacker doesn't know what, for example, the cluster looks like. And then
they can try to exploit data and steal, for example, a security tokens. And the third type of
attackers, when, for example, they have already stolen service account credentials. And now they can execute
CPC documents, create their own malicious containers, and look for this in the cluster.
So the way the checkpoint mechanism works is we have integrated checkpoint API, which in the
Q-plet, that then calls the container runtime. So this could be cryo or container D. And we have
essentially checkpoint infunctionality that saves the configuration that was used to create the container,
and other metadata data. And then this API calls a tool called cryo, checkpoint risk or
user space. This tool essentially saves the state and the runtime state of all processes or
you can set up the container. So this includes everything from open files, network connections,
memory pages, unique sockets, a pipes, pretty much everything that you need to essentially
reconstruct the exact same environment. And this is completely transparent. So the applications
cannot detect that checkpoint is being created. But it also, it preserves the exact environment
including environment variables and memory pages that can be used for analysis. We have developed
the tool called checkpoint CTO that essentially extracts the information from the checkpoint.
And then it parses in this serializes the data so that it can be essentially displayed and analyzed.
So one of the challenges with checkpointing for forensic analysis is that if you
ultimately automate this, for example, to perform checkpointic every second, you quickly run out of
this space. Because for example, large language models can be very large or some applications can
use large memory state. So we have a organization taking that's used in lack of
compression. For example, where we keep track of which memory pages are being trained, modified,
and then we stay on this memory pages. So we have integrated this with the checkpointic mechanism
to be able to reduce them out of storage that is required to create essentially a chain of
checkpoint that can be used to reconstruct the timeline during a cyber security incident.
And this in combination with, for example, audit walks in Kubernetes gives you
exact information about exactly what happened, why happened and it allows a security team to
essentially prevent future attacks. So in this case, we created a common scenario where we
have an escur injection. So I'm tackled with sent an HP request that will essentially use
net cuts to create a reverse of this. So it will essentially execute code that will, in this case,
check if the admin password starts with the number seven. So what we can see here is in the first
checkpoint, we have listening socket. And then four seconds later, we have, in the second checkpoint,
we have a this established disappear connection. And we can see that the TCP stream input and output
use mt. This means that the application has already read the data from the disappear request and
We can see that there is a new process thread.
In this case with Phd7 that has been created.
And then we can essentially see
what are the memory pages of this thread.
And we can extract in this case the code
that was sent by the attacker.
So this quite interesting that, for example,
when malware wants to hide the actions
that are being performed, they usually use encryption.
So even if you use something like Warshark
or try to investigate the data that's transferred
over the network, you cannot see the encrypted data.
But because we check point the memory state,
you can see essentially the data after it
has been already encrypted.
Another use usage scenario is,
when you have, for example, reverse show.
So in this case, we have an attack where
the attacker creates a reverse show inside
of a container.
Now the important thing here is that after the container
exits, all the information about the actions performed
by the attacker would essentially disappear.
So what we have is a periodic checkpoint
that creates a snapshot of the running container
every two seconds.
And you can see here, for example,
what are the different processes that are created
by the attacker.
And just to demonstrate how this looks like in practice.
So here we have essentially a Kubernetes cluster
with a web application.
Essentially showing the same sort of reverse show attack.
So in this case, we are starting
to make check point.
And so this will create a checkpoint every second.
And then while the web application is running,
we send essentially the exploit in this case
that will create a reverse show.
And then the attacker in the first show essentially
lists all the files inside of the container.
And essentially the checkpoint mechanism
will capture all of this.
And here we can, for example, generate a JSON file
that contains the full analysis of all check points
that have been created sequentially.
And what we can see here is a lot of information,
but essentially the command in this case
that the attacker was running.
We can also see the memory pages
and all other information about it.
So the key takeaways is today, what we have in Kubernetes
is most of the research is focusing
on how to detect different types of attacks
and how to preserve, for example, walks.
But this doesn't give you a lot of information
when you try to, for example, extract evidence
of the different types of malware attacks
that have been performed.
You cannot see what was happening in the container.
And this is especially difficult with traditional forensic tools,
mainly because in Kubernetes, workloads
are being constantly distributed across different nodes.
And if you just create a snapshot of the disk space,
you will lose a lot of information that has disappeared.
And our checkpoint mechanism allows you
to essentially preserve this state during cybersecurity incidents.
And you can use this information to analyze what
has happened during the attack.
And thank you very much for listening.
So this is a link to the project.
This is checkpointsity, always the tool
that can be used to analyze checkpoints.
And the crew page has more information
how checkpoints works in Kubernetes.
I will be happy to take any questions.
Hi.
Yes?
It's because you're trying to capture more
to carry on within the containers as things are happening.
How do you tell the needs doesn't
about the results in the client that I can't care?
And any kind of curiosity, and any use of my old years,
to decide, I'm only going to have the most risky parts
of my work, for example.
So the way this works is, we're, yeah, the question is,
essentially, whether we have analyzed,
whether we have a way of defining what information
exactly is being captured.
And whether we have studied what resources
are needed to do this capture.
So the way checkpoint use works is we create,
like a full snapshot of everything
that's running inside of the container.
So we can choose, in Kubernetes, the minimum
the point of the portal unit is a pod.
So you can have multiple pods within a container,
but you can check point individual containers.
So for example, if you have a monitoring tool,
we don't need to check point this,
but you can check point, for example,
the application that is inside it.
And we preserve the full state of the processes
that are running.
So this is also that you can recreate,
you can restore essentially the container
from this checkpoint, and you can live it in frozen state.
So for example, you can analyze how the container looks
like at specific point in time.
And you can pretty much see all the file system changes,
the memory of the allocated memory of the application.
And for example, if you have a malware
that is running gone in memory,
you can still preserve this state.
But we also save information about TCP connections
and other sockets.
So you can investigate this type of actions as well.
Did that answer your question?
OK.
Yeah.
Just to add something, rendition to what said the medicine,
I mean, the detection is not enough,
but it still works.
So if there's an alert, based on, I don't know,
and I reverse the shell has been run.
You may be focused on that process and that pods and so on.
So the detection still works, but it's not enough.
And one last point was about evidence.
So in digital forensics people, usually care about whether
you can use the evidence that are collected in court, for example,
as whether they will withstand scrutiny.
So this is also essential to try to do
is create artifacts that can be useful in that way.
Yes.
You can go first.
I think you were first.
So external state in Kubernetes is represented.
Oh, sorry.
The question is, what about external state,
like a secret, for example, volumes, et cetera?
So in this case, the way this works in Kubernetes is it's amount.
So essentially, it's, for example,
how that is mounted inside of a container.
And depending on what it is,
like, for example, for temporary file system
that will disappear after the container exits,
essentially, that creates the checkpoint
in two saves the content of this temporary file system.
If it is an external amount, this something
we have been discussing, what we usually do.
For example, in Podman is just create a copy of the volume.
But in Kubernetes, this solves something
that we're currently working on.
And yeah, I mean, it's still something,
it depends how big the volume is.
There are different snapshots in mechanism
that you can use to create snapshots of the volume itself.
Did that answer the question?
OK, thank you.
And yeah.
First one is, where is the actual snapshots?
So in the view of that, it's a file that saves
somewhere from the connection environment.
In case it would be saved somewhere else.
Where would you save that?
In the second part of the coming from this, the entry will give.
Did you consider that the checkpoint risk
or the effect of the factor considering
that you would have following information
or the ones that they would consider?
Yeah, this is actually very interesting question.
So the question was, if we have, during the checkpoint
mechanism, where the snapshots are being created, saved.
And essentially, I guess the second part of the question
is about anti forensics attacks.
So whether the attacker can prevent us from capturing this state.
Could you explain, actually, the state of the file
would check upon that that's being attacked and created?
You can have a domain of attack from there.
So even if, for example, the attacker can delete, for example,
the checkpoint.
So these are very interesting questions.
And so the first part is, we save the checkpoint
in a folder called QBlet slash checkpoints.
So there is a specific location that
is used to save these checkpoints.
And because they contain sensitive data,
so they contain the full snapshot of the application's memory.
So this could include like secrets, user credentials,
et cetera, everything.
So we use permissions that don't allow anyone other than administrative
to be able to access this.
So if the attacker gets ruled access on the node,
and they can see any file on the node, they
can be able to delete the checkpoints.
But this is what we have so far.
We have also worked on anti-encryption,
so that we encrypt the data with public, private encryption
during the checkpoint.
So this happens as soon as the memory pages are copied,
and then before they return to disk.
And yeah, in terms of financial forensics,
so we have been discussing this actually different ways of handling this.
One thing is you can use a random number
for how often you create checkpoints.
But there are, in general, there are a lot of things that can be done.
Like, for example, attacker can use a system code that is
not something that the checkpoint into is able to handle
at the moment, in this case, it would create a error.
So what we can do is we can ignore the errors,
and still save incomplete checkpoint.
But essentially, there are a lot of, it's still a research area.
So what I'm trying to say?
Hi, yes.
What is the number having found in this abstraction?
I guess he wouldn't just run it every all the time
that you would buy a template,
that actually might happen to get a change of something
that is happening, start taking these platforms, I guess.
Yeah, I would just repeat the question.
So the question is, what is the overhead of this checkpoints?
And when do you start creating checkpoints pretty much?
So there are different types of overhead.
The first thing is performance overhead.
The time, for example, the application is not running,
because we are saving checkpoints.
And then there is also storage overhead.
How much disk space we need to essentially save all the checkpoints
that we create.
And the first problem is performance overhead.
Well, we sort of address both problems with memory tracking.
So the largest component of a checkpoint, essentially,
are the memory pages of the application.
Yes, and during live migration, we are addressing this problem
because we care about downtime.
So when we are moving one application from one node to another,
we want to keep the application running.
And we essentially do this iteratively.
So we transfer first the full state, then we keep track how
with which memory pages have changed while we are transfering the state.
Then we transfer the remaining over and over until there is small enough,
essentially, data that we need to transfer.
So we kind of do this in king here.
So we keep track of what memory has changed,
and save only this within the checkpoint.
And this allows us to reduce the disk space needed for checkpoints,
but also to reduce the performance overhead.
So the amount of time it takes to create the checkpoint.
And when we create checkpoints, so this also
what Lorena was describing.
So essentially, when there are many tools in Kubernetes
that allow you to detect, for example, other suspicious activity,
or malicious HTTP request.
And then you can configure different types of rules
that will essentially trigger the checkpoint mechanism.
So essentially, the checkpoint mechanism is
for preserving evidence while thread detection is separate
category where you can specify the rules.
Yeah, I hope this answers a question.
Thank you very much.