Okay, so welcome everybody, thanks for having us, to this talk about keyboard and how it's used
let's say in the platform engineering team.
So first of all, little introduction, myself, I'm Nina, and today, together with my colleague
Thomas, which we are both platform engineers at Suze, where we're going through this and
a little agenda topics for today, we're going to talk about the challenges that may arise
while managing an internal development platform and we'll go into that in a minute, and
how to address those challenges and how platform team can enforce strict regulation on multiple
projects on a variety of customers, we'll see what this means in a really little time.
So let's start with something simple, so I hope everybody is familiar with this command,
but anyway, this is a simple, cube city or command, but what it will do, it will spawn
an engine export, and we will be doing this while specifying just to option flags, the image
and the village flag, and as we will see in a minute, here we are specifying the registry tag
and most importantly, the privileged option, which has some security concerns due to the fact
that it actually has some impact on the host where this internalized image is running.
So this is because, even if that is dangerous, this is a valid command, and as we thought a
minute ago, this actually does have succeed and existence shades apart. So, and this is
because Kubernetes, as long as the changes are valid and they're corrected, it will
happily accept it, and it will be, and they will be persistent. Even if this, those may have
consequences or they may be dangerous for our environments. So the question arises, what's necessary
to protect our environments, what can we do to do this? So to do this, we will need to enforce
cardrails. So let's briefly go back and review the command that we, so just a minute ago.
So here we can ask ourselves a few questions, and the first question that we can ask ourselves is
about the registry. So do we want to allow the registry.sucet.com as a valid OCI registry,
then next up, we specified the latest as the tag for the pod. So is this something that we
consider valid and is something that we want to allow in environments? And last but not
definitely least, as I was saying, do we really want to accept privileged pods in our environment?
So those are all questions that we can ask ourselves with this very tiny small and simple example.
But we can go further than that. There can be multiple and many more scenarios. So for
instance, if the image that we want to use a cryptographically signed should the pod have or not
have certain labels and for instance, regardless in PCI devices, should we allow the pod to
access to the side GPUs? This and many more questions are valid and we will get this to this in a
minute. So this kind of highlights very simply why will we need Godrails? And the reason is that
even if the changes are valid, we want to reject the configuration change changes that we deem
as bad. So that's before they actually do persist. So the reason is that if we don't do that,
our environments can become insecure and they this is definitely something we don't want. And as
well, managing a lot of code manually, YAMO code for those changes becomes not feasible and
especially impossible at scale. So this kind of requires automation in the mix and automation
possibly as code, we had policy as code. So to the core component that we would be leveraging
to achieve this is the community submission controller. So and what it is, it is a piece of code
that all it does it intercepts the requests coming in coming in and once authenticated and
authorized and once this happens, it actually can validate it or mutate it and this it does that
via the related webbooks. So for the sake of this presentation, we will just be considering
the validation of webbooks. So in our environments, we will be leveraging multiple admission
controllers and all those controllers together will make sure to allow or change or reject
the changes. And this is because we will be leveraging multiple policies that may be small,
but together in isolated, they will allow us to maintain that security and forcing the guidelines
and enforcing the governance that we need. So we spoke about our internal platform but
why we do so. We are a platform team. We need to provide tools and services and enable our
customers to spawn their workloads and do their tasks on different clusters. And at a very high level,
the internal platform, the internal development platform that we are using is composed of
those three parts. At the center, we have Ranger which is the open source container management platform
of choice and then we have the downstream arcade to clusters. And of course those are the arcade
arcade to is the Kubernetes distribution which is supported out of the box via Ranger and which
shows it because it's lightweight and secure. And then we have the NHTI, an hyper-commeraging
infrastructure in the form of arbister because this helps this is due to the fact that we can
actually manage compute, networking and storage as native Kubernetes objects. So and most importantly,
in both arcade to clusters and also on the arbister HTS solution, we are running Kubernetes policy
Engine which is what we will be going into detail in a few more slides. So now we get to the point
what is keyword and how can it help us. So first of all, it was to say that Kubernetes
CNCF policy engine and it allows us to use what we already know. We don't need to learn
a DSL or a new programming language or something different. To write the policies that we will
be using and those policies will be turned out turn into some web assembly with some policies
as artifacts and those artifacts will also be able to be embedded in FSA, CICD pipelines and
and such. So from now on, I would like to pass microphone to Michael Lee.
Hello. Now I would like to talk to you about platform engineering at SUSA and provide a few
examples of some policies that we've recently ridden. So let's first talk about our customers.
Our platform is the primary way for teams and SUSA to run their workloads and with us, they run
production services and they perform research and developments and they also can give sales
demos to prospective customers. And we need to enable our customers but at the same time,
we need to project them from each other. This is because we don't have unlimited resources.
So we can't get everything to one team. They shouldn't be able to do everything. So we need to
have some level of emissions and some environments are more protected than others, for example,
certifications where we actually run common criteria. But there will always be a new feature
a technology that our customers need and they need it right away. So let's go over a few examples.
Hey platform team, I need GPUs. For some reason, multiple teams demanded GPUs last year.
The problem is that everyone thinks that they need one and they need the most powerful one possible.
And this is usually because of they don't know what resources that they actually need to run the
workloads yet. And GPUs are scarce so we can't hand them out like candy.
And we run our GPUs in Harvestor and they're we split a physical GPU into multiple VGPUs
and attach in the virtual machines. And each VGPU has unique name. For example, here with the
Tecton 27A with the number at the end and the problem is anyone who knows that unique name can
attach it to their VM. And here I'd like to press that valid YAML is not the same as a valid
change. Just because you know the name of that unique VGPU doesn't mean you should be able to
attach it to your VM. So we need something more. If Kubernetes doesn't have built-in controls for the
GPUs, we're going to just build a keyboard and policy for virtual machines instead. And our goal here is
that Kubernetes will ask our policy whenever it's changed to a virtual machine happens. So let's look
in an example request. We can see that it's for a virtual machine. We can see the Kubernetes
name space. And we can see the unique GPU name. So how should we react? If we create a list of
allowed namespace and VGPU bindings, we can get the namespace in the VGPU from the request.
We can see the pair, see if the pair is in the list. And then with our specification is
if a VGPU in a namespace are bound, we will accept it. But if they're not, we'll deny it.
So here's some example code where we're checking if the GPU is allowed. And the first thing
that we do is we iterate over the list of bindings. And then we check if the GPU and the namespace
are bound and we return truth they are. And we return false if there's no match. And here's a
level up where we're actually calling that function. So the first thing that we do is we get the
virtual machine from the request. And then we get the namespace and the GPUs from the virtual
machine. And then we iterate over them. And then after that, we check if the GPU is allowed.
And if not, we reject the request. And if it wasn't rejected yet, then we allow it.
And here we can see where we're setting with the policy. We're creating a cluster
mission policy resource. And I'd like to draw special attention to the settings. We're
setting the namespace device bindings. And anytime we want to add a new GPU, we would have to
update it here. And here we can see it in action. We're on trying to add a GPU to a virtual machine.
And because it's not bound, keyboard and denies the request, where we can see the red air message at
the top. So for a second example, hey platform team, restrict that network. Because the
harvester is self-service, anyone can create a virtual machine network and harvester.
The problem is that if you know the VLAN, you can recreate it in another namespace. And some of
our networks are very secure, like for common criteria. And we need to ensure that they can't be
recreated. And while we can hide this information, it's better to prevent the change from happening in the
first place. So how should we react? If we have a list of allowed VLAN and namespace bindings,
we can get the namespace in the VLAN from the request. Then we can see if the list of our bindings,
and the specifications would be, if the VLAN and the namespace are bound, we accept it.
If the not bound, we deny it. And if neither the VLAN nor the namespace are bound, we accept it.
And this is to allow for unrestricted VLANs. So if we don't care, and there's no protections for the
VLAN or the namespace, then we just allow it by default. So here's looking at some code.
We're calling that phone calling this. So the first thing that we do is we iterate over the bindings.
We verify that the namespace and network are bound. And then the end, we return true if they're
allowed and false if they're not. And like the previous policy, we have to set up this cluster
mission policy resource. And the same thing with the settings, where we have the namespace VLAN bindings.
If we ever want to add a new namespace or VLAN here, we have to update it here.
And here we can see it in action, where we're trying to create a harvest or virtual machine network.
But the VLAN's protected, so we can see that it's denied.
For our third request, hay platform team restrict those big partitions.
Make partitions are isolated instances of the physical GPU and they're very useful for us to
give our users access to GPUs and Kubernetes. The problem is that we have a limited supply
of them and some of them are more powerful than others. So we need to control which
make partitions we give to which teams. So let's look at an example of request. We can see that
as fair pot. We can see the Kubernetes namespace and we can see the make partition name.
But instead of requesting a unique device, we're requesting a resource and that comes with a few
problems. So what's different with the resources? So a big partition name like the NVIDIA.com
make 1G12 gigabytes is not unique to one specific device. And here we're actually requesting a certain
amount of them. So we can request one of them. We can request two of them or we can request none of them.
And this raises a problem for us which is how do we keep track of the make partitions that
namespace already has. So this is different than our previous examples because we had all the
information that we needed in the request itself. So we had the Kubernetes namespace. We had the VLAN
and we had the device name. But now we have to keep track of account. Luckily, Kubernetes have
something built in which is the resource quota. And this is used to limit the amount of resources
that a namespace can consume. And this is usually used for the CPU or for memory. And we can
use it to keep track of the make partition. But it only keeps track of resources that it knows about.
So if the make partitions in the quota, it limits it. But if it's not, it's allowed without limits.
So how would the scale? If we use it as is right now, we even have to add the make partition
to every resource quota in every name space anytime we add a new make partition. And if we
forget a name space, then that make partitions allowed without limits. So this is where the
keyboard and context of where a policy is coming. And they allow us to reach out to the Kubernetes
API whenever and retrieve another resource. So in our case, we would actually want to get the resource
quota. In our goal would be to deny request if it's requesting a make partition, but it's not
in the resource quota. So here's an example of where we're trying to do that. So what we're doing
is requesting all the resource quotas in a name space and returning them. And then here's where we're
calling that function. So we're trying to get the list of make partitions from the name space,
iterate over it, and then validate that the make partition is within the resource quota.
And the best thing about this is that we don't need to add any settings for a policy.
Everything's actually controlled by the resource quotas. So we just have to update it there.
And here we can see an action where we're trying to create an Olamopot and use a make
partition that isn't bound to the name space. So we can see that Kubernetes is denying the request.
So the takeaway. As platform engineers, we can translate the business requirements to isolated
policies of keyboard. And the best thing about this is we can also do a program in languages
that we're already familiar with. So going, row, going, or rest. And there's already a vast
library of existing policies out there. You can discover them on artifact hub. And this is where
we publish our policies. And you can also find policies that are maintained by the keyboard
and team itself. And hopefully we've convinced you to try keyboard and you might already have
some guardrails that you'd love to create in your environment. My tips to be to start small and
let me your scope to be sure to add unit test as well as end-tests. And when you're done,
publish it to artifact hub and share it with the community. Thank you. Any questions?
Yeah? Yeah, if you want to take this. Sorry. And the question was, what's the difference with
the reason why we chose to use keyboard and rather than other solutions that the similar jobs
is because we thought that for the use cases that we were trying to address this tool was
the best fit for us. And on top of that, on top of that, we have a close relationship with the
keyboard and team. So just within suicide itself. So that would probably help us solve
questions and doubts sooner. So that was just where the two reasons why we chose this solution.
And on top of that, the fact that we were able to do this by writing, go-code or rastcode
tools that we programming, which is that we are already using. So that kind of helps us not
learning a new thing and that spends things up. That's it. I hope this answers your question.
Any other questions?
So I don't think that we can actually reach out to other, oh, sorry. So the question was basically
when we're reaching out to, sorry. Yeah, um, you might. So when we reaching out to other services,
if there are any of you, and any of you, and any security concerns, the experience.
Yes, it does. I mean, there's the keyboard and CL command that allows you to actually test it
locally. And as Thomas was saying, by writing into and test, you can actually, you can actually
test it on your local machine before action. And get all the button information that you can
require. Sorry, I did not hear here, but your question at the beginning, because there is a little bit of noise outside.
Any other questions?
So the question is out, how to,
okay, so the question is how would we install a keyboard and on a cluster? And a keyboard
is actually a CNCF project. So it's not really connected to the rancher. I think that there is a
supported version of keyboard and that you can have. But for everyone else, you can just go to the
keyboard and project and install it in your cluster. And it comes up a home chart. That's what we're
doing, and it's pretty simple. So if I may add, we will be sharing those slides. And there are
all the links that we probably provide more details and information on how to get this installed and
write. Any other questions?
Because you showcase how it works with French and overseas, right? The most
question we're going to understand is very low. And you ask for it, like,
as a button all part, with all of you, you have to implement this in this specific language,
I know real-world health call, recovery is just a computer I have, and as a
kind of user, then see how it's going to be like, why do this which?
The scope of this is just to explain our experience using this tool to achieve our
use cases. And that's why we chose cute wooden. And this is not about convincing to switch
from one solution to another. So every solution has its own pros and cons. And about the main
one, again, why we chose this and against other solution is because we could use tool,
we could use knowledge that we already have within the team, we go and, and was. And also the fact
that being creating a wasm artifacts, we could store those artifacts on, you know, OCI registry,
and we could hook up a CICD pipeline and run it and run it as part of our pipeline. So that was,
our use case, but again, it's not that we're trying to convince to switch from another tool,
that's the point. Yeah, I mean, also in the post-cube, if you believe you want to down,
like, it's not long, long, long, long, long. It's not long, they're being developed. You have to
pick something. So I was curious if you have any, if you were a certain selling
keyboard into me as a k-pad user, I'm trying to be an engineer, but yeah, the
try it out. And we can take this outside if you want to discuss this further, but
I'm saying it in a very, not harmful way.
Joking, of course. Any other questions?
Okay, so thank you very much.