Hello everybody, thanks for being here. I'm Simone Tirobozki. I'm working in a redata on the
CubeVirt project.
Okay, can you hear me now better? Thank you.
So here we are going to talk today about scheduling. Scheduleing is the process of matching
workloads to nodes. By default, CubeVirt is using the standard Kubernetes scheduler, the cube scheduler.
As you know, CubeVirt is about virtual machines, but a virtual machine at the end is
executed in a pod and the pod has to be scheduled. But pod is scheduled by the standard CubeVirt
Kubernetes scheduler. Let's talk a bit, let's do a diversion about how we define our
source needs for our workload. On Kubernetes on a pod and source on virtual machines,
we have requests and limits. Requests are the amount of resources
are lowered to be used with a strong guarantee of availability. You can request a CPU and you can
request a memory. Schedule is not going to overcome meet on a request. You ask for a certain amount
of resources and you are going to get that. Limit is the max amount of resources that can be
used regardless of any guarantee. The scheduler is completely ignoring limits. What's
the implication of this, requests should be typically less than a user. It's going to be less
than limit. This means that you are overcoming a bit and the resources can be available on an order.
If the user is over the limit, you are going to be totally in case of CPU. You are not going to
go over the limit or in case of memory. You can eventually be killed by the out of memory killer.
In CubeVirt we manage virtual machines. You define what you need on your virtual machine. You define
what you need a certain amount of CPU cores. You define that you need a certain amount of memory
for your virtual machine. The CubeVirt controller is translating it into a source of some parts.
We are not in general, we are not setting limits. We are not setting limits because we don't want
to have our virtual machines being killed by the out of memory killer. We are not setting limits
on CPU because we want to take the full advantage of the service available on that order.
Normally, CubeVirt is not over committing in terms of memory. If you require two gigs of
memory for your virtual machine, CubeVirt is going to render a pod with two gigs of memory. As
a request plus something more as a safety threshold for ancillary services within the virtual
home chat pod. On CPU by default, CubeVirt is over committing by a factor of 10. It means that
if you are requiring a 4 CPU cores, the CubeVirt controller is going to scale the pod, configure
to acquire 0.4 or 400 ml cores. This means that from the scheduler point of view,
a virtual machine with a 4 cores is just a pod with 0.4 to be scheduled.
The scheduling is the process of binding a virtual machine and then a pod to an order.
The scheduler should find an order that is capable of executing vector work load
ensuring that on the order you have a fire mount of a source to match at a list the source
that we require. The process of scheduling is a bit more complex. We have a predicate.
Whatever is a predicate for the scheduler is something that should happen on the order to be able
to handle the work load that it is going to be scheduled. The scheduler takes the definition of
the pod. It starts looking for all the nodes in the system. It scans all of them. The first step
is filtering. There are a set of filters, nodes that are not going to match the predicates.
Set on the nodes are going to be filtered out. They are not good candidates. Then there is a
calling mechanism. What is not mandatory? It is just a priority. It is just a weight.
nodes are weighted and the scheduler is going to choose the best fitting node.
Now, we have a virtual machines and we have pods. But virtual machine and pods are different.
We know, I mean, now, on a virtual machine, you have a different operative system. It usually requires more
source than a pod. You probably need to locate a few cores. Probably you are going to
locate a few geeks of RAM. The boot will start up time. It is as low because you need to
to boot a full operative system. Usually, a virtual machine is state full. We have data on a volume.
It is going to be safe. We don't have as persistent volumes. The virtual machines can be
live migrated between nodes without time. You cannot do that for pods. On the other side,
you can easily start a pod on a different node because of a start time. The starting time is short.
In order to scale out a virtual machine, probably you need to reconfigure it.
The user expenses are different because the user is not supposed to see his virtual machine
continuously booting. When you define a virtual machine, you will be surprised to be
as close as possible to Kubernetes. So, basically, you can use the same semantics using
node selectors or the finity pod. We tied it to as close as possible. Basically, one-to-one.
But what use really wants? I work on a virtual machine before working on Kubernetes. We know that
when cluster means migrating a virtual machine, they look to be able to select the node
where they want to have a virtual machine landing. You can find it in a virtual machine.
The semantics, the semantics. You expect something like that. On the other side,
on Kubernetes, a virtual in a live migration is just an instance of the virtual machine
instance migration object. It's a name-space-ed object, so an inspection can create it.
On the spec of this object, you can only specify the name of a virtual machine that you want to
migrate. This means that you have no control at all. It's up to the scheduler to select
a node, but it's going to fit this virtual machine. It's an out-defend from
what the users of the traditional virtualization systems or the school systems are supposed to do.
This has been debated a lot over the community. We had a few proposals in the past. We had
a lot of discussions here and there. It's not a new idea. We know that users are expecting
something different. Up to now we are still not able to converge on this. It's controversial.
Let me try to explain it. We know that experience that means users are used to control
where the traditional workloads are going to be moved to. Just because they are a bit
ready to do that, because maybe they are lying on existing patterns or automation,
because they are planning maintenance with a set of nodes after the other. The water, I mean,
it's very fast. They know what they want to do. On the other side, as a virtual machine
your owner, you don't want to see that your object should be updated or amended just because
a cluster admin needs to schedule that virtual machine on a different node. The goal of what we are
talking about is to allow a cluster admin to trigger a live migration of a virtual machine,
limiting the set of candidates of a validates node. The target node that is explicit
required for the actual live migration should not stay there. It should not influence the future
of a virtual machine. It's just for one off migration attempt and it should not bypass constraints
that are already set on the virtual machine. You should not be able to bypass what it's there
of a writing it. We are proposing now a really simple design. The idea is that directly on the
virtual machine instance migration object, we can add an additional node selector that is going to
be measured preceding all the nodes selectors and all the affinity that are already set on the
virtual machine. From a CLI point of view, it's just about passing an additional parameter.
At that point, you can inject on the fly, a set of additional constraints that should be
a specter for better migration attempts. The proposal is quite simple. It got a lot of criticism.
The first one, we are on Kubernetes, it's a cloud native solution. Kubernetes is the scheduler,
they use a should not interact, should not take over. Okay,
but there are the user, I mean. Then on Kubernetes, we cannot live in a greater
part when we have the node, it's not there, but of course we cannot live in a total
part in Kubernetes. And without Kubernetes, we don't have a virtual machine at all,
but now we have Kubernetes, we have a virtual machine, so we can somehow handle this.
Then of course, Kubernetes, we have a native paradigm to individually address some,
if not all the use cases that I presented before, doing adding attain central relations,
coordinating an accord on it nodes, there is an active way of doing that. Probably is not as intuitive
as it should be for experience at the cluster of the means that simply want to live
a greater virtual machine from this node to that one for whatever reason. Another crazy system
that got a easy in the community is that live migrations are a source of expensive
operations. We know that they are consuming a lot of bandwidth within the cluster, so they
cap at a certain number. The idea is that if we allow users to freely manage
live migrations, they can maybe start abusing on that introducing too much load on the cluster.
We solve that this, let me try to quickly explain it on Kubernetes, we have two different
admin roles, the cluster admin role, which is all the way to do whatever on any resource on the cluster,
and the admin cluster role that is available, that is all, that the normal it's supposed to be bound
inside a single name space to add user. The cube via the admin role, it's aggregated to the
admin role. It means that normally I think it's a common practice to grant the admin role to
select the owners inside an space where they name space owner. It's your kind of tenant, your own
resource there, and right now, by default, you are allowed to create and delete virtual machines,
and the virtual machine instance migration instances, and the Kubernetes RBAC model is purely
addictive. You cannot deny anything, you can only add, so since this is granted by different
installations, it means that all of your name space owners can trigger live migrations for the
virtual machines in the name space, of course. What's the issue? We have all your single migration
queue. It means that you can affect cluster critical operations like no that rains or upgrades,
because you can cancel a live migration request on the queue. So, in the next version of cube
of a cube width, we decided that the cube width admin is not going by default anymore to be
all over the to create and delete the virtual machines instance object, but this is going to be granted
only with an additional role, named the cube width migrate. As a cluster admin, you will be able to
grant it to individual to individual grant it to selected users, or eventually label it to be aggregated
as in the past to the admin cluster role to get back to the previous behavior. It's not an
API change, it's simply adding. Then, back to our initial problem, the month that guided us over
the years is the cube width eraser. If something is useful for pods, we should not implement it
only for the machines. The point is that the year we are talking about live migrations and
live migrations is something that is not relevant for pods. So, this is a weird machine specific
topic. We should address this in cube width. In the proposal, we had also fewer alternatives.
One of them is something that you can already do today without any, without any change in the
cube width code. You can set a temporary, not select or not affinity on the virtual machine, wait for
it to be propagated down to the virtual machine instance. If you configure the cube width instance
with life update or output strategy, which is very difficult, only at that point, you can take
a live migration with existing API, nothing special here, wait for the migration to complete,
and now, only now, you can remove the additional constraints that you set on the virtual machine object.
Why we don't like it, or at least, why I don't like it, it's an imperative flow, it's a
platform. It's still as to be somehow orchestrated, it will be completely up to the user.
It can mess up with DevOps and infrastructure as a code tools that are managing the
virtual machines on your craft. Another possible option, in Kubernetes, you can configure more
than one scheduler. You can add a seconder scheduler. We know that there are also loader-ware scheduling
plugins, so you can configure a seconder scheduler that is loader-ware. It's going to take
into consideration the actual source consumption of virtual machines, but still, this has to be
you have to still, you have to configure the seconder scheduler, and then each individual
virtual machine should be configured to be scheduled by that seconder scheduler.
And still, the scheduler is loaded-ware, it knows the actual CPU consumption on the nodes,
but still, it's going to schedule or according to the static observation that we set on
our virtual machine. And if you remember, we are setting by default one-tenths of the allocated
code. It means that if you need the four cores, the scheduler is not aware vector, and it's
going still to schedule only for 0.4. And this is also going only to affect the scheduling. It's
not going to watch over the time the actual consumption on your cluster and react to the balancing
the cluster. It's still up to you to monitor the cluster and eventually inject migration objects
just to get the scheduler doing something. Another option is to use the cube-desk scheduler
for automatic work or the balancing, eventually combining it with a lower-the-ware scheduler.
Since two months ago, the scheduler is, the opposite of the scheduler is at all, that it's monitoring
your nodes and it can decide to disquede or something. The scheduler has a ton of
machine, it means that the cube-desk is going to react and live a mega-it-it automatically.
Since November, the scheduler is a lot ofware. It means that it can, it's a new feature. It can
then, the scheduler will turn machines according to various CPU consumption. It's a really good option.
It's a really good idea to continue to balance your cluster. On the other side, this is just about
the scheduling. This is not affecting any out. Our schedule is going to complete the migration
because when we are triggering a live migration, the scheduler will trigger it, but then it will be
up to the scheduler to select the node and he is still going to count for what it knows.
In, but this is an interesting proposal. We are continuously working on it. We have one more
thing. We are trying to enhance it with precious tool information. PSI is a metric that is
supported by the Linux kernel since version of 4.20, so it's even not so new. It's supported at
a node and a Cgroup's license level. It's not a metric about the CPU utilization, but it's
exactly measuring the actual productivity loss caused by the scarcity of resources. And we have it
for memory CPU and IO. The kernel is measuring the amount of time where your Cgroup's license
is stuck because it's waiting for a CPU that it's not available at the moment. And the kernel is
supporting it. We did some experiments and the results are really convincing.
Unfortunately, PSI is still not a PSI matrix, still not a portrait by a C-advisor,
which is the tool that supports the matrix to the Q-blet. There is an open PR. This is pretty short
if it's from a one week ago. We are going to have it in the future. This is going to be a really
interesting way to automatically balance the cluster. So now we presented a few options. We have
a design proposal. The design proposal is still not accepted. We are a community. We have users.
So please make your voice heard. If you think that you need this feature even another feature.
If you think that you need something, please talk. I think that as developers, we have a vision
of a cluster. We have a vision of user needs. Maybe we are right. Maybe we are wrong. We want also
to get your feedbacks. Thank you.
Okay. First one. There was a problem in what I said because if I just sent a year ago to
I have my great and then from not A to not B and then my year will be good. And it goes back to
not C. That's completely unexpected. So somewhere must be stored, stay at not C or B or whatever.
Okay. So he said that if we simply add the addition of the constraints to the one of
the objects for the one of my patients, they are not going to stay on the virtual machine. Yes,
it's true. And it's absolutely expected. If you want to be a persistent change, please set it on
the virtual machine object. But that's a problem again. Because if I don't have the permission
to migrate, I have the permission to code migrate as well, which is completely either should be
allowed to do both on either. So the constraints are set of an object and if you are the
owner of the virtual machine object, you are all aware that we are at the object. It's up to your
cluster admin to decide if you are entitled to trigger a virtual machine's instant migration now.
If not, you can only specify where you want to have your virtual machine and sooner or later
it will happen, but it's not up to you to force it. And we can talk later.
So he's asking about the user of a pod disruption budget in Cuba,
yes, we are using them. We are using them to protect the virtual machine to be sure that it's
not going to be killed. And we are using a second PDB also to protect the target pod of the
live migration. So yes, we are using them. Next question.
So the disk of the virtual machine is a storage on a system storage on external storage,
depending on how you configure the virtual machine. It could be eventually automatically
a static. If you say that the virtual machine should be automatically a static, you can configure it.
We also have additional operators that are going to continue to monitor the node
to speed up the covariate process. If you need the eigenveller ability,
if I do a virtual machine. What if the node moves connected to the cluster,
that not with the persistent storage? I have two machines talking the same.
That's why we have a better additional operators. Normally, we wait.
So he's asking what is going to happen if we have two nodes that the node that was
hosting the virtual machine lost the network connectivity, but it's still able to
write on the disk. Potentially it could corrupt the virtual machine. We have locking mechanism
and we have additional operators that are going to use mechanism to be sure that the node is
a really bad if you need them. Normally, we have a long time out to be on the safe side.
Thank you very much.
Thank you so much.