So, hi everyone, can you hear me, little back, oh, nice. So, it is my first time in Brussels
and today we will present something called, how you can accelerate your CI pipelines with
help of a tool, which we have open source and build called V cluster. So, a bit about me,
I am Rithi Kroy and I work at a company called Loft Labs as a platform at Workgate.
I am a CNC ambassador as well.
Yeah, hi everyone, my name is I am Bartek and I am working as principal to work at Workgate
at Loft Labs, founder of Cube Simplifier and Bill Seve written a couple of books on certifications
and also keeps not happy to be here.
So, whenever we talk about environments in an organization, there are three types of
environment, if we simplify it. One is a developer environment, where your developers
go and build things, it can be remote with a lot of CD's right now or it can be
their local machine as well. The next is testing environments where a lot of testing
and all those things happen, where you provision your name machines whenever there is a
specific test case or specific PR. The last is production environment, where you just
push the binaries and all those so that your consumers can consume it.
So, right now what we have seen is communities is right now kind of a defective standard.
So, as a large scale distributed system, we see the bus about Kubernetes wherever we go
and the reason between widespread adoption for Kubernetes is because the community between
it as well as it supports a lot of use cases. You can have a abstraction for a lot of
nodes and all those as well as it is kind of similar that it is very easy to
spin of environments in that. So, whenever you create environments inside of it, it is
simple as that. But whenever we talk about testing environments, the traditional set of
us for Kubernetes native testing environment that you have different namespaces for different
environments of testing. But most of the times that is okay when your developers are building
something, but the problem arises when your developers are building a new CRD or something
which are more cluster level. So, the problem arises over there. And that is not a
only one type of problem that gets a RIOs. There are other type of problems. Sometimes there
is an inconsistent APS server. So, the problem is whenever you are test is creating thousands
of secrets, there is a heavy load on your APS server. Or sometimes there is resource
change as well. So, there is pod evictions and noisy never problems. Then you are problems
like delayed provisioning or slow test cycles. And the last type of problems networking
difference between your different type of environments. So, those are couple of traditional
challenges which we face. So, to get up from this challenges, we have seen a lot of
organization create a cluster. But the problem is as you create a lot of clusters, a cluster
of your PR, a cluster for each of your developer, a cluster for each of our environment.
And you start creating and deleted them. There stays a state where there is a lot of
unnecessary clusters. So, with the CI bottleneck, the problem is CI's time consuming. So, whenever
you create a full cluster in your CI pipeline, it takes time to be created from end to end.
And creating it takes time creating it takes resources creating it takes CI minutes as well.
That exists resource intensive. Understanding how much resource your cluster needs, depending
upon the task cases is complicated even. You define a couple of times of common clusters.
But suppose your cluster needs a specific GP to test out the new PR. So, how do you do that
smoothly? And the action time is again. So, you do not want your actions to be running.
The cluster is provision. You can use an M space, but again there is problems which are
associated with it. And the solution. So, when we see GitHub actions and all those things
become popular, the main consequence containers are fast. And it helps you to create
quick pipelines. It is very fast in comparison to each case or creating a new whole cluster.
So, we learn from those concepts. And then problems like CI is syncing any UI use cases
is also important. For example, you are having a specific CRD for a specific
name, space or your specific PR. Then how do you make it simple? So, those were the kind
of problems which had a solution. So, we learn from those. And then focusing on the type of
environments you have. You can have a slow but expensive thing,
call separate clusters. But there have a problem with that engineers need to wait. There
is to duplicate it tools and all those things. And again if you go back, the simple approach
was to having namespace isolation. But again the problem is lacking permissions for your
clusters and for your testing environments, resource and CRDs. As well as the weak isolation
between them which is sometimes problematic. So, with all those things, there was a requirement
to be a little ground. And there we came in. So, we are weak cluster. And we just
were launched couple of years back. And right now we have been used by a lot of companies
as well as growing open source adoption right now. And what we do basically is. So, we create
a, like you bring your host cluster. So, a specific host cluster. And we try to create
virtual clusters on top of that. And when we create virtual cluster, it is fast because
we create a containers and your APS server in your namespace, a specific dedicated namespace.
And then it is fully Kubernetes compatible. So, it is a certified Kubernetes distro. So, you
can have anything you are running in a normal Kubernetes distro. And you can use that.
The other part is you can have specific CRDs from the host cluster over there. So, whenever
you are running a simple CI pipeline, it can take few seconds to create the virtual cluster.
And with the virtual clusters over there, you can think things from your host cluster. So,
your CI pipeline has everything that is required for it to run. And you can also isolate
whatever you want to. And on a way, it is super cheap and super green because if you are creating
new EKS cluster or something like that, you need to pay the control print cost for each of those
cluster. But if you have a dedicated host cluster, where you are just using the CI whenever
it is required. And you have cluster auto scale and all those things enabled. So, it is
more simpler. So, that is the mic switching. All right. So, you know about me
cluster now, like me cluster lets you create virtual Kubernetes clusters on your host clusters.
So, you bring a base cluster like mentioned and then you will be able to virtualize that.
Each V cluster comes with its own isolated control plane. So, don't think that whatever you do
keeps it, you will get nodes, the request goes to the API server of the host cluster. Now, it goes
to the API server of the virtual cluster that is created. So, it sets somewhere between
the namespaces, which are on the left side, which are obviously weak restricted. But yes,
they are obviously cheap and in more less than the per cluster side. So, that is the architecture
how it works. So, if you see you have the host Kubernetes cluster, the physical host cluster
or any Kubernetes cluster EKS, EKS whatever that is or even your local kind Kubernetes cluster.
So, the virtual cluster, what happens is there is a component inside V cluster called
Sinker. So, whenever you create a pod, let us say QCTL run, NGNX, HIFANIFAN image, NGNX, the pod,
the request goes and it is stored in the ETCD component of the embedded ETCD component of the
virtual cluster. And then the Sinker copies that onto the host cluster. Now, as soon as it copies
that to the host cluster, it is the regular Kubernetes behavior. The regular scheduler kicks in because
it has to run a pod. So, it schedules the node to the best fit, reduce the pod to the best fit
node and it runs that. So, and then again Sinker puts that it is in running state. So,
Sinker is the component that talks to the API server. So, let us try to understand that
with the demo. Hopefully, it works. I mean, it was working. So, you let us say there is
there is a developer. When you what happens is that is a pretty standard scenario. You write
some code and your application is deployed. What you want is ideally as a developer that I have
something feature that I have implemented. So, where is a PR? For that particular PR, I want to do
this and I want an environment for that. So, there should be a way that as soon as I add a particular
label or whatever, I should be able to get a whole Kubernetes cluster and my application rebuild
packaged, my services deployment, ingresses, everything is created and that is deployed onto that particular
cluster. And I get an ingress which I can test whether my new feature is working or not. So, that is what
this demo is about and how V cluster helps to kind of do that in a way. So, this is the host
cluster that I am having. So, QCTL get nodes. So, the host cluster is there. Right now, there
is no virtual cluster in place. So, what we will do is this is the repository. So, let us say
a developer comes in and they create a new branch from the main. Now, they go to the app
edit, hello for STEM and commit the change, commit to their branch and go to pull request,
create a pull request, reach 100 float in now. Now, what they can do is they can add a label
test. Now, as soon as the label test is added onto this particular GitHub repository, there
is an action that gets triggered. What that action is and what it does is. So, it runs on the
pull requests that are labeled and with the label name test. So, as soon as there is a label
added to the pull request, this particular workflow gets triggered. So, it checks out the code, sets
up go, sets up code. So, code is a tool to build goal and applications. So, you do not need
to do that. Then, it logins to the Docker. I have already put my secrets inside actions. So,
the Docker have username, Docker password is already there in the actions. Build and deploy the image.
So, it is building the new image adding a tag to that. I hope it is clear if you want me to
zoom in I can. So, it is adding the tag. It is doing the code build for that particular image.
Now, it is also generating the deployment manifest. How it is doing that? I have a ginger
template. So, let me show you that. So, here I have a deployment. So, it is a simple deployment,
and in the image, I have it as a variable. So, that is a ginger template. And in the
ingress as well, I have a variable, ingress tag. Because for every developer and every
PR, I want to separate ingress address to be put. It should be dynamic. There should be like automated
stuff for everything. So, next step is what it is doing. It is replacing the variable value.
So, it is getting the deployment tag. For the ingress tag, it is generating like the current
full request number dot v cluster dot take. Now, dot the v cluster dot take is pointed towards
the ingress controller that is running on the host cluster. So, what internal it will do is
it will when it creates the ingress with PR. Let us say this is whatever this is 12 or 13.
PR 13 dot v cluster dot take. That particular ingress will have the deployment of the latest
code that is committed by this particular developer. Now, pushing the deployment manifest
back to feature test repository and installing the v cluster CLI logging into the platform.
So, I already have provided the platform URL and the access key. I will quickly show you as
well where it is. And then creating a v cluster. So, v cluster create command is platform
create the name giving the project and a template. I will show you the template as well.
We are adding a preview link so that a person can immediately go and click on the link.
Now, we are deploying that application. So, what happens is the virtual cluster is created
or deployment manifest are updated with the latest image that is there with the tag.
Now, that particular deployment is deployed to the virtual cluster. The ingress and then there is
simple test for running that and that it is calling trying to call the PR. The latest PR
number dot v cluster dot take basically trying to call the ingress that it is working properly
or not. But I will show you that as well how it is done. So, it is already completed.
So, if we see the actions. So, you can see all the stuff has been done creation of v cluster.
So, this is the v cluster. So, it is PR 13 that was created. That is how you can check.
You can also check in the v cluster list using the CLI. So, you can see that there is a
given it is cluster created. Give CTA will get for ciphon n. And that is how it looks like
on the host cluster. So, whenever there is a virtual cluster created as an admin of the host
cluster, I can see that there is a code in a spot. There is a hello world application.
That is there and there is a state full set. So, when I give the cube conflict file for this particular
virtual cluster to you and when you do cube CTA will get pods, you only see two pods which is the
code in is one and the hello world one. So, you are isolated from the host cluster and you cannot see anything
and you get a cube conflict file. But you feel that it is just a plain Kubernetes cluster.
And I am the owner of that. Just like you order a virtual machine from any of easy to
or whatever that is. And you think that you are owner, but in the end it is virtualized and it is
running on a bare metal. But you do not get access to the bare metal. You get access to the
virtualized machine. So, now what we can do is we can actually check if the link is working.
Hello, fourth time. So, the new application is built, deployed, the manifest that deployed
as a developer. I do not have to care about Kubernetes manifest as a developer. I do not have to care
about creating Kubernetes clusters. I just push my code. I am testing the code that I have written
whether it is working fine or not. And then that is particular that is it. Now, it is the
app as well. So, either you can merge it or you can simply remove the label. And that
is it. Once you remove the label, it will again run a GitHub actions which will just
to be cluster delete. So, it automatically then deletes this particular virtual cluster.
So, in that there was one thing which was the template. So, in v cluster you can just define
the templates like how you want your virtual clusters to be. In this particular, we have given
the template like I want to sync the ingress classes because again v cluster is about reusing
the host cluster resources because I do not want to install certain manager ingress controllers
to every virtual clusters. It is already there on my host cluster. I want to reuse those resources.
So, I am reusing the ingress classes. And yeah, this is something which is external like platform
putting it to auto sleep after one hour. I mean even if it is not there, everything good
work as it is. So, hopefully the cluster would have gone. It is full not four because
cluster is gone. And yeah, cluster is already gone. You cannot see over here. So, that is pretty
much it from how it would look like in a workflow. So, in a just to recap a developer raised
the full request added a label. Now, there was a kit of action that got triggered which
had the configuration for virtual cluster Docker hub as part of the secrets. And it did the
build. It also post with the latest image built by in a deployment file and then post back
to the repository and used that to deploy it to a virtual cluster after creating a virtual
cluster. And then created the ingress as well with the latest full request number. And then
the developer tested it. Remove the label and they can do manual or automatic or more
number of tests. And then the user can remove the label and it gets deleted. So, some of the
other use cases that it can be used. This was one. You can have everything in a get-op
way. So, you can have then R goes CD plugged in, watching a particular repository, a particular
folder, wherever there is a change like if I do a merge request. So, if there is a merge inside
the main branch, the R goes CD will already be watching that this particular thing is merged.
And then there will be auto deployment of the application which is there in the production
as well. So, you can do the whole CICD kind of thing. You can also do a CICD in which
you can use flux for creating more or connecting the virtual clusters back to flux. So, that
you have their own independent CICD cycles. And then specifically the one that Retic was
mentioning the CRD, if you are developing those CRDs. So, you cannot deploy different
versions of the same controller in one Kubernetes cluster because it is a cluster wide resource.
But, you can do that in virtual V cluster. So, the virtual clusters will be like let
say I have a host cluster and I want R goes CD version X and R goes CD version Y by a second
developer. So, they both can have that inside their own virtual clusters. So, that is possible
and that is something which helps. And then the resource sharing, you can have your host cluster,
you can have Prometheus, Argo or Sirt Manager, CubeBird all those on your host cluster and
you can reuse all the custom resources inside the virtual cluster. So, that you do not have
to reinstall. Also, just add one last thing is policies are respected cluster wide. So, you
can have Falco that gets the Falco is a runtime thread detection open source project. It is
a CNC app graduated project. So, you can have that and you can have the demon sets. So,
everything will be respected. So, you will be able to do the runtime detection from the virtual
cluster. If you have Kyverno, another CNC app project, you apply the cluster wide policies.
So, that will also be respected by the virtual clusters that gets created. So, yeah, that is pretty
much it. I do not know if you have anything. Yeah, if you want to conclude. So, improved
developer experience and reduce the build time, accelerated test cycles, all the stuff that
he mentioned, creation of cluster because it is in the end like I showed you over here.
It is the state full set that gets created for every virtual cluster. So, it is just a container
that is spinning up. So, it takes obviously less time to run that particular thing. And then
save money on the control plane cost. Because you have multiple human resources, multiple
control plane cost. And if you have multiple human resources, you will be having multiple
multiple engineers controllers, you will be having multiple certain managers, you will be having
multiple agocedes. So, those act to the resources as well, resource consumption as well.
Yeah, that is pretty much it that we have to show. I hope that is helpful.
So, this is on GitHub, we cluster, that is the project that you can try out for creating
virtual clusters. Any questions? I do not know how much time we have?
Four minutes, okay? Yes, please.
Yeah, so good question. The performance bottleneck won't be there. The reason for that is every
word.
So, the question is that is there performance bottleneck when you are using virtual clusters
versus when you have multiple Kubernetes clusters. So, the answer for that is that no, there is
no performance bottleneck. The reason for that is every virtual cluster comes with its own
API server. So, again, when I do, when this was run, basically the cube CTL apply or if I
do, if I connect to the virtual cluster and I do cube CTL operations, the request goes to the
API server of the virtual cluster and not the API server of the host cluster. So, it is the
API server of the host cluster is not loaded. So, the load is going to the API server of the
virtual cluster. That is why it is just a regular part that you are interacting with. So, it is
no performance bottleneck in that way.
Think between the clusters. Yeah, ideally there are some mechanism where you can think, but ideally
we want these clusters to be separate because we want separate teams to have separate virtual clusters.
So, everything will be having separate, but there are some mechanisms and maybe some use
cases where you would want to do the communication, then you have to enable certain flags and
things like that. Yes.
So, every virtual cluster has a single spec. So, you have to write a V cluster dot
YAML file. So, let us say there is a team A and team B. So, team A wants that their
ingress resources and cert manager resources have to be synced. So, they will turn on
the name of the virtual cluster. So, you have to write a V cluster dot YAML file. So, let
us say there is a team A and team B. So, team A wants that their ingress resources and
cert manager resources have to be synced. So, they will turn on those integrations and
team B doesn't want those integration and they want their own to be deployed. So, they
will turn off those integrations or basically they won't have added that to the
V cluster dot YAML file. So, one will be reusing the resources from the host cluster
and another won't be reusing any of the resources from the host cluster. And then the regular
practices of security has like from the general Kubernetes perspective have to be implemented
for everything. I do not know who raised the hand first but anyone can go.
And you configure the single to apply a node selector based on the instance. Yes. So, you
can have you can do that in the V cluster dot YAML file itself. So, you can specify the
node selector's like whether this particular virtual cluster has to be spend up on a GPU
node or a high CPU node and things like that.
Yeah. Yeah. Please go ahead.
So, all the control planes are different. So, they have the embedded ATCD and then
if you go for the like as a pro feature and stuff. I don't want to talk about that but
that has the external ATCD and stuff. You can do that but that is part of the pro
pro thing. Yes.
Would it be able to limit the resource in general that the circle D cluster can act? Yes. Yes.
You can specify that as well in the V cluster dot YAML file. So, single, single, single,
spec. You specify everything over there. Yes.
Yes. So, if a particular version, if a particular team wants 1 dot 3 0 and next team
b wants 1 dot 3 1 they can have. If 1 team wants to upgrade another refuses to upgrade
they can.
So, the host can be 1 dot 2 9 as well.
Yeah.
I think 1 last question?
Yeah. I don't know. Time. Okay. Last one.
Yes.
So, I saw that in comparison with the namespace and about the over here.
If you expand a bit without their similar, like talking like that in your cluster,
then it's a bit like that.
Where was it?
This one.
Yeah.
So, what was the question?
Sorry.
About the over there.
So, how can the namespace of the over hit end of the
class?
Yeah. So, basically it's.
The cluster should be. The cluster is low.
So, it's very low because it's just gives it a clear name space and the resources
it's not there.
But, the overhead for creation of virtual cluster, the time it takes for the state
full set, the persistent volume to be generated, attached to the port,
generating it. So, that's the kind of overhead in terms of time it adds.
Basically, it's spinning up, but the creation of the namespace is instant.
Yeah. I mean, because it's about the theme there.
Yes.
So, it's very low.
Very high. That's how it is. Yes.
So, we have some stickers over here as well.
If you want to take.
Yeah. We cluster stickers are here.
Thank you so much.
Thank you for attending the talk.
Thank you.