So the next session is about to start. Please take a seat and quiet down. Thank you. So we will
hear about lessons after three years of clubness. Kubernetes. So hello everyone. I see this
pretty packed room. So thanks a lot for attending and let's talk about Kubernetes today.
So my name is Nadia. I like infrastructure. Here you have my website where you can see the
information about me and now you're jumping to the fun stuff. So what are we going to talk
today? So I'm going to give a brief introduction of self-managed Kubernetes and then I'm going
to talk about some of the tools that I use that make life easy for me. I hope that this gives you
some pointers in case you want to attempt this. So let's start with a show of hands. We're going to
do once. So please read with me. Please raise your hand and keep it up. If you interact with Kubernetes
regularly, use your own definition regularly. Okay. Good solid 90%. Please keep it up. So lower down or rather
keep it up still. If you also manage your own control plane. Okay, still still pretty good. Good.
So I'm going to still go through the first section but I'll try to make it a speedy. You can lower
all your hands now. Don't get tired. Okay. So let's start with a very quick introduction to self-managed
Kubernetes. And to do that, let's also talk about what I mean when I say I run Kubernetes outside
of the cloud. So outside of the cloud, by my own definition, just may vary is when you run Kubernetes
in your own hardware. So let's say you need to care about the hard drives that are in there.
And you need to care about the memory that you have in there. When you manage your own control plane,
that comes with that was pretty much obvious one. And you do not really or try to not really
on external services provided by others as much as possible. For example, DNS, that's a hint
about the demo we're going to make. So what's a Kubernetes? Most of you will know this already
because you raise your hand, right? So let's go with about the real quick. So a Kubernetes and
by that I mean a Kubernetes knows is just a cumulative process that is properly configured and running.
And that's the reason to it. The cumulative will do stuff. We'll connect to a community
API. We'll find some pods that are tagged with it's no name and will run them.
Now, interesting thing is that the cumulative binary runs on a typical Linux system. It has some
configuration and you know, flags files and whatnot. It depends on other binaries. For example,
you're going to your container runtime or you see an I. And it's sure typically be a start of that
boat, right? So this looks very familiar. This sounds very familiar. This looks like a just Linux
demo, right? A Linux service. So in the end, what's a Kubernetes knows? But just a Linux box with
something running on it. So let's talk about control planes now. So what's a control plane?
So very roughly speaking most of you will know this already. It's a series of services that make
Kubernetes work. And we have a bunch of them, but roughly we have an API, the rest is API.
We have a database, typically it is a D. And we also have a bunch of clients that do stuff with the
API, right? And these services, these API, these database, these other clients, sometimes run inside of
Kubernetes itself. And that's pretty fun. And those, this service obviously very critical, right?
So we dedicate a whole machine most of the time, sometimes many to run these services. So when we talk
about a control plane node is just a node that happens to run these services on it. So there is
not a lot of mysticism on it. And I promise this is the last system in from now on, or everything on
the left is going to be jammed. You will feel like, at half. So how does a control plane look
in cluster? You can see the pods in there. I'm not going to go over them, you can see the
CNA, you can see the data data. So you may be wondering, so if the control plane lives inside Kubernetes,
most of the time, and is the control plane the one that typically creates the pods because the
cube will need to connect to the control plane to the API. Who starts the control plane, right?
That might be a very interesting question. So the answer is static manifests. So the cube
led has a very specific folder there in this, where it has some jammed files, and it will basically
blindly start these pods when the cube lets us start. We've forgotten anything, we've forgotten
what actually while trying to connect to the API, it will just read the files and create them.
There are some answers, there are some limitations, and there are quite a lot of limitations,
so what this static manifests can do, but it is enough to bring the database, the API and whatnot.
So what is Kubernetes really? So it's a set of Linux boxes that you will need to manage, right?
So you will need to typically run Linux and then, so if you want to do this, you will probably
need to pick Linux distribution that you are comfortable running, that you are comfortable
troubleshooting, you will encounter fun problems, you will need to install packages, you will need to
upgrade those packages. So my advice on this is just pick someone, pick other distribution that you
are comfortable with. Cool. Enough of the introduction. Let's talk about some tools about some
things I do. The first one, I'm going to talk is QVDM, QVDM's kind of the reference
way to set up a Kubernetes cluster, and don't pay a lot of attention to the jameleon right, I don't
want you to read that, just I will like for you to see that all the jameleon right is everything that
I use to bring up my cluster, that's it. So it may not be very readable, but it fits on
and I think that's pretty cool. Why do we need these tools? Because we talk about the QVDM
and bringing all these files, all these static manifest and whatnot, right? So that is kind of an
understanding, right? Those static files, those static manifest needs to be populated, need to have
arguments, need to be consistent with each other, right? So QVDM will take care of all that. Also
we know that stuff, like creating Arbach objects, have the clusters brought up, you know, creating
TLS certificates for the API for the, for it is the deploy avons, funny, that QVDM coordinates,
so DNS, and this is an avon, it's pretty funny, and it also renews certificates. So I think QVDM
is pretty cool. Another thing to use that I pretty like, that I like my match is kind, kind
of a project from the people who make KTS, and the reason, well kind of says for a couple of reasons,
but one of those is that it is the is great software, built with high-level ability in mind,
and something I have found is that it is also very hungry for your SSD. It will choose your
SSD really fast because it goes to this a lot. So as an anecdotal evidence, it is the two
10% of my reporter says the left is Pani Nagier, which is not particularly great. So kind allows you
to back for the allows you to replace the SSD with something else. For example, it is QLite,
which is pretty cool if you are running a pet cluster or it is more cluster, or it progresses QL,
which is what I use just because I cannot focus on one detail is. And funny enough, kind,
kind, so we deploy pretty easily with QDM. So don't pay a lot of attention to the jamele, I think
that there is not a lot of, you don't need to mind the details, just know that those two snippets,
the one in the bottom is obviously cropped, but the one in the top is legit, that is all
unit to use kind with QLDM. So I think, again, this is, this QLDM is a pretty, it is a pretty nice
tool, and very flexible. So more stuff, Philium, you may know what you may know the acronym C&I,
containment of working interface, and this is, CLDM is C&I plugin, and it is interesting that
just like the NS, it is called plugin, but nothing will work without it, so it is actually
quite important that you pick one. And I really like Selium, because I think it is very well
that meant it replaces QLite proxy, so that is one thing less to care about, and it is relatively
easy, or as I put it less hard to debug, than others, and I suddenly see my experience, that's
all my views, just may vary. And something that I find particularly relevant for this is that
it has a lot of knobs to tweak, that are useful for metal. For example, you can restrict it to
certain interface names, you can choose with kind of encapsulation, it is going to use for
no-to-no traffic, and so on. You can also find custom routes, so get rid of that encapsulation
all together, if you everything shares the same else you network, that is pretty pretty neat.
Another neat feature of Filium, actually, is the EGRS gateway, so if you are running a
Kubernetes cluster and you are kind of in a multi-tenant environment, and you want to have
and you have more than one AP address, and you may want to map tenants to IP addresses, or otherwise
dedicate these IPs who want to be used by this tenant, this is for EGRS IP addresses, and for most
of you, then you want to talk about this, but for some workloads you may need to, for example,
if you are using HR hosting an email server, email is pretty picky about IP reputation,
so you really don't want, if you run in a multi-tenant environment, you really want that contained,
so if one of your tenants behaves badly, it doesn't pack the others. It can also be used this
feature for road EGRS traffic through specific nodes, which I think is pretty good.
So we have talked about EGRS, let's talk about EGRS traffic for a while,
so you will probably need a load balancer, it is not strictly like necessary, but you will
really want to, one one special is you want to go HA, and why is that? Well, obviously,
all the IPs and service IPs are not reachable from the outside world, that will be annoying,
right, and a security nightmare as well. So your nodes may or may not have public IPs,
but it is not particularly straightforward to map the IPs of your nodes to those services.
There are a couple of things you could do, for example, you can use, you can create a service,
and use a node port, that will be not very migration friendly, so let's say, if your workload,
migration one node to another, you may have problems with that, and you will also have problems
with multiple applications that want to use the same port, right, even if you have multiple nodes,
with a node port they will fight each other. Same thing goes for external IP, external IPs,
and I shortcut, it allows you, it tells the CNI everything that comes to this IP address,
that this outside of Kubernetes, and to this port, please route it to the service,
this works, I think, a bit better, but you need to pick an existing IP address there, right,
one that is already reachable, and which node are you going to pick, whichever you pick,
is going to become your single point of failure, if that node goes down, your service goes down,
it doesn't matter how many replicas you have, and that is not particularly late, so we can
solve that, we can solve that with all those balancer, and the one that I particularly like is
metal LB, metal LB is a load balancer for metal, it allows you defining a set of virtual IPs,
your network provider will need to allocate this to you, or grant you permission to use them,
but the neat thing is that you don't need to bind them to any particular machine if you do not like,
you just create a CRD, that's on the right, these are the two CRDs you need for to make one thing
to work for one IP to work, and what metal LB will do is that nodes will take turns,
an ounce in the CPU, there is an internal literal action going on with gossip, and the metal LB
will ensure that one node is both casting ARP packets for this virtual IP, so no node has this IP address
in, if you go into the nodes and type IPA, you want to see it, but at a given point in time,
one node will be an ounce in it, and that means that for all intents and purposes the address is
reachable, if that node fails, then metal LB will figure that out, pick a new one to an ounce
IP and everything will work with minimal downtime, your CNI will typically take care of load balancer
in that, so a single node will be the entry point, but your CNI will typically if you don't tell
it otherwise, you will typically distribute that inside the cluster, so let's make a demo let's
save that works, so I have a bunch of terminals here which are like in a very big font,
this here that you can see on the top right is I'm going to actually make it smaller just
so it fits, is it still readable, okay, not a lot, this is a Kubernetes cluster that they
have created for this demo, it has a public IP, it is running on Hatsuna Cloud, so I'm going to show
you the things I am going to apply, right, which are here, I'm going to apply a name space,
pretty trivial, I'm going to apply those here this that we just saw, this are the same one that
we're on this line, this is an engine accident deployment, like there is not a lot of magic
to it, just deployment container and the next nothing special, and this is a service for that
with labels matching of deployment, this service is of type, what's to that again, this service
is of type load balancer, and you can see that it is annotated with this here, this is telling
metal will be, please know that this service will want to use that IP, okay, so let's apply that
and see what happens, and before that, so we can see the magic, I'm going to start what's
cool command, this cool command, I'm going to cancel, so you can see it, it's going to try to connect
to this IP address, and right now this IP address does not exist, nobody is answering, this is
arriving into actually Hatsuna data center, but nobody is answering, so let's apply this,
and let's see what happens, so I have created an in-space, I have got this, I'm sure this
and now the IP address works, it's like magic, okay, so one note about load balancers and this
need thing called external traffic policy, right, you most likely don't need to care about this,
but it has beaten me up in the as a couple times, so I think it's good to mention, so load balancers
are great for exposing services, but as you may expect, they are balancing loads, and they are
doing this thing, I just told you, the request, the ingress traffic will are right to one note,
and then it will be distributed by you've seen I applied into other notes, right,
as you think just for a moment about this, for this to happen, there must be some kind of IP
routing in the right, because the IP packet that arrive has a destination MP, and then we're
going to route it to another note, that is there must be some kind of trickery going on, and the
result of this trickery is that the service, the destination, the path that is handling the
ultimate in the request, will not see the source address for this packet, right, most of the times
you do not really care about this, like application doesn't really care about where it comes from,
it will see of course a source address, but it will be a one internal to a cluster,
there are some applications that do care about this, some applications that may want to do
some allow list in for this IPs, or that my have some nasty encryption scheme using the source
IP address, this happens, and this has happened to me, for example, with transmission, which is
a bit on server, so this kind of application to have in-house roles, your own encryption stuff,
like not TLS, will care about this, so you can use this property for service, external traffic
policy, by default is cluster, it's just a little local, this will not happen, and nothing will
be routed inside a cluster, metallurgy will be noticeable this, and will only force this IP address
to be announced within one node that can actually also handle this load, so it is not free,
there are some nuances in that, but it is an interesting trick, let's talk about everybody's favorite
three-letter acronym, DNS, so as the number of applications that you are hosting keeps
growing, your number of domains may also keep growing as well, especially if you buy the
domain as the motivation for your project, and then you need to give up on your hopes and dreams
and let the domain grow in there, so one typical thing that you can use while coming names,
so you just say, okay, this is the DNS, this is the address for every subdomain of
the other more, you could do that, why not, but that may fail short, and may also not be
particularly great for multi-tenancy, and besides, you know, when Kubernetes, and for example,
if you have an address, Kubernetes knows what is the host name of that address, right, you write it
in the YAML, so it will be kind of neat if we could use that information just to make DNS records
happen, right, and that is what external DNS does for us, this is another great tool that I like
very much, and it will do exactly this, it will walk your ingresses, it will log in walk,
your load balancer services that are raising the cluster with a particular annotation, also custom
resources, in case you want to, I don't know, else some other names, and it will create the
inner record for that, isn't that great? Now there is a, there is a gotcha, right, external DNS
is not a DNS server, so it will create this DNS records, but it will not serve them, it will,
the actual, the way it works, is that it will try to register them somewhere else, and that's
a bit of inconvenient, but worrying out, I have the solution for you, this is a very nice tool,
a very nice actual, it's a handsert called status DNS, hosting GitHub in the TX QLAN
organization, that's there is a joke in there that I'm not going to explain, and status DNS is a
handsert that combines external DNS with power DNS, power DNS is a fully fled DNS server, right,
and the, the thing, the magic thing, that this does tell the DNS does, is it makes power DNS
stateless, and that is great, because we love status stuff, and my phone is kind of buggering
if you, so sorry, and a second, there you go, and we love status stuff, right, so what will this
thing do? It will create a power DNS deployment, it will spin up, external DNS next to it,
and it will configure both of them so they turn nicely to each other, right, and again on the
right, there is a lot of jammel, there is no need to pay a lot of attention to that, just keep in mind
that all you need to make this thing work, and you will see it in just one second. This is
another nice feature that this card is that allows you to define your DNS zones as close,
that is something that external DNS does not manage, external DNS won't do your songs for you,
but this thing will, if you can see in the bottom, there is a bunch of text in there that you
may sound familiar, this is the bind text format for DNS zones, can understand it, and this
card has a nice init script that will pass this and see the power DNS database with this,
so you don't need to care about power DNS at all, this is kind of great, it also highly available
as the risk, so let's do a demo about this, the announcement over the internet, what can we
work? Cool, so we have the same thing that we have before, this this card is still running,
just to refresh your memory, this is calling this IP address, and the default NGNX
welcome page is, and it's announced, got it, and now something that we may want to do is to have
domain name for this right, so I have this another cool command here that is basically doing the same,
but this is doing this against a domain name, and now right now you can see it doesn't work, so let's see
one thing, I'm going to apply this demo DNS thing, just in a moment, this one, but I want to show
you what is the difference between this and what we applied before, because this demo will not
talk of each other, as you can see some deep wire plate, the thing you can do need to pay attention to
is this thing, this, apply command will add an annotation to this host name, and we'll tell
hey, external DNS, this service is served on this domain, out of cloud that online, so let's try to
apply it, okay, so we got it configured, and there might be some time for DNS propagation to do
its magic, I'm going to give this 30 seconds, and if not I'm going to cheat, let's say if we can
pay some attention, okay, so we have this is not answering yet, but now it is, and there you go,
our DNS name is awesome, cool, so just to close this with some other things that I think might be
interesting, but not really related to the topic, there are some other things I do like,
and I think it's super useful, but in Kubernetes in general, I thought about, I thought that I might
as well take this opportunity and voice it, so the first one is GitHub, GitHub is great,
GitHub's, and you can call me this, is what makes Kubernetes worth it, do yourself a favor and
of load your brain to get, this will draw, draw drastically improve the things, and the
drastically increase your possibilities to actually host this time, you want me to remember
about every single thing that you are hosting, if you get, can do it for you, both Argosidia and
Flexidia, I think are the most true, the two most popular GitHub Stools, I think they are both great,
renovate what also some call, also sometimes called renovate, is a very nice bot that you can set up,
is will run periodically, watch your GitHub repo, it understands Kubernetes manifests and also
hunter values, and it will look for images and check if there are new versions of those images,
and if they are, it will create a PR for you, so that is this in my view, this greatly amplifies
the power of GitHub, right, now you have something that is able to work this call and actually
improve it. Secure management, I have tried a couple things, I have tried sobs, I have tried
still secreted by the NAMI, in the end what I do, your malicious may vary, is I just have a separate
repository with diamonds, like it works, and yes, you can just me, feel free, but it really is in my view,
the less, the less, for databases that you may want to run Kubernetes, I find cloud native
pgqi nice, it allows you to define a custom resource and it will provision a database for you,
and more importantly, you know, provision a database is easy, anybody can do it, more importantly,
it will perform switchover procedures when that database migrates when one goes down, which is pretty
bad. About tooling for deploying stuff, I have tried a couple things, I have tried hand charts,
I have tried customized, in the end what I do is I just keep a lot of yummy files, the reason
for this is that, you know, every hand chart has a different API, like a different way to understand
the values file, my brain just kind of kind of keep up with all those API, right, I will always
hand up hand to all the templates and that's manual, manifest work. For storage, what I use are
just some statically provision, that means I have a basis script that I'll push jammal, that creates
previously some previous, and they are buying to an alt as one will respect, but they just work.
And finally, this is my last slide, some things I would like to try, I would like to try to
start with a few updates that are, you know, you can roll into your notes, how to mount
automatically, it's pretty, pretty magic, there are tablets that does this, like towels and
flacker, but I haven't got into that yet, for storage, I want to experiment with the
extra buckets of species, and maybe at some point I will scale my clusters to something
multi-region. So that's it, thanks for your attention, any questions?
Yeah, that's one there.
With regards to secrets, and just as a data, you said that it's just basically jammals, is it
you use e-amal or just plain text jammal, for example? Do I use what or text jammal,
the encrypted jammal file list? No, no, it's just text jammal, like a regular manifest,
it's just kept on a different Git repository, it's also synced with GitHub's an
algo, the advantage is that you can make your other repository public, and this one obviously
not, and you can also have different ways of controlling access, but it's plain jammal.
Okay, thank you.
Hi, about all the DNS dense that we did, so the idea is that you don't want to rely on the external
DNS provided by the operating system, but you want each pod to see their own work,
because we might be in a multi-tenant environment, so you don't want all pods belong
into another customer to see a collective list of records that might be accessible, that's the
idea. So let's see if I got your question, your question was about
about having an internal DNS to the pods? Aha, no, this is not an internal DNS, this is a
world-reachable external DNS, this thing I'm doing here, you can do it, you can try to resolve
this this name, it will work, this is all over the internet, so this is about how you make
your applications available to users, which will require a domain name, and you can do this in
a self-contained way, this DNS server is running in the same cluster, serving that engine
expage, there is one more in the back, we have time, was it always in the middle of today?
You should spread as much as possible, and make the room organizers do some exercise.
I have a question about the kind, I really understand why you use it, but
can you please be a bit quieter and the person is speaking a bit louder, thank you?
Yes, I am just asking if maybe kind is creating a single point of failure in your environment,
as it runs on Postgres, and if the Postgres server is going down, you won't be able to access
to the API, and actually there is some possibility to get some postgres frustrating,
but it's not very as strong as it is for example.
Okay, so the question was about, to say we got this correctly, about HAPosabilities, about Postgres,
as the database back in the API server, what was that it?
Yeah, and does it create a single point of failure?
Yes, sir.