All right, thank you, appreciate everybody taking the time to come to this session about
freeing your Kubernetes network. My name is Doug Smith and I am joined with truly the master
mind behind this work, Miguel, and I'm lucky enough to get to work with him. What we're
going to talk about today is a feature that we often refer to as user-defined networking
or UDN, which is kind of a way of changing the paradigm about how Kubernetes networking works
at its core, and especially related to virtual machine use cases. So what I'm going to do is kind
of introduce some of the motivations, some of the problems, and then Miguel's going to get into the
nitty gritty about the details of how to use it, and if we've got time run through a demo as well.
There is a large stack of open source technologies that are involved in this particular project
where already building on top of Kubernetes, and we're using virtualization on top of Kubernetes
as well, and really these two projects right here at the front of in Kubernetes, which sometimes
we call of NK, and Kubernetes are really at the top of this, but all of these technologies are
open source. You can pull the hood open and get into all of them. Whoops. So let's talk a little bit
about kind of the motivation here, and I kind of see it as sort of two sides to the same coin.
First, you've got really your traditional virtualization user, and in a traditional virtualization
world, like say if you're using open stack and you're using neutron, you're going to make an
assumption that you're going to get a layer to isolation across your virtual machines in order to
segregate the traffic properly and to create a tendency scenario. And if you're a Kubernetes
savvy user, you're going to probably have an assumption that you're going to have a bit more of a
like managed experience, a bit more of an opinionated way to do something. So if you are putting together
something like this today, you're probably, well, I should say today, prior to user-defined
networking, for example, you're going to have to kind of like baby step all of the things that
may be involved. So maybe you're going to have to figure out your own IP address management solution,
and you're going to need DHCP in order to handle that. So you'll have to set up these things yourself.
And this is kind of underpinned by something that is truly different between the virtual machine
world, and the like purely containerized pod world, which is in a virtual machine scenario,
you are often going to assume that the life cycle is different. If you have a virtual machine that
goes down, it's not really going to go down. It's going to be migrating, and what are you going
to need in that scenario? Probably persistent IP addressing. So once you get that migration,
your traffic is still going to go to just the right thing. It's not like a pod life cycle where
you kind of build your applications assuming that they can arbitrarily die, and they'll come back
up, they'll have arbitrary IP addressing. So that's quite a bit different. So really the motivation
is to try to merge these worlds and make something that fits this a little bit better because
the bottom line is very opinionated. It's a design for web scale, and it's going to have one
single network where everything is assumed to have connectivity between all of the elements in
that cluster. So if you wanted to make something that had this type of functionality, you'd probably
have to create that kind of micro segmentation, we'd say, using a network policy. And the thing
with that is that it's expensive on two fronts. First of which is it's going to be computationally
expensive because you're going to have to make a lot of different rules in order to get that
kind of segmentation. And then it's also going to be procedurally expensive for the humans that are
involved because they're going to have to create it. And some of the things may be like
really challenging to do that way. So these are the kind of things that we're like I said trying to
like weave these two worlds. So let's look a little bit at the use cases for this. So one thing
that user defined networking helps to provide is a native name space isolation. So instead of having
that single Kubernetes network where everything is connected, what you'll be able to do right
out of the box with user find networking is to create network isolation per name space. So you could
have just the pods in the orange name space can only speak to the pods in the orange name space.
Same with the red, green, and blue, there's isolation between those. In Kubernetes typically,
you would have connectivity between all of those namespaces. There's also a contract for a cluster
wide network isolation, CUDN cluster user defined networking, which allows you to logically
group together those namespaces and get that isolation per group as well. And looking at the goals,
we want to have that workload and tenant isolation just like I was a diagramming there before.
We want to have it so that you can manage that isolation between those namespaces or groups
of namespaces. Another thing that we really wanted to do is make sure that you are able to
handle overlapping pod IP addresses. So let's say you go and set up a situation that has
of whatever a typical like slash 24 subnet and it's 192 0.2.0 slash 24 and then you're going to
make yet another group of these pods with isolation. Instead of having to go and tweak it and change
that to have a bunch of namespaces so that there can take you as subnet, you can just kind of copy
and paste and just use that again and again and again, which will make it much easier. Additionally,
we want to have that isolation, but we also want to have the kind of Kubernetes APIs that you
would expect to have, such as services, network policies, admin network policies, we still want
to be able to use that kind of stuff as well. So that is still available and accounted for.
We want to have that stable IPAM configuration like I was saying. We want to have those persistent IPs
when we get that, when we have those migration scenarios, we want to have that persistent IP,
when that VM is migrated and comes up on another machine, we want to make sure that that works.
Another thing that we want to make sure that works for you well is to have your public cloud support
so that you can make sure that your traffic is going to get to the place where it needs to go on a
public cloud. So in a public cloud scenario, your provider likely is going to have some constraints
around how network traffic moves across their network, which may mean that you're going to need to
let that public cloud provider know about the IP addresses that you're going to use and you don't
want to have to make a phone call, actually no one has done that for the last one years.
I'm submit a ticket, maybe no one has done that and a while or provision it via their web interface
to like keep adding IP addresses kind of a thing. We want to make sure that that is handled
by these technologies themselves. And lastly, but not leastly is so while we still have that
network isolation, if you are doing functionality in your virtual machines that accesses the Kubernetes
API, access a controller, for example, or you need something out of QBDNS, we don't want to
deny you from using those services either. So we make sure that there is connectivity to those
kind of like core Kubernetes services. So let's dig into the API and I'm going to let Miguel take
the stage here.
Okay, this is not good. I'll let you do this. Okay, now no, it's okay, better. It's okay, it's okay.
I have to do it. Okay, so the first thing we're going to be seeing here is how the API looks like
like how do you actually use and configure the feature. So let's start with an example because
it makes it easier to understand. So if you want to configure namespace isolation for namespace,
let's say it's green. You would provision this the CRD called user-defined network. Again,
you have to decide to do to name the namespace in which you're going to be isolating your network.
Because again, the idea is for you to kind of override the default cluster network. You want
something that is not the default cluster network. Anyone to do that per namespace, you do not need to
do that for the entire cluster. Now, how does the configuration look like? Well, it's quite simple.
You have to decide the topology for virtualization. You need to specify layer two,
does the only topology that will work for virtualization. There are more topologies, but for
vert, you have to stick to layer two and then you configure your layer two. So you have to
to specify that the role is primary. So this is a primary user-defined network. So it knows
how to override the default cluster network. You specify the subnet. It can be
like dual stack. So you can put here a single stack IPv4, single stack IPv6 or dual stack.
You cannot do more than that. And you have to specify that the IPm life cycle is persistent.
So that the IP address allocation knows to stick to the virtual machine life cycle instead of
the pod life cycle. For the other use case in which we want to interconnect two different namespaces.
So in this scenario, you have the red namespace and the blue namespace. And there are interconnected
by the happy network. What we would do is to use this other CRD called cluster user-defined
network. As Doug said explained before, notice in the metadata that this is no longer namespaced.
So it's a cluster wide resource. And if you focus in the bottom and the network stands up,
you'll see that it's exactly what we've seen before. You specify the topology and then you give
configuration of what the topology looks like. So role primary, IPm life cycle persistent. So that the
IP address says will be persistent and you define a subnet. The big difference comes from expressing
how or which namespace if you want to interconnect. For that, you use the namespace selector.
So pretty standard Kubernetes part of the model. You define what namespaces you want to do.
You just put here, like for instance, values, red namespace and blue namespace. And you'd be
interconnecting those two. And everything else would not be able to communicate with them.
Okay, so we've seen how the API looks like and how you configure the feature and enable it on
your namespaces. The ones you're interested on. Now, how does this look like from the
oven perspective and how does this actually work? So for every UDN that you create,
oven Kubernetes will create an oven layer to switch. So and that's why these things are isolated.
So your red namespace is using one layer to switch that is totally disconnected from the rest of
the network. Your yellow namespaces using an entirely different layer to switch. And all these entities
that you see here are replicated for each UDN. The workloads that are in the bottom,
so you see like pod 1, pod 2, which happens to be a VM. And pod 3 are just connected to this layer
to switch. And then we have a gateway router in each of the nodes for it to be able to
egress to the internet to implement more features than that. This gateway router is also connected
to an external switch in each individual node. And this external switch has a particular
special type of port that connects it to an OBS bridge. And then we have a physical interface on the
nodes for the traffic to leave the cluster. Okay, so question now becomes like how do you
re-manage IPAM for the VMs? Because we need to do that in a different way than we do the pods.
So oven Kubernetes what it does is every time you see the pod being created, it will
introspect the pod and figure out, okay, this pod is a pod, I'll do nothing. If it figures out
that or if it understands that a pod is for virtual machine, it will provision a DHCP, an object
in oven called DHCP option, and it will put the following information. So IP address, gateway,
DNS, host name, and MTE. This only happens for the VMs.
It's quite easy to understand that the gateway, DNS, configuration, and DMTE are common for
all the objects in the network, but the IP address and host name are per VM. Okay, we need to move
along a little bit faster. So how do we enable this for cloud platforms? Well, the solution is
why, obviously, you have to SNAT from the pod IP address to the node IP address. Otherwise,
your cloud provider will just look at your traffic and totally stop on that.
And if you remember something Doug told before in the objectives, we want to have overlapping
subnets. This kind of, it's kind of the plot thickens a little bit because in order to implement
that scenario, you need to add an additional layer of NATting. So let's consider this scenario.
You have two different UDNs, primary UDNs actually. So we have the blue and the left and the red
and the right. For each of these primary UDNs, we have to compute, um, Kubernetes decides
something which we call like the a masquerade IP. So if you look like the blue network has the
dot 12 IP address in, in, in the last thought it. And the red has a dot 13. So there's like a specific
per network, masquerade IP. And on the gateway router, we do a SNAT from the pod IP to this
per network, masquerade IP. And when the traffic hits the obvious bridge on the node,
we do another layer of NATting. And we, uh, NAT from this per network, masquerade IP to the node IP,
the traffic leaves, uh, to the, well, outside the cluster. Once it goes back, we do the other
thing around. We do it the other way around. We units NATs back to the per network, masquerade IP.
And on the gateway router, we units NATs it to the actual workload IP address. So yeah, it comes with
a cost. But it's a very cool feature. So how do we, uh, actually do stable IP in? It's actually
quite simple. The idea is mainly like if your virtual machine migrates to a different node,
we just want the IP address to be the same. And there's another thing we want. They want the gateway
configuration to be consistent. So IP address and gateway configuration have to be consistent.
Because if they're not, you're existing TCP connections on east west traffic are broken.
And we don't want that. And on the north south, uh, kind of traffic. If your gateway configuration is
not consistent, they also break. And we also don't want that. Like probably your workloads will
not tolerate it. So the trick is quite simple.