Okay, good morning everyone, my name is Miguel Dwarth, I'm here with Federico Pauline
Nelly which is essentially the blood, the sweat and the tears behind the project that will
enable the things we are about to show and we're here to present a talk title, we've
been the fabric which is about EVPN overlays for multi-cluster cube work deployments.
The agenda will be, we will begin with a motivation, then go through what EVPN is which
essentially is like a shortest claim or the enabling technology for this, Federico will
explain to us everything about the implementation of this.
Then we'll show a few demos and finalize with the conclusions in which we'll pretty much
be telling you a summary of everything we just told you.
Now, first thing, why should you care about any of this?
So, the use cases for multi-cluster could be kind of intuitive but let's go through
them nevertheless.
So, let's say that you have a single cluster in light metal cluster located in a building
and for whatever reason everything goes down, you lose power in it after a while everything
goes down like the blast radius is huge, something happens, your entire set of applications
go down.
If you have multiple clusters and your application is spread across them, well, it will
keep working.
So, that's one thing.
We want to have resiliency for your applications.
Things like scaling, hybrid cloud, also require you to have your multiple clusters interconnected.
And the third thing which is very useful is legacy applications.
So, I'm guessing if you were running VMs, you have probably seen a virtual machine whose
identity is pretty much an IP address.
So, that means that throughout the life of VM and probably 20 years, you cannot change
its IP, it needs to connect to the same network, it needs to reach the same neighbor throughout
history.
So, you need it to be connected to the same subnet and you probably have changed the platform
many times.
So, these are the reasons of why should you want to have multiple interconnected clusters
and we will be doing that using a stretched layer 2 network.
So, as I said, goal is to fold.
First one, we want to have a stretched layer 2 network and this will be very useful to
provide us these two features, cross-cluster life migration and resource pooling.
So, in a way, you have multiple clusters but you'll be providing a unified front of resources
to the users.
The second use case is routing between these different networks using a VPN.
For the here you'll have, you'll be able to provide traffic segregation and also direct
routed ingress to the virtual machines.
What does this mean?
It means that, remember, this is to infert, so your VMs actually run in pods and this means
that you no longer need to expose services or use a net.
So, VPN will be our enabling technology of sorts.
It will provide us the ability of having a stretched layer 2 network.
It will also provide us with the other objective we have which is to route between these
different networks and it has some characteristics, first of which and most important,
please, me is that it has reduced a broadcast on-known unicast and multi-cast traffic.
So let's say you have 100 clusters that are interconnected, each of them is running thousands
of pods and hundreds of VMs.
Imagine the amount of neighbor discovery messages that are happening throughout this
scattered clusters or ARPs that is just too much.
We don't want to have that, it needs to be more efficient and if VPN provides all of that.
And remember, we're here to see like seamless workload mobility.
That's what we want to have.
We want to see a VM migrate from one cluster to another without impacting the applications
running in it.
So, VPN first stands for Ethernet VPN and we can see it as a twofold protocol.
So in the control plane, it runs multi-protocol BGP and the data plane can be multiple
implementations like VXLON MPLS SRV6.
In these presentation, the components we're using pretty much has a single implementation
which is VXLON which is why it is involved and let me begin with the control plane.
So as I said before, this is using multi-protocol BGP and essentially it provides two
things on top of BGP which is a type 2 route, we'll see what this is in a minute and
type 5 route.
We'll see what this is in probably two minutes.
So type 2 route whenever a VM connects to the network, this will send, we'll create
a type 2 route which is essentially a means for you to register to the entire network,
scatter throughout these clusters that the way for you to reach this Mac and IP address
is via this next hop.
So you essentially brought casting to the network that for you to reach this VM, you need
to use this particular next hop.
A type 5 route on the other hand is the same concept but not for a Mac and IP, you're
just saying the entire prefix.
So this entire subnet can be reachable via this next hop, you're essentially advertising
how to reach either a Mac and IP pair, a Mac address or an entire prefix.
And I'm going to end this over to Federico and he will explain this in more detail.
Okay, so now let's have a look at how we try to implement all this thing, why we keep
it sane and we try to follow some design principles.
One was try to integrate existing solutions, not to reinvent the wheel to have something
that is composable and plug-able with existing building blocks or to reuse as much building
blocks as we had.
Another thing is we wanted to have something that is kind of primary diagnostic and also
can work on some secondary motors plugins and this is what driven our whole architecture.
So the building blocks are obviously a cubeard, we need that to run our virtual machines
and Kubernetes clusters.
Kubernetes can work with any bridge-based CI, the two most famous R of the CI and bridge
CI, they are used to connect your secondary networks either to an obvious bridge or
to a Linux bridge and there is this other thing in the middle which is open peer-outer
is a new project that we've been working on more or less for the past year and it's about
running our outer on your Kubernetes cluster.
So the idea is of the project and if you're interested into this, we have a full-fledged
talk in the networking dev room later today, the idea is to have a router as a Kubernetes
component that you can deploy on your cluster with hand shards or manifest and basically
it will handle all the complexity of being a router exposing the interfaces to the host
as a real router as web pairs.
It allows you to integrate with the BGP-speaking components such as Calico and MetalB,
but also in the focus of these stockies, you have this kind of dungling wire on the host
that allows you to enter a broader layer to overlay spread across multiple nodes.
This is how more or less the architecture is done.
So OpenP router allows you to expose this leg on the host.
You can attach it to a Linux bridge on the host or a non-VS bridge and you also have facilities
to have this bridge auto created for you and all the rest is handled by multis.
So you just use a secondary network, BGC and I, you connect your VMs, secondary network
to the bridge and the magic happens, hopefully.
This support distributed in Cascatoi, so your local gateway is going to be the same whenever
the virtual machine runs and these enable some interesting features as we are gonna demo later
on.
Inwards is simple, the reality is that in order to enable EVPN on a Linux host, you have to
do a lot of things.
You need to configure Linux bridges, you need to configure Linux VRF, a VXLan interface,
its web pairs and doing it manually or with some automation is a lot.
That's why we try to put together something that handles this complexity for us behind
a cloud native approach.
So you can scale, you have a thousand nodes, you've crafted your nice or not so nice
YAML, you throw it at the cluster and boom, you have EVPN.
So this is what OpenPear out there is, it's just a Kubernetes controller, we don't focus
on a particular CNA plugin so it can work with your primary CNA of choice and it allows
us to scale our deployment in an efficient manner.
So what do we need in order to have EVPN configured on the cluster?
You need virtual tunnel endpoints which is the end of the tunnel or the start of the tunnel
and you need the tunnel.
That's the basic thing that you have to do as a starter.
In order to do that we have this CRD, it's called underlay, basically it's the details
of the session between your top of the rocks which now becomes number because all the
EVPN logic is done inside the node and then you need some extra things like the interface
that you want to use to connect to the top of the rock and a side there because we want
to show it on the next slide, we want the VTAP on each node to be different and so
like we leverage the Kubernetes way to do things and we synchronize and each node gets
a different IP and of course the data plane is VXL and so what really happens is that
the traffic coming from the host gets encapsulated in a UDP message with the local
VTAP as a source IP and the destination VTAP as the destination IP.
How do I know where do I need to send the packet?
This is the type 2 route that Miguel was telling us about.
So the BGP fabric knows how to announce the locality of each marker and each IP are around
the fabric.
Okay, so now how do you create the control plane for this again?
You throw a YAML at the closer and this open BU route you're thinking we just will just
bend over backwards, create the VRFs, the bridges, everything put together for it to just
work which essentially will get you a stretched layer 2 network and overlay across the
entire fabric by fabric I actually mean a bunch of connected routers which could be like
the internet itself and this is what an L2VNI looks like there are three very important
things here first VNI so this is a VXLan concept akin to the VLAN ID for it.
The VRF which is pretty much like the identity of this and we'll see later on how to integrate
this or expose this network to the outside world and finally the host master which is
how do you provide connectivity from the router to the host itself or to the Kubernetes
node it's runs on.
Essentially this is what it looks like so you have your Kubernetes node it has the router running on it
like the blue thing is the network name space of the router and for every VNI that you have created
you'll have a bridge like that one and then you'll be having a dangling VF on the host
and depending on the configuration like if you remember you've seen here that we are using a host
master of type Linux bridge auto create it will create a bridge on the host for you what
do you do with this bridge now this bridge is essentially what your workloads will be using to
attach to the network so you will if you're used to multi you will just provision now a network
attachment definition with the configuration pointed to the bridge with the proper name and you'll
be attaching to that bridge which essentially is this so we have our keyword VM running there attached
to the to that bridge which is using that VF there attaching to open V router and then it will
use the VxLontunnel Federico tool is everything about later a little bit ago.
Now I'm having this part here with to explain how the type two routes work so whenever
a workload attaches to this bridge it will add an entry of this MAC address is available on
this Linux bridge port right this will be reflected on the FDB table right and FRR zebra will
provision a type two route and this is it whenever the other nodes which could be in different
clusters see this type two route what'll happen is they will provision an entry in their FDB table
so they will know this MAC address is available on this VTP which is somewhere else in the world
this is for type two route that has just a MAC address as you see in the bottom sorry right here
now in order for you to have to jump into another network you will also need to have
an IP address right so there are type two routes that have MAC and IP addresses pretty much the
same constant whenever the workload attaches to the bridge it will send or a neighbor discovery
protocol or an ARP week west reply this will appear in the neighbor table FRR zebra will see
this thing will provision a type two route with MAC and IP address that you see there in the bottom
and this will be advertised in the fabric whenever a different node sees a type two route with
and MAC and IP address it will pretty much just provision a static entry in their neighbor table
and everybody will know this VM is available via this VTPP which could be somewhere else in the world
uh beyond layer two um well we have the ability of connecting different routes using these
type five routes I do not have a sequence diagram for that but it's the same constant but instead
of being FDB tables or ARP or neighbor tables it will be the route table and this will allow you
to access services in these other networks or and to provide direct route at ingress into the VMs
this means that you will be able to access to it using directly the IP address of the VM so
no need for that no services no nothing like that by service I mean no Kubernetes services again
and this is how it looks like the key thing here is the VNI must be different than the one you
specify for your uh L2VNI but the VRF must be the same that is how you connect a layer three VNI to
a layer two VNI so essentially how you expose your stretched layer two network to the outside world
essentially this just turns out to having an IP address on on your gateway on the bridge is
management interface and a VRF encapsulating all those routes oh boy could you get me like the
in there there's uh there's a thing so okay let's skip this let's go to the demo right now
so what we will see right now so we will have two different kind clusters
these kind clusters will have uh workloads running in them
we're going to check it
what it's done thank you huge thanks you saved us I owe you a beer so again VMs running in different
kind clusters everything is in this laptop but this is just to showcase a concept this could have
been like throughout the internet and then we will have a host on a separate network that will
try to access them we will see two things first we will see a stretched layer two network so these two
guys will be in the same layer two overlay and we will see another interesting thing we will see
live migration we will have this host a pinging this VM right here and then we will trigger a migration
to this other cluster while this ping is working and we will see how the traffic behaves hopefully
not many packets will be dropped so you can scan this for a link to the asking email demo there's
a link to the script that runs this demo for you so you can do this this at your home okay now this
is the recording unfortunately this won't be a live demo I hope the font is big enough
but let's see so okay oops so in the upper left I have the VM running and cluster a in
no I can in the upper right I have the V I have the cluster b so it has a different VM running in it
and this is where the VM one is going to be migrated to that is why if you look there there's also
VM one but it's waiting for sank jania right there and then we'll in a couple of minutes explain to
us everything about cross cluster live migration but for now keep this in mind VM one running
cluster a VM two running cluster b will communicate from one another and at the bottom we have the script
so okay details IP address in mac address of our VM one IP address of our VM two which is running
on a different cluster and the receiver thing which is the copy of the first VM
okay this is running now we're going to ping from the client so the thing that is outside a cluster
it's reached VM one it reached VM two so we have connectivity from a different network to these
two VMs running in different clusters and I'm just going to show that this essentially is in a
different network if you see VMs are in the 170 IP range and this one is in the 192 168
so I'm going to console into the VM and I'm going to ping this was too fast so from VM one I'm going to
ping the other VM which is the dot 31 so you have a stretch layer two between two different
Kubernetes clusters that was the first demo second demo will be seen now so this I'm going to
show you the type two route so we see here a type two route for Joseph Mac address but it's telling
you to reach this Mac this is the next hop you're going to be using interesting here also
MM2 this is Mac mobility we'll be seeing this in a while and then same information for the
other virtual machine managed to ping this pretty much essentially means that we have
started to ping from our client into our VM one I'm just putting here another ping so you see
that I'm not lying I've provisioned the migration in the destination cluster that is why the
VM one did something weird there and I will now trigger the migration in the source that is why
the virtual machine went to scheduling on destination cluster so I want you to look at this seeing
ping is working and I want you to look at this here saying virtual machine one is available via
this next hop meaning cluster A and traffic is working and the migration the VM is pretty much being
copying it'll take a little bit of time still cluster A schedule it knows to which note
it'll be and move to cluster B right here so this virtual machine meaning this type two routes
like this Mac and IP address is now available in cluster B which has a different next hop IP
address and as you see the ping is going at it like nothing happens it's not exactly like nothing
happened we'll see here like we've lost four packets during the migration we are using a 100
millisecond intervals so this totally amounts to like probably 400 milliseconds of drop traffic
it's not insanely good but it is quite good and I think this concludes demo and I
can leave this now to very go to walk us through the conclusions okay dropping up we saw how we can
leverage existing building blocks to implement a broader layer two layer across different clusters
we showed how you can wrap that layer two domain into a layer three domain and having
routable IPs from outside and I want to stress this it is in the very same way you would interact
with the real router with villains so you have your VMs today with the Mac villain
CNI plugin as a secondary interface you're have your villain coming from the from the router you
have your tunneling mechanism implemented into the router nothing changes from the VM it's the
same but with the component that is running on the Kubernetes cluster which is I think the
nicest bit of this implementation the fact that we are not reinventing the wheel but
reusing existing mechanism to to have a Kubernetes native way to do this and of course
live migration is the cherry on the on on the cake yeah this is more or less than
summer sources we try to keep the documentation aligned and nice to use there are a couple
of blockposts on the keyboard official blog in the open peer router documentation and repository
we are trying to keep a lot of examples on how you would integrate the project with existing
components with with the scripts and automation that you can run on your laptop and time
is up as far as the engine for the routes and that's it if you have any questions we are going to
be around