That's something I shouldn't have.
So hello, everyone.
Today's talk is a bit peculiar, I would say, I have a lot of stuff to work on, so I'll
go super quickly about who I am.
I'm Federico Workford, right at in the Kubernetes area networking, specifically for
telco-core use cases, and I maintain them at a little bit project, so if it doesn't work,
you can blame me.
This talk is about my journey into a VPN.
I knew nothing about a VPN, I was asked to find a way to have a VPN termination at Kubernetes
not level, so this talk is mostly about my prototype and how I ended up having something
more or less working.
Now somebody needs a refresher on a VPN and a VIXELON.
OK, see, a few hands will be super quick.
Let's start with VIXELON.
VIXELON is more or less L2 packets encapsulated into a UDP, so in a regular spin and
lift's topology, you might have the lift that is doing the VIXELON encapsulation, so your
house is connected to the lift, to VELON interface, the lift does the encapsulation, takes
the packet, wraps it around the UDP packet, sends it to a tunneling endpoint, what's
called the V-tap on the other side of the fabric, the packet lands there, reaches the target,
reaches the destination, life is good, but in a big fabric, who is telling us which V-tap
to target, and that's EVPN.
So EVPN is a VGPS extension that allows us to spread the reachability information of macadresses,
we type two routes and also reachability of networks, we type five routes.
We can have a broader network connected or maybe a Kubernetes cluster, it's an association
between a L3 domain and a V-tap.
And of course, in the same fabric, you can have multiple overlays, defined by different
what's called the V-tap, so this is how the context, what is our destination, what do
I want to reach, this is a regular Kubernetes cluster, I'm afraid to touch anything,
this is a regular Kubernetes cluster, we have some BGPS-speaking components connected
to the fabric, they talk BGP through the VLAN interfaces, these BGP routes gets more commonly
translated into EVPN routes and we have a mapping between a VLAN and a network VRF,
and then when we send the traffic to that, that gets encapsulated.
What we, what my goal is, what I try to do is to have this concept, the router,
brought down at node level, in a separated network, in a space, talking to the host via
bad legs that are doing the same thing that the VLAN interfaces were doing, so we can
exchange routes to the underlay, through those VLAN interfaces and whenever we send
traffic to those VAT interfaces, sorry, the traffic gets encapsulated.
So, we take the interface connecting the node to the fabric, we move it inside the network
in a space or at least this is what I will attempt to do, the routes are advertised
by our BGP-speaking components, Calico, QV, MATLAB and others, and we'll be able to learn
and advertise routes and when we send traffic, the traffic will be encapsulated.
What is the advantage of these approach, the advantage is that whenever we want to add
a new overlay, we want to have to go to the fabric and reconfigure it, for example.
And to add more steam, I want to try to do that on my laptop because it's easier than to
go finding data center.
One node, I kept note of all the progress, so I have a repo, we do the very steps that
I'm going to present and in my blog I have a very sparse about the topic, so it should
be our reference afterwards.
What do I need to do that on my laptop?
I need a router with a VLAN support and my favorite Linux open source implementation since
I've been maintaining MATLAB is FRR, FRR is, obviously my favorite BGP and other protocol
implementations.
FRR supports EVPN with a caveat, FRR takes care of the control plane, FRR, but it wants
us to configure the host in such a way that the data plane is implemented by the Linux
kernel.
So, FRR is very pollinated in what it wants, for each VNI, it needs Linux VRF, a Linux
bridge, VLAN connected to it, and that's basically it.
Then there is the configuration, well documented in the project site, we need to declare
that we want to talk EVPN with the external router and we need to spread around the
reachability of our VTAP, the virtual endpoint, so then when the other side is sending
UDP pockets, it's able to reach us.
And then for each VRF we need to declare which routes we want to advertise.
So this is the first part, second thing that we need is a network lab running on my laptop.
And this is where container lab comes into place, container lab is a very, very nice project
that is covered, it allows us to create a network topology based on container images and
vectors, so it's slightly different from using Docker compose or some regular Docker networking,
because you don't have a bridge there, you have a link, so it's more similar to what
you have in a lab.
This was my first example, I just wanted to make EVPN work, it took a while, but I managed
to, again, I have the blog post, it was early last year and my repo with this example,
and this is how autopology looks like in container lab, so you define the nodes and you define
the links, basically you say which node you want to connect to and those are going to become
vectors, with FRR is slightly different, because you can't overwrite the entry point,
so what I ended up doing was to passing a setup.sh file, in each node and calling it after
the topology was started, and container lab supports many, many versions of the images
that you can run, commercial or so commercial others in your topology, but I just use FRR.
So we run the setup on each node and the setup looks like this, so what we need is to create
the host layout at FRR once in order to have EVPN to work, so we create a VRF, we
has laid the interface connecting the leaf to the host into the links VRF and then we create
the VXL and the bridge and so on, and again this was the first example, my configuration
is using now ready to be connected, so if we will share across the EVPN fabric, the
reachability of the host connected to it, as type file droughts, so if I inspect the route
in table inside the leaf, this is what we get, type file droughts, the local network and
the local beta, and when you ping it goes through the leaf, it gets encapsulated and gets
on the other side, seeing this ping worker was pretty exciting when I tried it, I know it's
just a ping button. So again, the references I have a demo, a video, I have a blog post
and a rico. Next step, we want to plug a Kubernetes cluster, stealing my laptop,
still in the topology, so what I did was to use kind, kind is the one-on-project to use
for development, CI and whatsoever, and the container lab has a nice way to have some
nodes of the topology, which are actually kind nodes, and this is how you do that,
basically you need to declare it in some way, I had to do some configuration inside the
node, and that was it. Reminder, these are our ultimate destination, I container, vet
bears, Linux VRF, BGP session with the host, and connected to the fabric, just a reminder,
so I'm going to show what the setup of a kind node looks like, basically here, this is
what I'm running inside the container, and what I'm doing is using Docker in Docker to spin
up another for our instance inside the container, I take the namespace, I create the
vet per, I put one leg of the vet inside the namespace, and then I take the interface used
to connect it to the router for the underlay connection, and they put it inside the namespace,
and then I do all the setup required for FRR, and then as an example, I can plug metallic
be very in a project, it can then advertise metallic be, a lot of one-on-cerve service via
BGP, so what I'm going to see into the router is that the local router is learning the
BGP route through the vet as a BGP route, but it's also going to convert it as a type
5 route with a local endpoint as the destination, so if I go on the other leaf, I'm going
to see that route, and I'll be able to reach the service from one host connected to the
other side of the fabric through VXLON to the node to metallurgy. Again, I don't have a,
I have a demo, but I didn't have time, so I have the recording, the repo that you can
tinker with, and a blog post explaining all this layout. And then move much further, so
first, before it was a continuing on the host, now I wanted to have it as a pod, because
it's Kubernetes, life cycle is easier to manage with pods, so I try to tinker with things
to see if I was able to do that. And again, the destination is this, now we have two pods.
One is one classical pod, it has its own network namespace, runs FRR, plus some things,
one entry point to do the setup, and then I have another pod, a host network with a lot
of privileges, a lot of sockets mapped in order to mess up with the host. So the other pod,
what the other pod is does is it takes the interface, and it moves to the pod, to the target
pod, creates the method, and puts a leg into the name space, and more or less it works.
So the new difference is, I don't know if we can left about I copied a lot of the logic
from his multis dynamic controller. So at runtime, we use the container runtime to find
the right target, and more or less the stuff is more or less the same. So this is the main
difference from before. And this is the high level, I want demo it, but I have the demo in
my reference, and here we have a far overnight, it is able to exchange your hours with
this router in a pod. And this is around November, when I started to say, but maybe I can
try to write some real code and having a proper Kubernetes controller with that can be dynamic.
So things went on a bit, and I have this project. I don't know if it's going to become
a real project or just a POC, but I build a repo for it. It's the same architecture as before.
We have the FR pod, we have a controller that is, it has a CRD-based API, and all the logic
now is in the controller, because it was easier to have the hosting configuration and
FR configuration in the same place. So the lifecycle is like the controller takes the interface
and moves into the main space, it creates all the interfaces required by FRR, creates the
developer, provides an FR configuration, sends a signal, and it tasks FRR to reload it.
And this is the high level of having given as a pod. The API is pretty simple, again, subject
to change. I need a slider, because I need a different data for each node. I need to
tell the thing what is the interface that I want to move, and I have the session with
the external router. On the other side, for each one I want to define is the session with
BGP speaking component on the host. It's basically the evolution of the previous
examples, but in a Kubernetes session, and the beauty of it is that it can interact with
any BGP-enable component running on the host. It can be, again, whatever is able to
talk BGP. And one nice thing is that the API of the router side is the same for all
the nodes. So you don't have one session per node with a different API, but you just have
a metallic BGP-peer, for example, and it works. Still very working progress, don't
blame me if it doesn't work. Come back to the chat.
A couple of months on the repo, I hope it's going to work flawlessly, more or less.
I have a demo. Basically, I took two nodes, connected to the topology, running
Calico, Calico is connected to the router via the VF, why Calico, because metallic
B would have been too easy. I know the ins and outs of it. I wanted to take something that
is BGP speaking, but I didn't know anything about it, to show that it was working.
The way Calico works is that it talks BGP with the, using bird with the router. Each is
announcing it's not IP-sider to the router via BGP. Those get translated into even
routes across the fabric. The other node learns how to reach the pod side of the previous
one. We send a packet encapsulated and encapsulated. Now, I have a real demo. I have a recording,
but I also have a shell prepared. I will try my luck with the live demo.
Trammorow. These are the Docker containers, the topology. Two of them are nodes of the kind
of cluster. I have Calico that wants to peer with the IP of the local router, but it's not
established, because I didn't configure the thing yet. If I look inside the FR container
into the pod, it doesn't have the external interfaces. I don't have routes into that.
Now, I have to work loads. One running on one node, one running on the other. I will try to
ping and ping. It doesn't work. Now, I'm going to apply the configuration. I'm going to
wait Calico to tell me that the BGP status is established. Now, the session is established
on one node, but it should matter. Now, if I inspect the FR container into the pod,
I see all the stuff that FR needs. So, I see the DRF. I see the Linux bridge.
It is laid into the BRF. I see the VXLON and I see the lag, the vector lag, connecting
this thing to the host and I see the app link to the external router.
Now, I can see that on one node, I have a route targeting the pod
side or to the other node via the IP of the router through the vet.
Again, workload pods. I will try to ping from one to the other. Now, ping works.
I try to ping from another host on the other side of the topology and ping works.
If I try to TCP dump while I ping, my neck is hurting.
So, I see the pocket crumming from the host, the ping, the ICP request,
then we are going to send it out because we have our route to the destination
through the Linux bridge, then we see going leading to the fabric as a VXLON pocket
and then we see the reply coming back as a VXLON pocket and then get in the encapsulated
sent to the host again where it finds its way to the pod.
So, that was it for the demo. It worked. I am pretty happy.
And then I can skip the recorded one.
So, where do we go from here? First of all, I need to make it work flawlessly.
I need to test a few things, but it's a nice prototype.
We want to enable L2 it depends. So, we have to have a way to expose some way to connect
an extended L2 domain on the host, possibly with an extra VF, or I don't know yet.
We need to sacrifice one interface. We could use VLONs to move the to have a main interface.
But, what if we want EVP and connectivity at day zero? Because it's a chicken and a problem.
If EVP is all we want for our node, we will need to be able to reach the APS server.
So, we need something that is alive before Kubernetes.
So, I did some experimentations, running podmen pods,
as SystemD units. There was a talk yesterday in the container dev room,
but I'm not solved yet on the idea. Still, I still have to do some experiments.
Maybe I faster data path. Like, this is kernel-based.
We might, we had a great talk by Michael League about the ground today.
Maybe we can join forces and have the PPK based data path,
or maybe this is going to end up being a PLC and no one will use it.
Rapping up, I think I talk about a few things.
How I went from knowing nothing about EVP and to having something that more or less works,
using stuff that was happening in my laptop, that made me,
my productivity, like super, super quick compared to having to deal with the real routers,
because every time I was missing something, I was just dropping everything and recreating it.
So, that was, the container lab is, if you are into networking,
is a very, very, very nice project.
If you resources and then done, I further routing my best favourite protocol implementations.
Container lab, I have this repo and my personal blog and the repo of the project,
and the idea comes from my discussion with the Dostaticon folks last year here at Phosem.
They have a nice implementation, probably, I think it's going to be converted.
These are pros, but the idea of running EVP and termination and the node comes from that.
With that, I finished writing time. If you have any questions.
Questions, anyone?
Hi. Thanks for sharing your work. I look forward to reading through what you've put together,
because I've personally been trying to solve...
Can you erase your voice or put the microphone in the air?
I've been trying to solve this problem myself, and I was wondering,
so from your example, we have metal LV, metal LV advertised the IP addresses, the BGP IP.
So, for ingress, into the Kubernetes cluster, that just basically works with the way metal LV.
Because I had to log in that very specific example, I had to add the static routes.
But with the far Kubernetes, now the metal LV, you can also learn the routes on the node,
so you can have this metric on path for free more or less.
Yes.
Good catch because there was a hole in that.
So, I have a second question, then, have you considered how you might implement the same sort of logic for egress?
If I have a pod in Kubernetes, I want to be able to talk to some other...
But that was my colleague, example, did basically.
So, the pod is part of the fabric, so the house was able to ping it, but not only that,
the reply was getting back to the house on the other side,
so that was made up with the EVP and VXLON.
Okay, I missed that.
Thank you.
First of all, thank for your presentation, because it was like mind blowing from an active perspective, so many thanks.
My question is, have you tried to play with the interconnection of the CNI or to say another word?
Now we have a mechanism to connect the host with some kind of VRF on the networks.
My question is, have you tried to connect the network of the Kubernetes with multiple networks?
I'm putting VRF on the CNI itself and connecting the network.
Like, VRF CNIs are not equipped for that today, so it's easier to have that kind of exposure at all levels.
But of course, you would need to have orchestration with the CNI to be able to...
The CNI would need to be VRF a word, basically.
Okay, thank you.
Thank you.