Hello everyone, so today's talk is kind of the segue of the talk that I gave last year here
in the networking room, it was the last year it was more like a prototype and I was a
full-fledged project. The agenda for today is why we might want to run router on our
Kubernetes nodes, what's this open-peer router project about is a router that you can run on Kubernetes
nodes, how you install it on your cluster and how you integrate it with your existing component
of the cluster and what's ahead. So first of all why we might want to run something like
a router on a Kubernetes node and the thing is on cloud providers everything is simple. The network
is abstracted away from us, we only have one big 8-h, 0 interface and that's it, but where I work on
is the dark side of things. On bare metal you have users want to do their own things, you have
multiple networks, you want network segregation, you want multiple violences, secondary interface
and a lot of stuff that require your fabric to be configured around your cluster and vice versa.
You need to have your clusters being talking to the fabric in a way that both are configured
together and also I think the life of our network engineer is a bit less boring, but that's personal.
So how is the fabric configured and most importantly who is configuring this fabric? So we said that
we want multiple networks and the most common way of doing this is with the violences interfaces.
So you go and configure some violences interfaces on your nodes. Those those villain interfaces
are translated into VPN tunnels in some way in your top of the rack switch, you have multiple
BGP sessions between your nodes and your top of the rack and everything works.
From a data point of view, there is a mapping between the tunnels and the violences and the
fabric is configured to encapsulate the traffic coming from into those violences into some kind of
tunnels. If we focus on the host, the host sees multiple interfaces, it sees multiple BGP sessions,
I'm a bit biased because I work on metal at BGP as well as an example, but the same happens with
colleague or with any other component that talks BGP in your cluster. So what happens is that every
single deal on every single interface becomes your access point to a different tunnel.
But the elephant in the room is who is configuring the top of the rack and whenever I want
to add a new network, who I am asking to configure that network into the fabric. And these
brings a common theme that I observed talking to multiple users, which is this kind of tension
between the networking folks that do not want to touch anything because they are afraid that
the network will break and the cross-efolks, which want to spin up a new tenant, they want to add
a new type of network to our nodes. And so there is this tension, which results in discussions
in our usability and so on. So what we are doing with open peer-outer is to take this concept,
the VPN termination, and we are moving that down to the node. So the same thing that was happening
on the top of the rack, the translation between a given VLAN and or multiple VLANs, and a
VPN tunnel is now handled by this new OpenP component, which runs into the node and which
exposes the networks as that lags toward the host. So open peer-outer, again, is separate network
namespace, the network namespace, isolates all the complexity required to set up some kind of
VPNs such as EVP and DXLung. It uses FRR as the control plane for the BGP part, and the
idea is that it behaves like a router. So the same interaction that our components would have
with a router now can happen towards this component. And of course the configuration is CRD-driven,
so we can leverage all the stuff that comes with Kubernetes. And of course we can have multiple networks.
So we can have multiple networks exposing multiple BGP sessions and acting as the access point to
multiple tournaments. And this should remind us what we just discussed. So the interaction is
exactly the same. The difference is that this is driven by CRDs and configured automatically,
whereas what we had before we had to go to the networking folks and to ask for a new configuration.
Okay. I work for QVirt pretty much. We run virtual machines in Kubernetes, and if something is
our bread and butter, it would be live migration and layer 2. So I'm here to speak about
everything related to layer 2. Okay. Thank you for that. So how would you typically do this without
open peer router? You need a new network, you plump a new VLAN into it, you plump dots into your
top of rack, your top of rack will pretty much have the IP address for the gateway.
And then the network people would go and configure the top of rack with the VPN blah blah blah.
Do a lot of people have to coordinate between themselves? So friction. This is not what we want to have.
Hence, we don't want to be a router. What you essentially have is you have your Kubernetes node,
you have your router running, which is that blue thing over there is like the network namespace of the
router. For every overlay you're creating, you'll essentially create a dedicated bridge for it.
So that number right there is the VPN on your using. So you have three networks. Each of them
is encapsulated on a different VNI and you have a VESPIR connected to that bridge and you're
left with a dangling breath on your host. Then you will plump this VESPIR in you will find a way
for this VESPIR to extend into your workloads. Essentially fabric becomes number.
So what happens to the fabric then? On the fabric side we don't need to configure
tunnels anymore. We only need to configure leaf routers to relay around this VESPIR router
related to the VPN and the traffic becomes completely routed. So it's plain IP traffic that
is being routed by the fabric according to the VESP constructs. So in this case there is not
need to involve the networking people anymore. The configuration is dynamic, it can scale and it's
very easy to add or remove overlays and please note the hand is not completely right.
The first kind of tunneling that we added is the VPN VX1 and in order to implement that we need
two things. We need to configure the session between the node and the top of the rack where we
can spread around our VGP routes and send out the VX1 pockets. We can do that in a couple of ways
we don't can peer out. One way is to steal an interface from the node. Can work on some scenarios
can not work in other. So we take the interface from the network main space of the host and we
move it into this new network main space. Or we can use multis because these runs as a pod.
So we can use multis and make VLAN interfaces for example to have one interface that allow you
to establish the VGP session with the top of the rock switch.
Yumult because it's Kubernetes. Like this is how you configure it is very simple. You need to
have a session between the top of the rack and your peer outer and of course we have not selectors
and all the nice things that come with Kubernetes and in this example we declare the interface
that we want to steal. Then there is this L3VNI that enables the creation of the battle
like that I was talking about with listening to BGP and what is going to happen is that all the
routes receive it from and to the host that's going to be translated into the event type 5 routes
with the local VTAP as the tunnel endpoint and from a traffic point of view the traffic from
into the fabric gets encapsulated and this is how you do that. You declare the VNI which is the
identifier of the tunnel, one VRF that is used for the Linux VRF inside the main space,
a local cider, one thing to notice that getting back here, one thing to note is that the
and the inside of the VF on the router will always have the same IP regardless on the node and
the host side is going to be different so your BGP configuration should be easier than
lay it to okay L2 again so essentially what you've seen is how to provide like an
interconnected type of networks this is how you would have like a layer to stretch layer to network
across your fabric does doing things without like interconnecting different type of clusters
this is what you would use an L2 VNI the interesting thing is you as I said before you will have
one VF bear per each VNI that is connected to a bridge a dedicated VNI for that bridge
optionally encapsulated in a VRF so you can have a network segregation in it and in the
host side what you'll have is a bridge you can be an OVS bridge you can be a Linux bridge you choose
what you want to plumb with and then the magic is that this bridge is what your workloads will
attach to you and how do you do that well this is how you create L2 VNI thing and the most
interesting things here is the VNI number that you choose and the VRF if you are if this is
essentially the key of the interconnection between your L2 VNI and your L3 VNI in case you want to
expose this stretch layer to 2D of sideworld okay all right have the mic so there is also this
pass through mode that was made more for testing integration and also if you have a process
on the host that needs to access directly to the fabric but you want still to have a single session
with the top of the rock switch so that is still the same you have something on this side that
talks BGP with this the only difference is that knowing encapsulation won't happen what will happen
no route translation will happen but it is going to just be translating the BGP routes
so second part of this is how does this integrate with existing projects and as I said before
one driving concept is we want this to interact with our existing components such in the same way
that they would be interact with their real router on the official website there are a few examples
that you can run on your laptop I'm going to just talk briefly about a couple there are layer 3 ones
so I'm going to talk about metal B again because quite familiar with the project with metal B we can
establish one BGP session pair of vf and again each vf is the access point to you for for you to
the death overlay and you can advertise different services to different overlays and this is what
is going to happen when you deploy the example on your laptop this is a container lab instance
connected to a kind cluster and metal B will advertise the the BGP routes are related to the
services to the open peer router open peer router will translate them as type 5 routes and they will
be spread across the fabric and the leaves are acting as the termination so this host here
will be able to access the service and again this is something that if you look at the website
you can deploy on your laptop and from data point of view you have TCP on the terminations and on
this path and all the rest is UDP pockets sent to and from the the VTPS. Calico is another thing
that I tried to to play with because I wasn't familiar with the project so I wanted to see if
effectively we could integrate with him and in this case the bird instance running on each node is
at the peer in with the peer router and advertising their podseider and we can have pod to pod traffic
so it's too astrophic happening through a week's land and with the routes spread around as
EDP and routes and we can even have those pods routable from the host connected on the same
overlay. Okay for L2 integrations we have two of them like with Q-vert pretty much like running
virtual machines and the equivalent but for pods so both of these are using secondary interfaces to do
that and this is a slide that I thought we would have before but remember that bridge we've seen
this is how you attach it to your workflows we rely on multis so secondary networks and you just
use your network attachment definition of choice to attach your VM to that bridge which again could
be an OVS bridge to be a Linux bridge could be a IPv LAN or a Mac file on top of that.
Interesting enough we also have like a distributed gateway IP so pretty much no matter in
where your VM runs because your VM can migrate to a different node in a different cluster it will
always point to the same gateway Mac and IP address does reducing the downtime while you migrate.
From the Q-vert perspective this is what it looks like it's essentially the same and our integration
like the demos we provide to showcase the integration also allow for a host on a separate network
to access the VMs which run on a separate network and for demos. Yeah so basically we didn't have
time for live demos but this is our reference to YouTube playlist with a bunch of
demos of the stuff that we spoke about just now I want to stress on the fact that you can run
those very same demos on your laptop by running a bunch of make-omans from a checkout of the
repository. So last bit is what's next what we are planning to do. So one thing is that all this
stuff that we spoke about now works beautifully for if you are leveraging your primary scene
I but whenever your application uses an SRV interface is an SRV interface that is completely
bypassing the host so it's like having one wire that bypasses your the peer router and goes
directly into the top of the rack and so you are back at the squid square one you have to ask somebody
to configure your your fabric. So this is something that we'll try to solve in some way.
Second bit is there is this chicken and egg problem because in order to configure their
router you need to access the API server but what if you want all your network to go through
this thing and so we are working and we are halfway in in emerging PRs to have a way to run
these as a system we unit that can be bootstrapped with a study configuration and then whenever
the API server is available then it's going to start consuming it's going to start consuming the
new configuration. So it's a fancy way to have the basic connectivity required for the
class to come up at boot time while being able to to configure and out the networks dynamically.
SRV seeks from the host point of view it's going to be just another type of vf so if you're
on colleague or metal be or cube virtual here nothing will change what is going to change is how
you configure those those vfs and how those vf where those vf leads so in this case some vfs will
lead to an unnecessary system which I think is the nice part of this architecture.
The data path is kernel based so we are investigating and discussing if there is a way to have
a plugable data path that can be faster and can also solve the server interface problem.
We are talking to the team behind grout which is a dbtk based graph router
it has already some integration with FRR so the idea is to have a plugable data path one can
be kernel for those users who are kernel speed is more than enough and grout or any other
possible data path available. Resources we try to keep the documentation easy to consume again
examples on how to configure open peer router to integrate with the existing projects there is
this talk that I gave last year about how the European works and how I started prototyping this
and that's basically so long story short is still a project that we are bootstrapping it works
it as CI we are open to collect feedback if you have use cases that we didn't think about
just file an issue and also we are trying to be a contributor friendly we already have some external
contributions which I think is very nice and with that we are done if there are any questions.
So we don't have much time but maybe just one and Maxine the next speaker if you're here
please come on up.
All right so thank you for the talk. The CI integration is just a BGP or does
colleague have something more fancier that you are using for it? I don't know. Like the only thing that
they tried we use is a BGP. Okay great yeah thank you very much sorry we're out of time
and again the point is that colleague was easy as a router so the configuration is identical with the