Hello. Good afternoon. My name is Miguel Wartz. I'm here with my colleague Ormergi to present
this talk titled GoBGP or Go Home, which is about simplifying keyword ingress with your
favorite routing protocol, which is BGP. So first, the agenda. We will begin with the motivation
for having, oh, for simplifying. Is the audio on it? Well, well, well, I don't know. Better now?
Okay, no, let's fix this. I think that you're stupid. I'll shout if I have to. Better?
Well, I think this is M-D-Wab-Del-Suck. But again, I will try to speak as loud as I can. So first, motivation.
Well, if you keep at this, nobody was going to ear anything. But, so I think we'll just carry on
and I apologize for people at home. First, why should we care about simplifying ingress? What is
there for us in this second thing? Why BGP? And then we will make a brief BGP introduction.
We will describe the use cases we're tackling. We will describe our solution, show a couple of demos,
because it's always cool to see things that work and we'll wrap this up with a few conclusions.
So, I'm going to start this with a show of hands to make some sense out of this. So,
who of you have used Kubernetes in the last years? Yeah, I guess a lot of you cube-vert.
Okay, some just a question. How do you expose a cube-vert VM to the outside world like
outside your cluster you want to access in? Do I hear service anywhere? I heard service somewhere.
Yeah, you would use a service. And how would you implement a Kubernetes service?
Well, the solution is somewhere in the slide and it's not. Thing is, not breaks your performance.
We don't want that. Plus, some verticals based some of them because of performance, but for
other reasons, they really hate that thing. Verticals like telco, financial services,
they really dislike having this. So, we want to try to avoid that. We do not want to have to
expose a workload or a couple of ports on a workload using that. And on another separate topic,
we want to simplify the Kubernetes admin life. We don't want them to have to be manually setting
up routes on your clusters. So, these are two things. Simplified the admin's life and getting
great of not complexity. This is what drives us. Now, YBGP. First of, it's an industry standard.
It scales a lot and it's proven to work. Well, it powers the whole internet according to what they say.
Another thing, policy control. You don't have to expose everything. You can have policy to
describe what you want to import, what routes you're interested on, what things or what routes
you want to expose to the outside world. Does you have a lot of control over how you design your
traffic segmentation. And it is a routing protocol. You just deploy it. Your routers will speak
to one another and they will scatter your network or the way to reach your network throughout
the entire fabric. And those are the things that we are interested on.
Very brief introduction about a BGP like fun facts, I guess, like this thing. I mean, some of
you today could like be inventing the next cool thing like BGP. If you go for dinner, have a few
beers, write a state machine on an app can. This could be it. 20 years from now or 40,
we'll be here hearing about it. Now, going back to what we're here. We are providing
integration between Q-Virt, so we run virtual machines and Kubernetes clusters and BGP.
And for that, we are using a particular C&I server, one particular C&I plugin. This one
work with all of them. This work with one specific one. It is often Kubernetes.
I'm a Kubernetes, a CFCF project. And it pretty much translates from Kubernetes API. So things like
services, pubs, network policy, and translates that into oven lingo or oven entities.
Good question. What is oven? So oven is an SDN control plane to open a V-switches. So it just
orchestrates a bunch of open V-switches, which would run each on a dedicated note.
And it supports Q-Virt virtual machines. By this, we mean that it has
a special code. Like he understands that a pod holds a VM and does special stuff to it.
And it is the default network provider for, well, for at least one downstream distribution
of Kubernetes. Then how do we implement BGP? This is implemented using open source. So we are using
FRR. FRR is pretty much like an open source implementation of a router. And in the Kubernetes
ecosystem, that is exposed to the cluster or packaged into the cluster with FRR Kubernetes,
which integrates with oven Kubernetes. In a way, we will see in a few minutes.
So how does this work in real life? So each of the nodes in your cluster will be running an FRR
Kubernetes pod, which means that it is running a package one version of FRR.
And this router will communicate with your cluster edge router, which will propagate
your routes or import routes from the fabric, which could be the internet, and would connect to,
let's say, your thing that is running somewhere else via a provider edge router for you to access
your provider network. Now for the use cases, I will leave it to my colleague, order.
Hello. So let's present a few use cases that we find interesting in our customers.
So the first use case is to get in access to the provider network. This is the painful point where
you need to mess around with services and these kind of stuff. So the use case is basically
to connect to enable a pod or VM to access your cloud network or provide a network.
So we do that by importing the provider network routes into the cluster network. We use BGP for that.
And we have a few custom resources that we can to control it. We will show we will see it in a few
moments in the demo. And this resource called FRR configuration. Remember this name,
we will see it in a few. And the other use case is the other way around, is to expose your cluster
network provider into the outside. It means that you enable services running on your cloud network
to tap into your VMs or reports. We do that with two resources. The same as I mentioned
earlier, the FRR configuration. And another resource called root advertisement.
And now let's get to a few implementation details. So API, we have two resources as I mentioned.
One is FRR configuration. This resource controls with whom to peer with. It controls
who are my neighbors, which means the BGP routers that I want to connect to and to manage the
sessions, what their autonomous system number and which node to consume this configuration.
It's provide granular configuration that allows you to take subset of nodes and do the BGP
sessions over them and not your whole cluster. And the other resource called route advertisement.
This one controls what to advertise in your BGP session. It means it can be
connected to your cluster, different network, you know the name or use of the fine networks,
secondary networks. And which FRR configuration to use for that?
Yeah, this is the resource set. So let's talk about the relationships between
organ K, organ Kubernetes and FRR instances. So we have organ Kubernetes.
It monitors your each node kernel routing tables and it provisions FRR configurations
according to your to the API, to your configuration that you want to make.
FRR Kubernetes take these configurations and translate it to the router configuration, running
your each node instance. It's pretty straightforward. This is how it's done. In addition to
FRR configuration, organ Kubernetes also monitors the route and that my advertisement,
a resource that I mentioned earlier. And according to that, again, translated to FRR configuration
and so on until your router, no router configuration. And one more nice topic is the failover
capability using the by B-directional folding detection for BFD for short. This is basically
what makes BGP to enable a high availability and failover to the next best route.
So BFD is the protocol for detecting fault the links between two devices,
device can be switched to router or router. BGP utilizes it to detect broken links and converge
to a 12-tran 8-1. And it can show rapid failover and high availability, which is essential
for production application and minimize the downtime. And with that, we will see a few demos when
we get what we do. So I'm going to try this now. Yeah, we're still a lot better. Should have thought
of that before. Yeah, so you can scan this to two access to demo, but I will be showing it.
I really hope the font is big enough. And let us begin. So first thing, we are using a kind
closer for this, plus a couple of containers that are deployed on the laptop. So we have our
kind of nodes right there. So we have a common control plane, a common worker, a common worker
too, like fairly intuitive naming. And then we have two extra containers. We have FRR and we have
BGP server. So to set up we are running is kind of FRR is router. Obviously. And then you have
BGP server, which is connected directly into the router. And then you have the Kubernetes cluster
containers, which are also connected router. So FRR has two interfaces, one connected to a switch,
which interconnects with the Kubernetes nodes. And it has a second interface, which connects to the
BGP server. The default route in BGP server is the router. Okay, what are we going to see?
I'm going to start this script that does this. And this is the configuration that we have.
So we have our cluster ESN. So 64, 5, 12. I mean, just has to be a number. And then the network
we want to import. So remember, we have policy. It's not like I'm going to, I don't know,
bring all the routes from the outside world. No, I just want to be able to reach this specific
route. So you have policy for that. This is the prefix we want to import. 172, 26, 00,
16. And we are deploying a virtual machine in our cluster as well.
Thank that is it. Oh, an important like the BGP server. So the container that we have
outside the cluster is has this IP address. So the dot 3 IP.
Okay. So, okay. This is the FRR configuration. Remember what or told us,
this defines the neighbors. So you're specifying whom to peer with here. So you're defining one
router in your cluster. The ASN I told you about and this router has two neighbors. One using an
IPv4 address and another using an IPv6 address. Essentially, it's the same router. The router is
just dual stack. So as you see dot 5 address both in the IPv4 and the IPv6 address.
And we define here and this two receive allowed. You're saying I just want to import this
particular prefix from the fabric. It could have been like in the allowed stands. It could have
been like all. And you've import everything that the fabric knows about. You would bring that down
to your cluster routers. That's not what you want. We just want these this particular prefix.
Okay. This just shows that the routers are running in each of the nodes. And now we are
provisioning the IP address. Sorry, provision the virtual machine. We see a discheduling.
Maybe this forward. Okay. So it is now running. It got an IP address. So it just has the
cluster default metric attachment. And I'm showing here, yeah, does the PMI spec for this virtual
machine and the interesting thing here. It just has a cluster default network attachment. So you see
here, masquerade interface type. Okay. Now what I'm going to do is to show the routes on the nodes
itself. So I created a debug pod on the oven worker, which is where our virtual machine was
scheduled on. And as we see we have imported this route from the fabric. So you see this
172 dot 26 dot 0 dot 0 slash 16 is a route on well under under on the cluster nodes.
And what we will do now is access the BGP server thing, which is located outside the cluster.
So for that we console into our VM. For the refer to the best word. And we will now
ping the IP address that is same right here and we have we can ping it. And we can also curl the
thing, which is what we wanted to show in this first demo. Okay. We can see the
ping again. Now what I want to show is that this is actually doing something. So I'm going to
delete the configuration does saying that my router is no longer connected to this. You see here
DFR configuration. And so you notice the ping still going, still going, going to delete it.
And the ping stopped. And what I'm going to do again is I'm going to provision like the script
is declarative. So it does nothing. If it already exists, one of the things it does is provision
again, the FR configuration. So the ping resumes working. And this includes our first demo in which
we try to access a profiter network. Let's move to the second one, which shows the second use case
or tool is about in which we want to advertise to the outside world, our internal network, or
our cluster network. So for that, we use the second script. Or we are using IBGP here. We can see
that from D. It's gone. From the cluster SN external ACEN, they're the same. So IBGP,
our neighbor, the .5 address. We are no longer using dual stack for this. We could. There's no reason
to. So let's not. Now interesting enough. The cluster network we want to advertise to the outside
world is this prefix right here. So then dot 100 dot 0 dot 0 slash 16. We want to advertise
the bottom network. And the provider service that we have outside a cluster is dot IP address the
same one as we have before. So this is the FR configuration. As you see, the only difference is
in the allowed. It has mode all. So I want to import everything the fabric tells me. Don't do this
in production ever. And this is okay. This is how we configure our UDM. So this is an isolated network.
This is just a detail. So let's skip over it. And this is relevant part. We have right here. So this
is the second CR dot or tools about this is the route advertising. So while the other told us
whom to P with this will tell us what are we advertising or exposing to the outside world. So
we want to use the FR configuration. So we want to define the one to consume the neighbors that
we defined on FR configurations with these labels. And we want to expose the networks that has
these this label right here. And that is pretty much it. Let's see what happens when we do it.
Because again, remember our second use case, the thing we will be seeing now is how can we get rid of
not and not have to use a Kubernetes service to access into our workloads.
Okay, the router pods are running. I'm scheduling the VM. Scheduling, scheduled, running.
Doesn't IP address in the range of the network we want to expose. I'm going to copy that IP address.
Now I'm showing the routes in the fabric, right? So interesting enough. You'll see here that
the fabric knows a way to access this prefix. And knows how to reach it in three different ways.
So you can use either these three next hop IP addresses to reach it. Why don't three like
these are the three closer nodes that we have like. So this means that the traffic could go
into a workload into any of these IPs.
Yeah, here you see the exact same thing, but not using like FR API, just the typical route
table on your nodes. And we can now try we will Docker exit into our container and try to reach
into the VM we have deployed on this isolated network. So Docker, exact BGP server.
And we will bang the IP address that we are seeing right there and it is working.
Which is what we hope to see. So you're exposing a workload running on a Kubernetes cluster directly
using its IP. We did not have to create a service. There's no not involved. This is literally
what we wanted to show. Now I'm going to do the exact same thing I did before. I'm going to delete
the route advertisements and we're going to be seeing that the ping will stop. I could have
done the same thing as I did before. I could move the FR configuration the result would have been the same.
As you see it stops, I'm going to do the same thing again. Run the entire script, which is
important, which will make the ping resume working. And it has resumed. So we'll take it away.
And wrapping up with some conclusions. So before we go, we want you to go with the following points
so we saw the integration between open Kubernetes and FR Kubernetes that allows us to connect
workloads over a BGP and drop the excessive service configuration and have seamless,
somewhat seamless experience of exposing your VMs or ports outside. And the most important thing
is that you can reach your VM with its original IP. No not no fun stuff and the weird IPs.
We saw that the integration with the provider network is dynamic. It means that if something fails
in your provider network, thanks to BGP, it will recover and learn and learn your provider
fabric routes and distributes them to your notes. So you don't need to do manual configurations
and it's just make your admin happier because it has less job to do, less work to do.
As he has no longer need to mess with annoying static configurations and manual work.
And as we mentioned earlier, we finally dropped the nut overhead and annoying configuration.
Increasing the VM and exposing it by its original IP. So in this feature is something that our
customers look for a long time. It makes them happier and admin happier.
And this is the end. We have a few resources. We have the BGP, it's a nice BGP intro.
The project's docs and the integration doc is pretty straightforward. I hope you you find it useful.
And that's all. Thank you.
How are there any questions? Yes, it's been a long time ago.
Do you happen to have some set of well-metrics where it shows how much improved
the performance is going the BGP way compared to not.
So the question was whether we have some metrics to compare between how we improve the net talk
between comparing BGP to not configuration. So rough metrics for that would be
like 40% for the not overhead and we didn't even talk about the underlying net work
that may be used VxLan and other encapsulation protocols. This basically, this basically
do we have more time? So the question was whether the VM can
ingress and egress with the same IP? So the thing about the egress IP is that you'll be pretty
much as advertising that to the net work as well. So the answer is yes, pretty much. If you're
using it to egress, we want to try to force it to go in the same way. Otherwise, your contract
tables would go bananas, your traffic could get dropped somewhere and debugging nightmare.
Yeah, that's partially one of the reasons why that featured that integration was created.
Sorry. I do not know what those are, sorry. And I think time's up, sorry. You can catch us
outside, okay?