So, yeah, I will be talking about what I have learned working with GOPGP and all
to scale it.
So, first about me, I am a software engineer at Cisco.
I have many worked on data plan VPP and GOPGP, well, VGP, a control plane.
So, I will be not talking about VPP since there have been many talk about it yet, and
also I like some things, so if you want to talk about that or so, we can talk about it at
the end.
Small disclaimer, you will see this logo, this is a real GOPGP logo, and I have made some
AI generated one for the torque, because we are not the real one.
So, first, what is BGP?
BGP is the air traffic controller for all the routes on the internet between the autonomous
system.
So, basically, you tell a BGP server that you own some routes and that you want the traffic
to go to you when somebody wants to talk to that route.
So, GOPGP is in between to distribute all the routes between different autonomous system.
So, what is the province that we want to solve is that needs to be for which we need GOPGP
to scale.
So, we have different tenants that want to distribute the routes between different devices
that they own across the world in different pop, but we don't want to share the routes
they distribute with us.
We want them to have a segmented way of distributing their own routes and have multiple
peering with us.
But for that, maybe we could have on-brand and 1000 of peering with one single customer
in one pop, and we want to have multiple customer in the same region.
So, if we take an example, at an end one to connect with us and start sharing some routes,
so it will register itself through our fabric.
The management plan will inform our BGP plan, the control plan that's a new tenant
one to peer with us.
We will start a new GOPGP server for it, and then some devices for the tenant will try
to peer with us and maybe multiple of them, we expect some time thousands of them in one
single region, and they will be able to start sharing routes with them.
And we will register but read just a bit, then across all the different peering that they
have with us in this pop and in other pop where the client is connected to.
So we can see that, for example, the first one we start appearing at a given address 1625401,
then we have a not clear that we will be mapping this client to its tenant server and
if another device break to connect, they will have another device ID and we will be able
to know which server we need to map them so they can peer with this exact server for the
tenant.
So what do we need?
The tenants can come and go in one single pop or multiple pop, that can be thousands of tenants
in one pop, they can create different devices and connect at once in one pop, so it is
really dynamic.
And also we want to be able to customize the pass, they collect some metadata in the pass
in our fabric, because as I said, maybe the client is connected in multiple pop and we
want to, you know, just admit a data to those pass, to pass them to other pop, so that
we know from where the pass is coming from originally, so we need a way to manipulate the
pass as we want to vehicular just metadata.
And as I said, there can be thousands of tenants in one single pop that are coming and
going, potentially with also a lot of peers and in the end, it's make a lot of routes to
catch and vehicular.
So we want to be able to serve all the tenants as rapidly as possible, so we made around
all the different open source, BGP and implementation on the market, so there is burned,
which is fast, but it's only usable with static configuration, so you give this
it, you give it a huge file that describe where are your peers, or they will connect to
you, etc.
The same for FRR, it's also really fast, but still you need to give it a static configuration
on all your aspects, the different tenants to connect, so this is not dynamic, and this is
the first point of what we need for our system to be able to add rapidly new tenants,
so the third alternative is GOBGP, which allow to add a new peer-spass, etc, via an API
directly in Go, and that's all our problem.
There is two main way to integrate with GOBGP, the first one is as a GOP package, so you
can just use it as a library, so you have access to all the internals, and you can just,
well, whatever you want with the GOBGP plane, so with the routes, the peer, etc.
All you can also have it as a demand running on your server, and you can make JPC request
to it, but we choose GOBGP for treatment reason, the one is as I said, we want a dynamic
system, so GOBGP is the two-go solution, we want strong pass manipulation, I know that
bird has some DSL where you can manipulate the pass, but not as strong as you can do with
GOBGP, GOBGP, you receive the pass, you can do whatever you want with it, and register
redistributed elsewhere, and also, as I said, you can use it as a GOP package, and most
of our agents are written GO, so it is really easy to use it.
The only issue compared to bird and F error is that it does not scale well, so I will show
you some numbers on some benchmarks that we have done on GOBGP, and the benchmark setup
is as follow, we have a bird server with a fixed number of thread that can have access
to an unlimited number of RAM, we have a static configuration simulating a huge client that
want to peer thousands of times with us, and through which pairing we will be sending
a certain amount of routes that we expect our clients to send us, and we have on the
GOBGP server on which we will inform on the GO, new tenants, and so we will expect
new pairings to arrive, and so we will be monitoring the pairing, we will be monitoring
the number of routes, and we will also do some profiling on the GOBGP server.
So for example, with 1,000 pairs that exchange 50 routes of their pairing, we can see that
in one minute, we have connected all the pairs, and all the routes have been received
on the GOBGP side, and we can go over with 2500, also in two minutes, we have all the
pairs that get connected, all the routes received, but we can see there, sorry, I cannot
press there, there is a small step, and we will see some issue with the GOBGP 337 that
we have on-culture, and there with 6000 pairs, 20 routes each, we can see that we connect
all the pairs in five minutes, we receive all the routes, but it's kind of our limits,
we don't want to be too long, because five minutes still can be too long for some
customary, we are waiting a few minutes to have all the routes being distributed across
your network, you don't want to wait that long to be able to receive your traffic.
And we can see that there are small steps there, is one of the limitations of GOBGP
337, where everything goes through one single channel, GO, so your management, your
IPI schools, your GOBGP message, all of that goes through one channel, so that makes
that when you have some pairs that get established, that starts sending routes, that you
have GOBGP updates, super populates to other pairing, you also push that updates into the
same channels that is bounded, and so you cannot manage all the IPI calls to the IPI, so we
are tending GOBGP on the fly, that we want to add some pairs, as a start of the test,
but we can see that we cannot add all the pairs just at the beginning, because other
BGP message are polluting the channel, so it makes the test go a bit longer, but also
it shapes all the pairs are able to connect with, so at the beginning a few pairs can
connect, starts exchanging routes, and we add over time other pairs, because the previously
connected pairs started sending routes and polluting the channel, same things with 7,000
pairs, we can see that it takes longer, the more pair we get, and same thing with 10
k pairs, and after that in GOBGP 4.0, we worked with the maintainer of GOBGP to remove
the limitation of sending everything to the same channel, and splitting the management
and API channel with the BGP updates, but still there is an issue, when we receive an
update, we have a big lot, that's prevent anything to be updated, because we need to fetch
the table of the pass, we need to fetch all the pairs that are connected, start sending
routes, etc, so the first solution was to have a big lot around that, but then you have
a huge contention around that lock, and we'll start seeing the issue at 6000 pairs, we can
see that there is no more the step on the at-pier of the time, because we are able to
add all the pairs at the end, at the starting of the test, we are not limited by the
bounded channel, but because of the contention, some pair we can see will not be able
to receive the k pairs of messages needed to keep the session alive, and so after the
all-timeer expire, we can see the number of pairs decreased, because some pairing just
dies, and we start over, because we release some contention on the lock, and we are able
to establish some connection, but as soon as we receive messages, there is more contention
over and over, but in the end we are about in 15 to 20 minutes to connect all the pair,
and for 7000 pairs, we can see that it is an issue, we are not able to connect all the
pair at any given time, so some of my colleagues, that is in the room, Nikola Planell,
worked on raising the contention and on the lock, and we are able, so now to this, with all
the work that have been on the channel, and on the lock, we are able to have 6000 pairs,
and multiple tenths of routes, pair pairing under such seconds, which is better than what
was capable, go to GP, treat 37, and what we have now. So as I said, treat 37 was capable
of multiple thousand of pairs, but there was this limitation, because of the main goal channel,
treat 0, decouples to the channel, but in 12 is another issue, because of contention, and there
is a pair that is open for review, that solves all these issues, but still we want to go
higher, multiple thousand of pairs with multiple tenths of routes is great, but we really want to
go higher than that, so what can we do? We can just kill out, go to GP, so what we can do is that
since we have already a nut layer, just before the GOBGP servers, that map devised through a
tenent server, we can just replicate multiple tenent server and redistribute all the routes
that server receive to all the other server, thanks to the GOBGP API, and the nuts will also
be a load balancer in that case, and when you receive a new connection for from a device,
you just assign it to a tenent server, so we can spread the load more efficiently on a different
GOBGP server for the same tenent. And so currently there is 4.2 that is released, but still there
is not the great patch of Nicola, that's all the issue of GOBGP, and still there is a lot of
places that can be optimized for a GOBGP where other locks can be removed or optimized, so if you
want to, if you are interested in a program of software optimization, you can overlook and thanks
for your attention.
All right, thanks back to him, so we have time for questions.
What about open BGP-D, an example BGP-D, do you try with them?
There are two other three software BGP diamonds open BGP-D on the example BGP-D.
No, we didn't go with that either, because we make, I just presented the three all of them
with BIRD and NFR, because GO, well, it's already done in the city, and also the fact that we
use GO in a lot of our codebase, so using GOBGP was really to go, and actually I don't remember
if they are also an API driven capabilities on this project.
An example BGP-D has, because the entire point is to be able to manipulate from the program.
It is in Python, if I remember, yes, yeah, okay.
I don't recall why we got with the GOBGP, but I think it was a choice also, because it was
solving our issue of dynamicity, and that we were familiar with GOBGP and that a lot of
codebase is in GO and it was more integrated with what we were doing.
Thanks for the talk. I have a few good baseline numbers on FR and BIRD for the same number of
problems. I don't have Zen there, but here we tried some time ago, we were facing these issues
of scaling before adding multiple favour for the sentiment, we tried with BIRD like,
updating the static configuration file on the GO, and basically we got exactly the same numbers
than with GOBGP. So, update the fix of BIRD-D fix. BIRD-D fix, BIRD-D fix, BIRD-D fix.
Yes, any other questions?
When you load balance the BIRD-D pairing, or do you make sure that you are still always
connected to the same pair? I have a backup slide, but maybe it's not sure.
I removed a lot on all we are passing the GOBGP messages, so the device first
go through an IP signal, and there we can know which identities and which device it is,
and we have a tunnel, so not also the encapsulating of the tunnel, so it knows which device
and which identities, and after that we just icing based on the device ID, you know, you just
do a module, the number of server, and you end up always on the same server. Thank you.
Well, thank you for the talk. I wonder if you are targeting a use case with high availability
roads, and if so, then you probably would want a BF-D-D. Do you plan for a BF-D-D protocol?
Oh, BF-D? Yes, BF-D. Yes, BF-D. For it's a companion protocol for BGP, that allows faster, faster
switch of roads. Do you work on it? No, actually, maybe we didn't look into it because
our needs were not, were fulfill already with what we were doing,
but you say it's complementary? It's complimentary and it's targeting fast switch when
Piersville down.
Sorry.
The B of D is used for faster expires to part of the health level and so on.
I'm out.
So it can be like milliseconds.
You have some of the, I don't think you're more efficient.
Okay.
No, we did not look into that.
So I'm sorry.
I'm not able to turn the cell to that.
Okay.
Anyone else for the question?
All right.
Have a nice hand to Maxine.
Thanks.