So we can begin. Thank you for coming. We're going to talk about securing the internal
control plane and I am Antonius. I work at Cisco for the past eight months, but this
is some work that I've been doing personally for many years now for my own network.
And let's talk a little bit about the problem that we have, and I've seen this problem
in enterprise networks, like especially if they're large enough and complex enough, you start
using EBGP internally. So you have private ASS and you speak EBGP to all of your routers,
maybe you use some rout with reflectors, and that's how you glue the network together,
and if you use VPNs, so things like OSPF are more tricky to get working, eventually it's
very common to rely on BGP. And this doesn't matter, like this ASS can be different
rocks that you have in your network. It can be different branches you have around the country.
Whatever this ASS is meant to you, you usually have a set of that looks like this.
And of course, every ASS that you have has its own prefixes. In this case, if they are branches,
they have the Wi-Fi prefix and the gas prefix and all of that. But this also applies to provider networks.
I've seen provider networks in the wild that use EBGP internally with private ASS.
And this is something that basically applies to them as well.
So I run my own ASS. It is AS4601. And I've been using that for several years now.
And all of this that we're going to talk about today has been applied there.
I had the problem that I'm about to describe, and I'm going to tell you how I solved it,
and all of the operational experience that I got out of it.
So let's talk about how routing is done. And first of all, there is no IBGP, no router reflectors.
I knew we're in the network. All of the sessions are IBGP sessions.
So it's very similar to the thing we described earlier. The only difference is that I'm using BGP confederations.
That's a feature where you can use your public ASs for all of your routers.
And then internally you can split it into private ASs, and they have one for every router.
So every machine I have has its own private AS.
And it's router acts independently. It does not rely on other routers or router reflectors to make decisions for it.
It has its own rib, and it makes its own decisions.
And if we want these routers to collaborate, I use large communities.
For example, if I have an IXP, root server session, I target with specific communities.
So then a router further, like maybe in another country, can make decisions where that there's prefix was learned from peering or transit.
But these communities are informative.
And it's multi-vendor because it's something I do as a hobby, I pay for it myself.
I did not go to a vendor and get the latest and greatest hardware.
It started with whatever used hardware I could find.
And now I'm slowly as I replace the hardware that's changing.
I'm moving to a Linux based routing with Tabian and BERT2 as the routing demon.
And with kernel routing or VPP.
Right now it's not a huge network or a large one.
It's around 20 core, like IP routers that have full tables.
And it's a large enough where you cannot start to see what breaks and what works.
So the networks look like this.
So it's the same as configured everywhere and the private asses that are used to communicate with EBGP on all of the sessions.
And by profession, I am a security engineer, like networking started as a hobby.
So I wanted to see what I can do to improve the security of this design.
And I wanted to move to reducing trust.
So right now, only routers only trust themselves in this network.
In most designs you'll see today, every router, every equipment you have, everything that participates in BGP.
It is usually assumed to be trusted.
I try to move away from this design to something where each router can only trust itself.
It doesn't have to rely on everything else.
And everything is treated as a hint.
Like if you see this large community, okay, it's there.
But you don't necessarily trust it unless you somehow verify that.
And this is working.
So it's kind of like a mini-internet, the same way that when you have an AS and you connect to your transit and your peers,
and your down streams, you don't inherently trust them, you apply filters, you apply prefix limits.
You do all these sorts of things.
And this is what I'm doing internally as well.
And hopefully this means that even if you get root access to one of these machines,
the damage is contained to whoever is connected to that.
You cannot take down the entire network, all of the services,
even if you have root access to one of these.
However, this is more of an end goal.
It's not possible yet.
So I'm not going to give you root access to try.
And when we validate public sources,
when you connect to other ASs in the Internet, there is a playbook.
This is most of the NL-Nox best practices and how to validate prefixes.
So you drop RPKI invalid, hopefully you have the IRR,
RPSL stuff that you are doing to generate prefix lists and all of that.
Then there are certain ASs, there are IP addresses that cannot appear on the Internet,
and you drop these two.
And of course, you know, if you have a Slash 49 or a Slash 100,
this prefix is too long and it is being dropped hopefully by your filters.
So you do all of these things for your connections to others.
And then eventually if you have a prefix, you can certainly, you can be well,
as close to certain as possible that this should originate from this AS,
and it's not a hijack or a route leak or something like that.
But when I looked at how we can validate internal routes,
like what do you do internally,
it was slightly different.
Like the rich URLPF to drop the traffic,
but this can go only as far as the control plane,
the BGP will allow it to go.
However, a lot of the problems that exist on the public internet,
they usually exist in sufficiently complex internal networks as well.
So you have route leaks like,
okay, like this end branch in the middle of nowhere,
somehow is part of my backbone now.
Or, you know, this prefix was assigned in New York, right?
Not Seattle. So why is it being sent to Seattle?
Why is it originating from there?
And it's usually even messier than the internet,
because you don't know the IP address holders,
you don't have a definition of valid.
So, you know, you can say, okay,
we started all of our company with a Slash 24,
as the smallest prefix.
And then suddenly we merge with another company,
we acquired another one,
we had IP conflicts,
and you have all sorts of exceptions that start to add up.
So your entire playbook is kind of useless,
like what you do for external connections
cannot be applied within your own AS, within your own domain.
And then I started looking at other networks
I doing, I spoke with operators of mostly enterprise networks,
and the most common answer was nothing.
We trust every single piece of equipment that we have,
and most of the time it works,
like, you know, sometimes the network may go down,
but it's too complicated for us to deal with it.
Others have static prefix lists,
that they generate and update.
So when they install the equipment,
the SSH, you know, into that,
they configure a prefix list,
and then if they get a new link,
they have to SSH again and update it.
But this is very manual,
even if you try to automate it,
it's complicated,
it has a lot of issues.
So you may have, you know,
two links that connect to data centers,
and you only updated the prefix list in one of them.
So you think you have redundancy and you don't.
When that links goes down,
you have, you know,
your data center, your network is split in half.
So I was thinking that,
it's something that has to be solved.
It's something that's complicated enough.
What am I going to do personally to try?
And make this better.
And instead of creating something new from scratch,
I tried to reuse the playbook.
Like we have all of these systems already,
and this is the playbook by the way,
and it works.
It works well enough in the public internet.
So why shouldn't it work within your own network?
And this is to complicated,
like you have IRR there.
I think everybody hates it,
nobody does it right.
And the bug on list,
it's difficult to define all of that.
There are problems that you,
you start to have.
So I even went and tried to simplify it.
And this is a simplified version.
Let's only use RPKI.
So how does that work in practice?
So first of all,
you cannot use the five IRRs for this.
And what you do,
the rough idea is that you run a private RPKI.
And instead of this having,
your public addresses,
it has all of your internal addresses,
and all of your internal aliases.
And then, like in theory, again,
you go to every BGP session you have,
and you drop all of the invalid.
And okay, that's good enough.
But ideally, you move a step further.
You also drop the unknowns.
And this is a little bit more dangerous.
And this creates a single source of truth.
Your RPKI data contains everything
that you need to know about the network.
Here's a visualization of how that would look like.
You know, you have a slaster to 1.2 points link
that's being originated from this private AS.
You have this slaster 48.
And even if you have internal aliases,
or things like that with multiple origins,
you just issue multiple rows for this prefix.
And then you can just get basically
the entire representation of how the network should look like
into RPKI.
So filtering suddenly is simpler.
You need to generate filters around them by hand.
This is bird syntax.
Basically, if the net belongs to your IGP,
check if it's valid.
And if not, just drop it.
Or, you know, do something extra with it, maybe.
And suddenly things are much simpler.
If something is RPKI valid,
you let it propagate.
If it's not, you don't.
So this looks amazing.
Let's talk a little bit about how it's done in practice.
So first of all, you have to build this private RPKI.
Like it was a single bullet point earlier,
but it's more complicated.
When I started in 2021, there was no software.
So you had to issue all of the certificates
and build everything manually.
But thankfully, things changed since then.
And it's now much easier for people to set up something like this.
There is Krill by Annel Natlobs,
and it's designed for the public RPKI.
But there is now a feature where you can use it
to create your own hierarchy internally.
And it's, you know, an open source software
that you can use.
You can issue rows for all of your prefixes and your ASCs.
And you can have a separate structure away from the public RPKI.
The next thing you have to do is, you know,
calculate your data.
This obviously depends on the quality of that in the first place.
Some people use Excel or something like that.
To document their network, some people use networks.
So this really depends on what you are using.
Personally, for my network, it was networks.
So it has a great API.
Krill has a way to import everything and create all of the rows.
So it was fairly straightforward to get the source of
networks and import it into Krill.
And as long as you can, whatever you have,
even if you don't have anything, as long as you can create
a map of prefix to internal AS, this should work.
The next thing you have to do is to keep the data in sync.
And if you use something automated,
the API, you can probably rerun the same script
and you can update Krill all the time, keep it in sync.
Now, if you use something like Excel or something,
I don't know, then it's more complicated,
but consider dropping this not automated old tool
and move to Krill for your source of truth.
It supports the IP to AS mapping, so maybe it's a good IPom.
I mean, you can try.
And the important thing is that this should be kept accurate.
The positive thing about that is that if you don't keep it in sync,
and you set up a new link or a new prefix,
it won't work.
If you drop unknowns, your network will not work,
so you will know your database is not in sync.
The only way for you to use a prefix is to actually set it up
on your source of truth correctly.
So, for me, it's a feature, really.
And then you have to get the data into your routers.
This is the same way you do RPKI for your public internet stuff.
You have to do it internally as well.
You have two options here.
The first one is you can run two instances,
like, for example, routinator is the easiest software.
You run one instance for the public internet stuff,
and you run another instance for your internal RPKI.
And of course, this is not, you know, two physical instances.
You can have 100 virtual machines, it doesn't matter.
It's two hierarchies in reality.
And then you point into your distribution points that create
creates for you.
And it's the same thing.
Same configuration, nothing changes.
And now your filters are if this prefix is mine,
then and sure it is valid.
So, we're done.
Right?
Well, unfortunately, no, we're not done.
Because when I try to run this,
I stumble into so many problems.
And some of them went away, but not all of them.
And this is what I wanted to share with you today with this talk.
So, let's start looking at all of them one by one.
And the first one was that not all BGP implementation
support multiple RPKI sources.
I used birds, so it was very lucky that it can do it.
It can have unlimited sources and everything works musically.
However, if you use like FRA, for example, I think it doesn't
exactly work.
What you can do there is merge them, merge the public
and the private RPKI.
And I didn't have any problems with that when I did it.
So, I think it should work just fine.
The second thing, if you use BGP configuration,
is that handling them is not standardized.
So, for example, in Verde, if you query the UI like you do,
the show routes or something, you can see AES 65,000,
if it's a configuration, AES, it will be in brackets.
But if it's the EBGP 65,000, it will not be brackets.
However, when you write filters, there's no way that I know
to distinguish between the two.
But this is still not a problem.
You're not going to get AES 65,000 from the internet.
So, as long as you use private AES is only internally,
it should not be a problem.
The other thing is that RPKI, at least today,
does not protect from all of these types of attacks.
Like, currently it only ensures that this prefix is
originating from this AES.
But if there are multiple AESs in the path,
it's more difficult to get all of the properties you want.
So, it's still not up to par with IRR data.
But we're slowly moving that way.
And there is the Aspa standard.
It's recently been added to a routing nature.
I think to Creela as well, where you can also specify all of the valid paths.
And on the internet, getting a really legitimate AES path is very difficult,
but hopefully on your internal network it should be easier.
If you have, you know, from a visual drawing to, you know,
something more advanced like networks,
as long as you can export all the valid AES paths,
and you import them into this pipeline,
this will automatically get validated for you.
And your filters will take care of everything for you.
In the meantime, if you don't have Aspa,
it's not supported by your software.
Then what you can do is just have a list of route maps or prefixes
of all the AESs behind the BGP session,
and this should be good enough,
like depends on your network design, of course.
And the final problem that I found
was that if your RPKI valid data goes down,
and the table is removed from your router,
your whole network goes down.
And I discovered that in a very unfortunate situation,
where I prefer to be swimming instead of fixing that,
but I had to go back and fix it.
And in order for the valid data to get all of the information,
they need to reach certain IP addresses.
And if they can only reach them because they are valid,
then you have this cyclical dependency that breaks everything.
So some networks are very lucky.
They have 100% independent out of bond connectivity,
and you can run the RPKI there as a critical service,
but most of the networks, including mine, don't.
Like we don't have this privilege of having this option.
And what I did was that I host all of the RPKI stuff
that's necessary in a special prefix, an exception, if you will,
and this is okay to be unknown.
So when you're bootstrapping, where your router starts,
and it doesn't have any RPKI information,
it's okay for this one prefix to be unknown,
because you can bootstrapping everything off of that.
And since all of the connections use authentication,
at least in the application layer,
like TLS for Creal and SSH for Routineator
and RPKI client plus Sutter, this should not be a problem.
And I also have multiple instances that are closed to the routers.
I don't have one for my network.
You know, in Switzerland and then everyone has to connect
to this single instance, I have multiple VMs,
and sometimes they're even running on the same hardware as the router,
wherever this is something that's supported.
So I got away with this problem,
it's been working for three years in this architecture
without any problems.
And so one quick final slide,
about the long term goal,
eventually I would like to get to a point
where I don't have to trust any of my own devices.
Despite me being the person that owns them physically,
set them up and configure them,
I don't want to have to implicitly trust them.
And I think that it's very smart to move to a model
where you don't rely on this trust.
Because then you can verify things.
If you have RPKI, you can use all sorts of existing
BGP monitoring tools that will alert you.
So if you have routes, likes, hijacks,
you wouldn't even know about them before.
And now you do.
So that's it.
Thank you very much.
Thank you.