Ladies, gentlemen, the otherwise fabulous welcome.
As stated, we're going to be talking about a package journey through PF, so yes, yet another
firewall.
We're free BSE, we get that a lot.
Yes, another firewall.
Very briefly, who am I?
I'm Christoph.
You can get me on KPF3BSD.org.
I've been maintaining the FN3BSD since 2015, and yes, I do feel old when I say this.
Started out 10 years ago, looking at PF, it doesn't really handle IPv6 fragments, particularly
well, I bet I could fix that.
Those of you who have experienced with open source projects are groaning right now because
you can never ever escape this again.
So as I said, I've been doing that since 2015, since 2021, and that's why I brought
the hat.
I've been contracting for nitgate, and I've been maintaining PF in PF since then,
so thank you very much nitgate because I do like food.
Very briefly, this talk is based on 3BSE main as of about today.
There's been a little bit of refactoring work that has been stolen.
And I mean, imported from openBSD, they still have theirs, it's fine.
This talk is also inspired by, there we go, I've avoided the S-word, by a talk called
a package journey through the openBSD network stack by Alexander Bloom.
It's quite good talk.
It's on YouTube, it was presented at EuroBSD in Dublin, in September.
Here are the slides, we'd recommend it, especially because this talk is sort of a spiritual
successor.
Alexander talked about the network stack at large, I'm going to talk about the firewall,
but you kind of have to, yeah, but I don't want to be on the camera anyway, nobody wants
to see that.
So Alexander talked about the network stack at large, you kind of have to understand
how the network stack works, to understand how the firewall fits into it.
Unfortunately, I ran into trouble compressing a 50 minute talk into an introduction to
a 25 minute talk.
So there we go, there's already essence, everybody's seen this, everybody's on the
stood this, moving on.
No, so the main thing to keep in mind here is this, this is a very simplified version
of how stuff works, but essentially, packets arrive on the hardware, the device driver
does things to speak to the hardware, the liver's ethernet packets into the ethernet.
ether input function, yes, I have assumes that all networking is ethernet, it's okay,
everybody does it, because all networking is ethernet.
The ethernet function will demultiplex this, we'll go look at the packet on the side, you know,
what type of packet is this, this is an R packet, is it an IPv6 packet, is it an IPv4 packet,
but this is a legacy presentation, so this is IPv4, it really should have been IPv6.
You reason that input function goes into your IP input function, which depending on where the packet
is intended to go to, might call forwarding to happen, so you know, pass the packet onto to the
next router, or it might be an input packet, again, with demultiplexing, based on your IP protocol
field, is this a TCP packet, is it UDP, is it ICMP, is it God knows what else, and then we might
hand it to the socket layer for local processing, same thing on the output side, if you're
have a TCP socket, you will TCP output, you will IP output, ethernet output, hand it off to the
device, which will throw it on the network, and if you're very, very lucky your packet will
actually turn up somewhere else, main things to remember when you're hacking on network
stacks is the order of operations here is difference, when packets come in, you do ethernet
first, and then IP, and then TCP, when packets go out, you do TCP first, then IP, then ethernet,
or possibly you do, you know, ethernet IP, ethernet, when you're thwarting, so where does the
firewall fit into this, right there, so in 3bSD we have your choice of firewalls, you can
have PS, that's the one I work on, you can have IPFW, that's the one done by smart people,
and then there's also IPF, so because we've got these multiple firewalls, we just don't
have code in the kernel that goes if PS, then call PF inputs, if IPFW, do IPF check, IPF
W check or whatever, we've got an abstraction layer called NetPefill or PFO, and that hooks
in an appropriate location, so I've only really illustrated inputs and outputs on the IP layer
here, there are also hooks where you can do layer 3 filtering, there's even some very strange
things that NetPefillings do where they actually hook into the device driver, even lower,
there are layer 2 hooks here, I've not shown them here because PF does do ethernet filtering that
we're not going to talk about that here, P Concepts for PF is, it is a stateful firewall,
so states are important, you need to remember that those exist, and we are a stateful
firewall even for stateless protocols, so I'm sure we all know that UDP is a stateless protocol,
you send the UDP packet and as far as the protocol is concerned there is no other packets,
it's just a UDP packet, in practice however you like to do things like get a reply, so what
do we use UDP for basically only DNS, right? And when you send the DNS query, it is a UDP
packet that goes out to port 53 from whatever source port you've selected, but to be really useful,
you kind of want to get an answer to that packet, so there is a reply packet that is source
port 53 destination port whatever source port you picked, and then PF will recognize that actually
this UDP packet belongs to states that I've created for this UDP packet, so you can configure
your firewall and say, I will allow outbound connections to port 53, and you don't need to worry
about the reply, that is implicitly going to be allowed as part of that state, the next concept
of course is rules, your firewall is not particularly useful if you can't tell it what you want it to
so I like to describe firewalls as policy instruments, where you can decide that you know this is
the type of traffic I want to allow traffic to these addresses to these ports, these protocols,
whatever you want it to be, and that's defined in rules, so rules will end up creating states,
what happens when we hit the people inputs, these are the main functions that you need to
understand, we hit PF test, actually that's a lie, we hit PF check in or PF check six in or PF check
out, but those functions don't do much, they all just call PF test, so you end up in PF test,
first thing the PF test function does, and it does this for every single packet that PF deals with,
is it calls a PF set of P-desk, which parses the packet, it sets up a P-desk or a packet descriptor,
with a whole bunch of information about the packet, so all of stuff like what's the source IP address,
what's the destination IP address, what interface did it come in on, what's the protocol,
what's the source port, what's the destination port, all of those things get set up in a separate
descriptor, so we don't have to parse and reparse and reparse the packet, we do that once,
would it also does is what PF calls packet normalization, and the most important thing it does
in the normalization step is reassembly, because packet fragmentation is a thing, yes even in IPv6,
which is part of the reason that I have this much gray hair is IPv6, packet fragmentation,
but we have to reassemble the packet, because if we don't see the full packet, it is entirely
possible that we don't see the headers, so if we want to make a firewall decision on a packet,
quite typically we care about the layer 4 protocol, TCP, UDP, SCP, ICMP, and we want to say,
we want to allow TCP to port 4, 4, 3 for instance, we really need the TCP header to be able to make
that decision, so we reassemble the packet and we look at the fully reassembled packet until we're
all the way done with it, there's a special case for V6, which hopefully we'll get to,
typically we only really look at layer 3, the IPv6 header and layer 4, TCP UDP, if it does not
find state, this is a packet that is creating a new connection, so ideally it's TCP sin or it's
the first UDP packet you've seen, then we will evaluate the rules, which is PF test rule, it's
right through all of the rules that we have configured see if they match the packet and eventually
get to a point where we make a decision. Either way, eventually there's output handling and
there's only so many things that you can do to a packet, right? You can pass the packet, you can say,
yes, I approve of this packet, you can continue on your way on the lessons, we can drop the packet,
which is when we don't approve of, you know, the packet sense of style or whatever,
why ever we decided that it's not a good packet, we drop it on the floor and buy its goal.
We can route to, which is PF's policy routing, where you can say that, yeah, okay, my default routes
is that the packets need to go that way, but actually if it's this specific port and this specific
protocol, I would like you to send it that way instead. We also have F2, this is new,
well it's new in 3b as the, it's other family translation, it's not 6-4, it's doing unspeakable
things so that your IPv6 clients can talk to the IPv4 internet. This is very new, this is work sponsored by
NetGates and it's going to be in 3b as the 15 and the upcoming PF sense 2503. And then IPv6 is special
for your output handling because we may have to refragment. So we're familiar with fragmentation,
right, we have a large packet, we want to send 20,000 bytes of data, unfortunately Ethernet is
terrible and you can only send 1500 bytes of data in one go. So you chop the data up into fragments,
you shove all of the fragments through the network all by individually and then eventually you put
them all back together. As we've discussed, your firewall has to put them back together as well.
And for IPv4, this is simple, you've put the packet back together and then we just hand it to the
output codes to IP outputs and if the 20,000 bytes don't fit down that link, it'll just refragment
it for us and everything will be okay. IPv6 is fragmentation is different, so those of you who were
taught IPv6 might have been taught that there is no fragmentation in IPv6, that's not quite true,
there is no intermediate router fragmentation in IPv6. So in IPv4 every single hop can chop your packets
up into smaller pieces. In IPv6 they are not allowed to do this. If you get to a point where
your packet cannot fit down the link, you don't fragment it, you send the network message back
to the sender with, yeah, I'm sorry, but you send me 1500 bytes and I can only fit
1400 bytes down this link and then the sender will split his packet up into 1400 bytes chunks
and send them all. And then potentially discovered that the next router says, yeah, I can only
fit 1300 bytes down this and then an error message goes back and you try again with an even smaller one.
This is called path empty you discovery. If people don't walk I see MP, this actually sometimes
even works. But we can't break this in our firewall, so we've got this reassemble IPv6 packet
that's 20,000 bytes and we can't shove it down 1500 byte interface. So we have to refragment the
packet but we have to remember what size of fragments we got when they came in. And then we refragment
along the same size, actually for simplification what we do is we look at what is the largest
fragment that we've seen and we refragment basal that size. Now typically your implementation
is not going to be stupid and that is just going to match exactly the same packet sizes that
came in. But it means that we, because we respect this packet size, your path empty you discovery
is going to keep working. Which is kind of important. Even though fragmentation is not typically
a thing, all of the internet is TCP right and in TCP you make sure that a single frame, a single TCP
packet passes in whatever you want to do. It's a stream and TCP takes care of all of the reassembly.
Now there are some implications to this design. You may remember that the first thing we did
for a packet in PF test states protocol is we look up a state. Now this implies that it doesn't
matter what your rule set is if there is an existing state for a connection it will pass.
So as a PF user you should be aware of this that when you change the rules
even if your new rules that would mean that a specific connection is not allowed if the
connection was established before you set your rules the connection will continue. Sometimes you want this
sometimes you don't. If you don't want this you can flush existing states you can kill specific states
but be aware that this is a thing. As I said you know, block all might not actually block everything.
What it also implies is that the state look up is quite important because we do this for every
packet. How does this work? We keep a hash table of state so we have a very big hash table
and we hash states based on their source and destination IP address source and destination port.
Others family so that you know we don't mistake IPv4 and IPv6 connections for each other
and then whatever protocol it is. Based on these things we can find a specific hash row and then
every hash row has a little linked list that we can iterate through and go or you my state or you my state.
Oh your my state and now we can continue. You can configure how big this hash table is there's a
Cisco or tunable rather nettle pf.states on the score hash size. This is important because if you
have many more states than you have rows in your hash table you will end up with many states in
every single row and there are many hopes on the linked list that you need to walk through before
you find your packet. As a general guideline you know try to have as many rows in your state
hash table as you expect to have states. So if you're dimensioning your firewall for 10,000 states
or 100,000 or a million you want 10,000 or 100,000 a million rows you can be off you know by a factor
of two in either direction and it'll mostly be fine but if you're off by more than that it's going
to hurt. If you have too many rows you know you've based memory memory is cheap.
Probably almost always better off wasting memory rather than having enough rows but it is something
to keep in mind. There's actually a dethrase probe that is quite interesting so dethrase let's
you do dynamic probing you can you can know when certain events happen in the kernel in the middle of
a function we have some static probe points or defined in the codes that when you get here these are
the values that you can measure and one of them runs in the purge thread so that's a background
maintenance task in pf where it will clean up old states and potentially old rules and so it runs
through this entire state table every so often and for every row that it runs through it can tell you
you know I'm in row number 45 and actually I had three states in this row and I'm in row number 46 and
I have one state and I'm in 47 I have zero states so you can build a histogram you can monitor
how your utilization is there. Another thing to remember that is potentially interesting to know
is due to the way state limits are implemented actually hitting your state limit is quite expensive
so you don't want to be in a situation where you're constantly pushing up against that limit.
What else is important we have a control plane which is how you interact with pf either through
a utility pf cut all which you may all be familiar with or programmatically for the kernel it's
all the same thing there's an api towards the kernel and right now in pf there are three different
ways to do this this is because of historical reasons it started out with an octal interface
basically where you copy structures in and out of the kernel this is simple this is relatively
efficient and this is a pain in the everywhere when you want to make changes because you know you
modify the struct and all of your fields move around and use of space and kernel have a different
idea of what the struct layout is and nothing works so it becomes very difficult to make extensions
to teach pf new tricks which we've been doing so one of the ways to deal with this is actually what
they did in previous what they described in the previous talk for npf is nv list name value pairs
you have an abstract representation of what you're sending backwards and forwards into the kernel
a lot of the kernel and that means that when you add fields and one side or the other doesn't know
about this field just ignore it and everything works so this is good but there are performance
implications as discussed in the previous talk the experience in 3b as d was it worked fine
right up until we try to use it for state exports so you can get pf give you a list of all of the
states it knows about and you know if you have a hundred this is fine if you have a hundred thousand
or a million this is very much not fine not fine to the extent that I think we topped out at
maybe two million states and then we were copying four gigabytes of data between the kernel and
user space yeah that was painful painfully slow as well so the new shineiness is not link yes that
Linux thing I like to describe it as solving more of the problem than what I optal plus nv list does
it also has this notion of type length value encoding for your fields so you can add new field
types without confusing the other end of the connection even if they don't know about it but the
really nice thing is its efficiency it's essentially just as fast as the old I optal style but because
it can stream data out when you're exporting two million states get a peak memory use of 16 megabytes
you know trivial amounts of memory actually in use so it is both fast and memory efficient
and it gives us the flexibility of of extending interactions which again is really important if you
want to have new features because if you want to teach pf to do things like address translation
address family translation that's six four things you need a way to configure this you need additional
fields in your rule structure so you need to be able to make those changes my hope is to migrate to
net link for everything and then you just get rid of the old interface I point this out because if you
are writing an application to interact with pf there are multiple ways that you can do this right you
can just exec pf cuttle and pass information into its standard in or standard out which sometimes is
the right thing to do but I abore the notion of forking off other binaries to do work for you it's
just ugly and error handling is terrible but sometimes it's the right thing to do what you want to do
is there is a library called lib pf cuttle that makes abstraction of these things so it just gives
you some functions and some structures that you can fill out and interact with or get information
out of pf that way it's actually what pf uses it is also a work in progress so no everything is
implemented this way but every time I make changes or I convert stuff to net link they get a lib pf cuttle
function for this so if you're going to build new tool to interact with pf please please please look at this
or just exec pf cuttle do not try to interact with the actual interface yourself because sooner or later
I'm going to break it for you you've been warned you know we're in free bz we try to be very good about
about backwards compatibility and things like that but I'm not going to keep maintaining the
old ideal code once it's it's no longer relevant for for mainline pf it is going to go away at some point
I think we have some time for questions if you would like otherwise I have some spare slides and I can
talk about some other things was everything completely clear yes so the question is how close is
free bz pf to open bz pf and net bz pf so I'm not at all familiar with net bz pf my understanding is that
it is essentially on main time there but they're in mind that I'm not very familiar so I might
make completely wrong but I do see the net bz people nodding I saw some noise in net bz that they would
like to kick it out at some points but that has not happened right okay so pf in net bz is going to
go away soon so I can stop talking about that one open bz pf is the source of free bz pf that's where it
came from so they share a history the implementation is different so free bz pf has a couple of
features that open bz pf doesn't have and vice versa definitely the main things that pf gained in free
bz the art performance changes so open bz d still have they are in the process of removing it but
they still have essentially one big kernel lock or one big network stack lock so you get into pf there
is one lock and there is only one core that can work on pf at a time free bz pf and this is what work
I did this is work that glove smirnov did I believe years ago to make pf be able to use multiple
cores at the same time and essentially the way this works is you have a read right lock of the rules
so every time you come into pf and you need to do something with the rules you take a read lock
and you can have as many cores as you like with that read lock at a time and then when they need
to manipulate states they will lock the hash row of that state so as long as there's no other
states in that row or not too many other states in that row other cores can be operating on
other states at the same time so that's locking so you do get a fairly substantial performance
difference there the other one of the other differences is five minutes thank you
is vnet support it's the the virtual IP stack thing for jails that we have in free bz pf is
aware of this so you can have a pf per jail which is a really really really useful feature to have
both because it gives you flexibility in your jails but also because I use it for testing
so I can spin up two jails set a pf rule set in one of them throw packets from one jail at the
other jail and see what happens so a whole bunch of automated tests in free bz d for pf thanks to
that feature they also recently added a ctp support to pf that's also only in free bz dp f
other than that there's a whole bunch of refactoring work that is in open bz dp f that is
multi in free bz pf although a lot of it has been moved over already so when I said that
there is only one pf you enter pf you get into pf test you used to be pf test and pf test six
those have now been unified in free bz d main and that's something that was imported from open
bz d so I'm in the process of importing patches from open bz d minor bug fixes small new features
but there's still a lot of stuff in open bz d that's small in free bz d yet some of it might never
get into free bz d uh the open bz d people take a little bit more of a liberal attitudes towards
backwards compatibility so they will just change the syntax and then it'll break you know existing rules
that will break and there are good arguments for doing things that way but there are also good
arguments for backwards compatibility and that's kind of a philosophical difference there
next question can you can you comment on the difference between free bz dp f and mp f because
both are kind of as you said working on making the multi core story yeah so can I comment on the
difference the design difference between pf and bz dn pf unfortunately not really I'm not very familiar
with the n pf internals so I can't really tell you I can't say anything meaningful about the
n pf design I just don't know what it is I work your ability really powerful to use the
base tile well that would have to invest so if question is if I want to build a powerful
pf based fire wall what do I invest in so now because my customer is here my obvious answer is
you should just give them money but what you care about well in part you want to fasten it
work cards you know we can we can talk about one gig a bit network cards it doesn't matter
what firewall you use they'll all do it just fine so you want to 1040 100 gig network cards
you'll want multiple cores because pf is actually able to make use of that to some extent you know
there is still contention internal to it but it can do this you want some memory but really it's not
that's insanely much memory if you're buying you know 32 64 gigs of memory it's almost certainly
going to be plenty I don't know the numbers off the top of my head but you can go a long way with 32 gigs of
memory so be on that you know fast individual cores but a sizable number of cores last time I
ran performance numbers I think pf topped out on my hardware in my specific setup something out
in the only order of 18 million packets per second thank you all very much for your attention
and your time but I'm going to be quick out now I'm going to be down at the freebie as
the stand in k so if you have further questions feel free to find me there