I was actually started at 35, because that's one of the slides, but I don't want to keep you
waiting, so we decided maybe I'll cut off and edit the video later.
Mine is actually a very short lightning talk today.
Luca was on stage earlier talking about packetfest and ziracanomic too much advertisement for it,
and you should come there if you like this type of thing.
There will be a longer session talking about S flow and vector packet processing there too.
In case you notice there's four talks back to back on a VPP today.
I'm talking number three or four after me or two gentlemen who also have excellent stuff
to share.
So I'm going to keep the boiler plate of VPP stuff to minimum today.
My name is Pam. I work at a small company called IPNG networks.
I've been in right community for a while, and IPNG started in the pandemic because I was kind of bored,
and we work on a DPDK and a VPP applications.
We run a ring in Europe that is the number three, I think, best connected ISP in Switzerland,
which I thought was always kind of funny because it's kind of a basement ISP.
I wanted to talk about S flow today. It's Anne Method for collecting traffic in Switzerland,
routed networks that uses statistical sampling, not flow probing for the entire flow.
Originally, RFC 317 has been superseded now currently a version 5 on the S flow.org website.
What it kind of does is it uses data plane typically A6, right, to copy forward one in end packets,
where N is a large number, like 1,000 or 10,000 or a million.
And it copies the first couple of bytes from those packets into a little buffer.
Adds things like ingress and egress interfaces, macadresses, sampling parameters, all that type of stuff.
And if you want also extra metadata like which ASPath, which AS number,
which types of things do we know about this packet as we see it in transit sampling it.
It also periodically reads interface counters, like how many multi-casts and unicast packets.
Did we see how many disk cards did we see and so on.
And then it stuffs these all together sort of in data grams,
and mostly UDP data grams, and forwards these batches of samples plus the stats like the counters
into a central place called a collector.
And because a lot of this heavy lifting is done by the ASIC to do the copying and so on,
it's easy to get tens of thousands really of these agents in the field like switches and routers
to talk to one or two central collectors, right?
That can then get a holistic view over where the network is at.
So VPP, you know, we talked about it a little bit, it's an open source data plane,
similar to Grout, that provides super, super fast networking using a DPDK,
VMA, Virtio, VMXNet, if you're on VMware, AVFA, if packet, all sorts of inputs really can easily
do hundreds of millions, if not a billion packets per second, ask my buddy Vlad about that later.
And we've seen on GCP as well, 100 gigabits is in the cards, like for a single VM.
And it runs on commodity hardware, there's no binary blobs and things like that,
so you can just download the packages from FIDO and install them on DBN or even Red Hat if you'd like.
So line up today, like we talked about FIDO and Mohamed talked earlier about the GCP story,
not on about MagLev, I'm going to talk a little bit about Sflow and Varunov and Gauthor going to talk about TLS after me.
So I just wanted to really quickly go back into that graph and talk about it a little bit.
So a packet enters in VPP, a directed A cyclic graph, starting from an input node,
for example, DPDK, which we talked about before, or RDMA, or Vertio, these are all in gray.
And then it goes on to layer two, all these packets go into these nodes, so I'd all the Ethernet packets anyway.
And then we sort what type of Ethernet packet is this, this is IPv4, IPv6, and ended off to sort of vertical stacks,
if you will, of graph nodes for IPv4, IPv6, MPLS, are that type of stuff.
And then it ends hopefully, like in an output node, typically we figured out what to do with the packet,
we're going to route it to this other destination, so we'll bring it back to Ethernet output,
and then select an interface, put a MAC address in the Ethernet frame, and forward the thing on its merry way.
That would be an output node in green, but then we also have drops.
And actually we had an interesting question about where did that 0.004, you know, percent of drops come from.
I would say, do not underestimate the importance of monitoring drops in your network.
Like people talk about what made it through all the time, but we typically don't really know what happened when the packet died.
Was it because of an app all, was it because of input congestion, like whatever we got,
descheduled for a microsecond on the kernel, is it output congestion, because there was no TXQ in the hardware,
that was willing to accept our packet, like we don't know, so it's actually really important to get visibility,
and Sflow would be able to do that with a little asterisk.
So what we've now done is we take that graph in the middle, and we try to insert a new node that takes all these packets,
does something with them, and then typically moves them back to the next graph, right?
So we insert this Sflow node into what we call the input arc.
So that means every time a device had a set of packets that it was willing to give to VPP, we took them,
and we looked at them, and we sampled maybe 1 and n, and then we moved them into,
ostensibly, the next node, which would be Ethernet input in the vast majority of cases.
And we're planning to do this as well on the green path here, the output node,
so we can do egress sampling, that's actually pretty straightforward.
And a little bit more tricky, at least in VPP, is drop monitoring because there is no easy mechanism for us
to insert this node into that part of the graph that drops packets.
So I'd like to talk to the VPP devs about that one a little bit.
So then when you do this, that bubble that I had that I inserted into the graph is called an Sflow Worker node.
And really what it does, it takes all the packets from the input and just moves them back to the output,
which is kind of a little bit wasteful in a way.
But one in n of these, we can copy, and what we do is we create a set of fee foes that are lock free,
and multi-threaded, and wait free, and we just put the packet in there.
And if the fee foes full, we'll drop them.
That's one way for us to make sure we don't overload the rest of the system.
But typically, there's enough space in these first and first out buffers for us in the worker
to just add to the top of it, and then in the bottom side, we'll consume them somewhere else.
So it's actually nice because this fee foe allows us to really just do what we do well in the data plane,
which is shuffle packets around, and make all the rest of them more complicated stuff someone else is probably.
And after messing around with it a little bit, Neil McKee from Inmond, who did most of the implementation,
got us down to nine CPU cycles for every packet we touch.
And about 17 cycles, if we decide that we actually want to sample it.
So grabbing all the extra information and putting it in the fee foe, and so on.
And just for reference, end-to-end, on at least this machine that I was testing on,
layer two cross-connect.
Ethan, at any extent, it out is about 144 CPU cycles.
So we're adding nine and a half or ten to that.
It's not trivial.
But IPB4 is about two hundred and eleven, and PLS is about two hundred and nineteen when we do the load test.
So it's not that terrible to add ten cycles, but obviously, it tends cycles is ten cycles, right,
that you could otherwise spend on doing something else.
Anyway, we put all this stuff in the fee foe, and we move on so that the data plane workers can stay fast.
And then every now and again, we wake up this other thread in Maine that says, hey, is there anything in these fee foes
that we should be aware of?
And we just drain those fee foes, right?
And what it does, this Maine task, is it grabs these packet counters every now and again,
every ten seconds or so.
And it grabs all of these samples from the fee foes.
And it puts them in a Linux kernel construct called a P sample channel, which is a net link library,
and it just gives them to the kernel.
And then whoever would like to subscribe to those things can get them out the other side.
This is another handoff point, right?
So Maine in VP is grabbing all the stuff from the fee foes and the counters,
and every now and again, a meeting P sample messages into the kernel.
That's the arrow in dark gray at the bottom there.
And then there's a third component to this architecture called the host S flow demon,
which is already existing code that most everyone in the S flow world will have come across.
And it then subscribes to net link and gets all these packets back out.
The samples and so on, puts them into a UDP packet and sends them to those collectors in the middle.
Right?
So that's the multi stage.
The data planes they super fast, the main thread just collates from the data plane,
sends them to net link, and then S flow D will copy them to your collector.
All right, so how do you configure this?
I have this lab set up that I wanted to show.
It's one machine called hungry hungry hippo because it always likes packets.
And then a T-Rex load tester from Cisco both running on a rather old Dell R730.
Then I have two loops.
One at the top really is a layer two cross connect that is S flow enabled in VPP.
We're sampling here, but also an IPv4 and MPLS that is enabled and sampling here.
And then in green and red down below the exact same thing, but without S flow.
So we can see what is the impact on performance.
I'm not going to go over the entire config, but if you were to run this on your own machines,
download VPP, run it in Docker as we saw before.
You could type these types of things and you would bring up that IPv4 or MPLS,
giving a label in a label number 16, pushing it out to this other interface with label 17 imposed
that would be an MPLS P router in principle.
Or a layer two cross connect is really just easing that frames in,
copy directly to the other interface out.
It's the cheapest thing that we maybe could do.
To turn on S flow, there's a couple of things that are defaults and I put them here.
The sampling rate would be 1 in 10,000 packets.
The polling intervals, I think, 20 seconds by default, but I said, pull the interface.
That's every five seconds.
And then copy 128 bytes from each of the Ethan frames you do sample into the P sample.
And turn it on on these four interfaces up top there, right?
And that means that four interfaces down below have S flow disabled.
And then host S flow D is pretty simple.
There's a collector where we're sending our samples to.
And then we subscribe to P sample group equals 1.
And we enable the VPP module, which is in the HS flow D,
which knows how to sort of take that stuff from P sample.
We restart HS flow D.
And pretty immediately we see all these samples coming in.
And if we want to grab for the counters, we also see the counters every now and again, come by.
So operators notes, if you were to start using this stuff.
VPP often has a data plane network namespace where it puts its interfaces.
And collectors can be created either in the default namespace or in the namespace with the namespace option.
And for Linux control plane, a very powerful feature in VPP,
we will by default use Linux control plane interface IDs from the Linux side to sort of be more seamless with respect to our pullers.
But if LCP is not loaded at all or there exists no interface pair for this thing.
Or we literally tell HS flow D do not use them.
Then we will use the VPP representation of the interface.
If you're curious as to why this is a big topic, come see me later.
So Act 3 is about performance.
That's also the last thing that I'll talk about.
I'm taking one of these 2012 deals that has like 88 CPUs or something.
It's not insanely expensive or powerful to run up the mill thing.
And then I turned on a low test here with these pairs that I'll show port zero is sending to port one and port one is sending to port zero.
And they're doing IPv4 and MPLS with Sflow turned on.
And then the layer to cross connect, which is extensively cheaper, I guess.
And then IPv4 and MPLS with Sflow turned off and finally layer to cross connect also without Sflow.
And kind of nice when you do 80 gigabits or 47 million packs per second.
There's absolutely no difference in performance with Sflow turned on or off.
They're both doing 40 gigs of L1.
They're both doing 23.6 million packs per second.
Except the left hand side with Sflow turned on is also emitting 1 in 10,000 packets.
This is easily 22,23,000 packets per second that we're sampling.
So I was curious to see what is actually the limit of this thing.
So this table here shows first if you take an interface of 10 gigabit one with 64 byte packets,
which is the smallest we're allowed to send.
You'll get 14.88 million packets per second.
And if I do that with IPv4 turned on, VPP does a lot more work.
Of course, and it'll do about 10.8, 10.9.
And with MP left turned on, it's 10.1.
That's just baseline.
And then as soon as you turn on any sampling whatsoever,
we insert this node in the graph, remember?
And what it's doing is it's copying all the packets to the next node again,
which takes 10 CPU cycles.
So there's an immediate regression when you turn this thing on at all.
So we go from 14.88 to 14.3 million packets per second when it is turned on
with a synthetically large sampling number, like 1 in 1 million or so.
But then it's not actually too bad.
If you ratchet it up and go to 1 to 10,000, 1 to 1,000, even 1 to 100,
which your vendor would not recommend you do on hardware most of the time.
The further regression is not that bad.
So for layer 2 cross-connect, we're going from 14.3 to 14.15 million packets per second,
while sampling 1 in 100 of them.
And that's where we also see that we start dropping these samples,
selectively, and by choice, in the data plane, because we will be overloading the rest of the system.
And so by design, we have this FIFA that is limited size, and when it gets over full,
we just drop them from the tail.
And that's what you see here in that last column, down below 1.8 million dropped samples.
If you build a larger FIFA, you would get more throughput, I think.
But it's really nice to see that this quad loop overhead that we saw before,
really is just in moving the packets around, which we can do in 9 CPU cycles per,
which is pretty great if you think about it, but it's still an overhead.
And I don't want to show some interoperability slides.
Again, this may be how you might show this stuff on the other side.
Inman has a tool called S-flow RT, where you can just point S-flow agents app,
and here's a screenshot from that.
I wish of Aquarado, that's one of my personal favorites,
actually fabulous piece of software.
But it could be that if you're here, I would like to talk to you about counter samples and EF names
in the Aquarado inlet.
We also use end-top NG with many thanks to Luca,
who personally went and added the S-flow collector to the open source version of this
as we talked about this project.
And you can see all that graphics really nicely line up in end-top NG as well.
Thanks to Vlad for this beautiful picture.
These are like really big machines converting like two kilo lots of power into heat.
After they forwarded a terribut plus of traffic through the middle machine there on one VPP instance.
Okay, thank you.
Sorry, there is not time for questions, so we move to the next.
Thank you.