All right, so people can grab a seat, we're going to get started again and for the next
talk, we have David talking about an XDPE virtual server to try an EVPF load balance
of library, so there you go.
Hi, I'm David, I'm going to show you how you can build a high capacity load balancing
application with XDPE and go, I hope to go as well, a bit of stress because the cast
sitter locked herself outside the house and my partner had to dash home, so I'm a little
bit on edge, so the global radio is UK's largest commercial broadcaster and I haven't
some time ago running legacy hardware load balance appliances handling all of our traffic
and they've reached end of life, we host websites, live audio streaming, which is the
book of the traffic audio on demand and some video, look over 30 gig a day, gig a
bit a day, join the day, peak, and there's a mix of these high bandwidth media services,
which are public and low bandwidth business services, so stuff we do with partners, uploading
ad copy or whatever, but we're encountering some a federal port exchange bug for net services,
which affected both sets of services because we have to reboot the appliances occasionally,
we delay it, but still be kicking 100,000 listeners off of the effects it, which, you
know, it's not great.
So I thought we could move the bulk of the traffic to a simpler high capacity low-banzer,
which would leave less work for a more sophisticated, featureful solution, TLS offload and
WAF and that sort of thing, and should create a more resilient platform, wouldn't necessarily
see the same bug, but why mix the different classes of services, media apps are ice-casts
we use are really quite simple, it's IPv4, layer 4, oh balancing only, it's just TCP,
UDP, no HTTP headers or anything like that to take into account, and it's a direct server
return at layer 2, so what is direct server return on here is also known as direct routing
or F5 calls N path, basically the packet gets sent by the client, comes in through your
firewall router, hits the low-banzer, low-banzer just updates the macadresses in the ethernet
header to set itself as the source and the backend that it selected as the destination,
forwards it on to the backend, which can see that the IP address, which has been unchanged,
here, we're going to want to update 100.1, it's got that on its loopback address, so it
is supposed to it, replies directly to the client, so you've got this asynchronous route
for your packet, so the packet is unchanged, pop from decranting TTL, there's no port mapping
you can do, you can translate port 80 to 88 or anything like that, all backends must share
the same VLAN with the low-banzer because it's layer 2, and virtual addresses must be on
the backend, so no federal ports allocated or anything like that, basically you don't allocate
really anything, so you have millions of connections through a single low-banzer, and there's
no state necessary to keep, so you can fill over to different low-banzers and the same
ways you can do with a router, essentially a router, we want to make sure that the same
backend is chosen after a change to the pool of the backend, let's skip over this
really, just for brevity, but essentially the packet can take any route to get to the backend,
let us have a low-banzer in this case, and we should be able to fail over between these
and will without breaking connections, so consider a few things, Google Maglain, it's
some great blog pieces, and it's a really good consistent hashing algorithm for backend service
selection, but it's not open source, so I'm going with that, same thing for a cloud-flare
unimog, not open source, just count that, get a low-banzer look interesting, but to fully
utilize it, you need to run a kernel module on the backends, we don't want to change anything,
so yeah, that seemed a bit of a stretch, and it used to be DPDK, I don't know if that's
changed now, but the main thing was layer 3, it uses layer 3 tunneling rather than layer 2 for
selecting the backends, so essentially encapsulating the packet in another IP layer, so we didn't
want to do that, so you have to change all the services, which use it, it's really want to do a
like for like swap, and that looked a bit more promising, but again it's layer 3 only, and also
it's C++, and that's really a library, and I don't do any C++, so I do that, there's got like
PVS as well traditional ones, but I can find a way that you can get away without configuring
a bit locally on the low-banzer itself, which, potentially, resulting exposing services like SSH,
so yeah, and also issues with accurately doing health checks against the vips on the backends
service, so I thought, want to try building something with XDP and EBPF, yeah, Director of
Server Return is straight forward to implement updating macadresses, Catran, shows that it's
possible to do something like this, we just need a bit to be configuring on the backends,
loop back into face, that's already done, so no change there, and yeah, we just have
the basic macadresses, which should be easy to do, how hard could it be, turns out, not very hard to
solve, is a program, EBPF, XDPF program, which will do low balancing on a single bit for all traffic,
define the bit at the top, produce the list of your backends, there, got to match the packet,
modify the ethernet header, and then forward it on its way, and should be able to do like
tangy line rates, no problem with this, we've just just a couple of calls and take a couple of
tens of nanoseconds maybe to handle each packet, obviously this is inflexible, you need to
recompile it and reload it to change backends, there's no health checking, so you end up back
black polling a lot of traffic, if the backend is down, backend selection is poor, it's just
you know what's used as suitable hash algorithm instead, something consistent hashing algorithm,
which maximizes the chance of hitting the same backend after a pool change, and like traditional
hash table modular table size like this, which breaks the majority of connections if anything
changes, so we need a way of making this flexible, go with the EBPF maps, I want to store
a service and destination info, bits, backend IP addresses, backend macadresses, that kind of thing,
we want to map VLAN IDs to interface indexes, so that we can use BPF redirect map,
and with the deal we either native interfaces or VLAN trunks, back in selection, I want to choose
a destination and track the flow to it, preserve connections when the pool changes,
then use the consistent hashing algorithm, I want to keep account of packets,
an opportunity and any errors to each bit, service backend monitoring of various reasons,
we want to push packets to use a space with a queue for an hour of sizzle relaying or
flow state sharing, I sort of need to actually in the kernel, the EBPF code, what about user space,
so we need some control plane functions to add or remove services and backends and enable
to disable them or change their parameters, we want to be able to send a ping to trigger art discovery
and then read macadresses from the product map, we can discover local interfaces,
F index numbers for them, hardware macadresses, and local IP addresses, anything else you might
need to know like bonding and trunken, we need to ensure that each bit is present on backends
to avoid black holding packets, it's not enough to know that a web server is running it needs to
be listening on the VIP that you're using, so we're at the S library, it's borrowers the interface
from cloud flares, IPvs, go library, so to reinforce that, to, yes, you iterate over the results of
the services and destinations, functions and you can ensure they match the desired state
if something's missing, you create it if something is there, you don't want, you can remove it
and you can update to enable disabled backends and other things, I add some more functions to
help with health checks and flow at the state sharing, there's an example program, you've got here,
yeah just pulling the library and reading with the command line arguments, create an instance of
a client and create an instance of a service, in this case it's port 80 protocol TCP, so that's
HTTP server, and then add backends to the service, that's it, requires the BPS dev for linked
things that used to see go, my school to the CBPF API, that's enough to set up a service and
distribute traffic, but what about health checks? So state previously, the server needs to respond
to a VIP, not just its own IP address, we send traffic for a VIP to backend but the VIP is not
configured, then you'll pack it back and we'll be backholds, we need a way of probing the VIP,
we create a map of virtual IP and backends, real IP addresses, to enact address, we'll send
a packet to say 25502 via a special VEP interface that we create and we can have HTTP map the
packet and forward it to MAC address of the real server, 10124, but with the VIP address of 1212,
what's it say, 100.1, similarly for 2450, so it's a three, sends it to the same server, but we'll send it
to the 100.2 VIP address. So yeah, we need to create an interface with a route to the
NAT addresses, I'll tell you that NAT address for a virtual IP and real IP pair, use a
client to dot NAT function, we can then use standard networking stack, EG goes network, net
HTTP library to make requests, make a get request, however the host name of the service
will be set to the NAT IP address, so if you want to make requests that will rely on the host
name being set correctly, virtual hosting or TLS, so the name indication, then we can override
goes HTTP transport dial context, which will always return the desired NAT IP address in the
amount, what the URL you give is, and each backend and each service gets its own client,
it gets its own HTTP client. So minimum viable product for a load balancer, it's got a health
checks for each individual service, I've got to advertise VIPs to neighbouring routers for the
BGP, simplest protocol to implement, known as health injection, I don't know if the surface
metrics and stats, EG with previous, do a bit of logging locally, and maybe with some
logistics search, and also perhaps some kind of a learning via teams or Slack web hooks for
VIP failures, flowchowing would be nice to support for preserving a long-running connection
when failing over between load balancers in the face of backend server pool changes,
you don't know where that connection was going to, I need some way to configure the service,
JSON, yeah, or something. So there's an app for that, this Vc5 application, which is written using the
XVS library, it's actually proxy influences you can probably tell, health checks were indicated
as you'd expect, VIPs services back ends and green network thorough K, read if they're failing
and grave, they're administratively disabled, it speaks BGP, the top right there, there is a talking
to the local bird instance in this case, but you can have any number of routers speak to you,
metrics, you can see the stats are summarized in the tables, and there's a metrics link at the top,
which is a link to a previous endpoint. As a way, box in the center of the screen for logging,
and this is much more detail, can also be sent to a elastic search, and at the top, this status link
is a JSON endpoint, in which the view is built so you can change it, just all done via via JavaScript.
If you need to modify configuration, it's a very YAML file, it's very compact,
should cover 95% of cases in this fairly trivial manner, and there's a config PL script,
which expands the YAML to a much more explicit JSON format, so you can see all the health checks are
explicitly defined in there, there's only one of those two services shown because it's a much more
of the both, but if we need more control, we can define custom checks under the policy points there.
Standard ports can be overwritten, HTTP here is short for 80 slash HTTP, so you could use 80, 80,
slash HTTP if you want a different port, and if you're not keen on the YAML, then you could write
to your own script process, if something any format or time or use a scripting language,
but I'll draw in DSL, whatever. I'll run the YAML's JSON stage before deploying this with
them with Ansible to catch any syntax errors, and it's all therefore put into a Git repository,
it's pretty infrastructure as code, and set up. Yes, starting deploying it in 2024, run for a year on
some VMs, running small services, test services, obviously a restatic website, that kind of thing,
once we're confident, we've started running in ANGO on tier 1 services, but it's not a January,
it's sort of 2025. It runs on pretty modest servers, yeah, 32 core 32 gig, they're not nothing big,
pretty inexpensive, and one server can handle a full production load in low single digit CPU,
partly because the DSR, you're only getting acts back mainly from the clients,
the bulk of the data going out can be up 20 or 30 times the ingress load. Runs one, some intel,
called X-710s, with VLAN Trunks running over AMLAG, so aggregated interface server,
carous switches for resilience, and we create a new VPS report to DNS to migrate it,
both some services, we could just cut over one shutdown at one of our co-loads and bring it up again,
and then flip over to the other side, it's all quite simple. This is a week in April last year,
so the Easter Brakes are slightly lower traffic than normal, you can see where upgrades took place,
and the orange area gives way to blue, and the brown area gives way to green, possibly a
colorstone commitment, one on there, and that's where we failed over between low balances,
when we upgraded them, we looked at the spreads and they lost a traffic, so it was nice,
large spikes there are related to some very popular podcasts that we host,
I think they're automated, one of our mobile apps, so they're in some kind of self-DOS attempt.
Sorry, even distribution traffic at the backends, but I'm left here, so yeah, that's the
hatching algorithm working nicely. Give me an idea, remember this old XKCD strip,
map of the internet, so I held it space-filling curve, and maps a linear address into a 2D
fractal, essentially. I've got to be in the pressing way of visualizing the traffic now that I'm
seeing all that, so you know, trapped prefixes from across the net, each pixel here is a slash
20 prefix, most traffic from European IPs, quite in Brake, as you would expect, as being European,
and the large white patches, when there's lots of traffic, generally they look like the UK ISPs,
some traffic from USA, our in addresses, but some of those might be reallocated prefixes,
I don't know. I'd imagine that if we were getting a DDOS, you'd see a much more evering covering
of points, so I thought we used to visualize that, and you can use these stats that you report
over time to calculate the probability of a packet, being legitimate, in the case of rubbing,
DDOS, and got with their sort of commitment from below the 50th, 95th percentile of regular traffic,
because I don't know you do the same for IPv6 though. Once this is a working, I've also now
I did lathe 3 and IPv6 support, so lathe 3 is a towing traffic over IP, encapsulated, it uses
well supports the IPNIP, generic routing encapsulation, fill over UDP, and generic UDP and encapsulation
standards. IPv4 IPv6 traffic addresses, maybe used for VIP or real IP addresses, and you can have any
combination of those, so you can have an IPv6 address backed up by IPv4 backends or vice versa,
any combination. You tell the library what gateway to use, because it doesn't use the standard
Linux routing table, but that's about it, and it's a lot simpler to only support lathe 3,
than having multiple VLANs with layer 2. We have a few issues, the Ex710 had problems getting
media negotiations to work at the client. A colleague did look at some system D parameters that
seemed to help, although I think it was mostly fixed by just doing kernel upgrade. I tried the
native driver modes for the VMAX net 3, VMware driver, but there was a bug in that which had
the wrong point for the data end, but again, I couldn't come up, fixed that, so it was all
easy enough. Adding the lathe 3 support increased the size of the program a lot, so
verify wasn't very happy with me, so I had to use tail calls for some of the passing packets there.
I can't get a termination of food or GUE over IPv6, I don't see a way to set that, anyone knows
that we're amazing. Future development, the config, which I say currently stored in get in
YAML, can be centralized in a database, or with a nice UI on it, the surface of the JSON,
and URL end point, and have the lowest balance hold it, to make it a bit more self-service,
teams could be given BGPPS to speak to, and a bit range, and they could manage the
own load balances, just a demon you run, we could use service discoveries, so have back ends
registered to something like Zookeeper or console, when they're ready to serve traffic, and then
the load balances could discover them automatically, so you don't need to update it.
An EBPPS futility to terminate some of the layer three protocols in white the handy,
you can currently require IP tables for my warning stuff to make sure it only goes to the vets
and you're doing all of that with this post and an EBPPS program, so that would be useful,
and that would also solve the problem that I can get food and GUE to work with IPv6.
Yeah, and do stuff at our own pace now, we're not worried about switching vendors and
migrating all the config again, or any extra licenses if we want to deploy more instances,
and popping the cloud flat IPv6 interface to now to be quite good, as you can basically adapt it
to use IPv6 instead of XDP for some net-based load balancing.
So quick demo, which I think will work.
All right, so there's balance of program, it's in go, simple, I'll look at it.
Yeah, like before, essentially you pull in the library and pass your manual in arguments,
create a new client, create a service on that client, what IPTCP again,
then for each of the backends you've got for the for it, you create the destinations,
and there it's using GRE, so that's a layer three service,
from layer three backends rather, loop for 60 seconds and display stance and then clean up,
it's very easy, just remove each service that's returned from the services function.
Yeah, so every second we print out some global stance, print out the stats from the vip's,
then for each service and each backend in each service, we get the net address for that backend,
make a query to the net backend and pen on the result, we can either enable or disable backend
and update it so it gets to bring back to the kernel and that's it.
Pull down the library and then again, that's all done, and then yes, we just touch
run with the interface name, the v-land description that we're on, the vip address, we're going to use
and then list the backends, let me send that going, now it's immediately that one of the backends
isn't responding, this because it hasn't got the vip set on it yet, so if we send traffic to that,
well I'd see it just goes to the backend one, and if we then add the vip to the backend,
we should see that most immediately it comes up and it's healthy and then if we send more traffic
it will go to both backends, so it's being balanced, check a bit more traffic kind of,
yes, so I'm this will handle huge amounts of traffic, I don't want to conserve all
handle potentially 100 gig backend services because of the asymmetric nature of DSR,
and that's it,
so links to the programs and stuff back to the movie and questions, thank you, that was a really great
talk, just a quick question, basically around high availability, particularly around two points,
I think you've kind of touched a little bit better, one, basically if you're compiling a new program,
is that going to disrupt things when you push it, also the if you have two little balancers,
what your kind of strategies there, so yeah, I think one of the service for maintenance and so on,
for there's nothing particularly fancy about loading a new new EVP F code, that would disrupt it,
but because it works in a equal cost multi path kind of set up and there's a
state table sharing, then failing over from one to another, it's not probably, you know,
pretty much all the connections of maintenance, so you just shut down one and restart it with the new code,
thank you,
great talk, we also run into the BMX net 3 buck, I think that's your, yeah, I was wondering,
you mentioned you are using a map for the gateway lookup, is there any reason we didn't use the
kernel for the lookup? Yeah, for the gateway lookup, before I just did see it at the time,
I think don't necessarily want to use the routing table, you know, so there's just have a default
route on one white administrative interface and then the actual services are done on completely separate
and the lands which don't have a route to go, if I understood the question correctly, I'm not sure,
yeah, it's fine, I'll thanks for your talk,
I would do you maintain the flows they're sharing across all the level for load balancer,
for example, if one goes down or if one of the server or for service goes down?
It's done relatively lazily because of the, it's just an hashing algorithm, then everything
kind of sees the same stuff and, you know, generally you don't break connections if a back-end
server goes down, it's more for long-lived connections, so just of every 30 seconds or so,
we can kind of update the flow, multi-cast it out to the other load balancers and they can
write the flows to them. Okay, and for a vain encapsulation, do you use a do you enable
jumbo frames, something at work, for a vain crazy size of a packet? Not at the moment, it will handle
the send-back and I see MP in the packet too big, kind of responsive if it needs fragmentation,
so we're not really switched off two layer three yet, so that will be a kind of ongoing
thing and say how it works, but yeah, should be okay. Thank you.
Right, so as you mentioned already some load balancers that exist
different projects and yours is right where we wish I think is great because that means
everybody can basically reuse it. So my question would be, do you have that, do you have actually
external people contributing or using your project for now? I think a couple of people do,
I've only seen a couple of she's reported, so some people do use it, I don't, I don't,
I can't say for sure who does that, I don't know. Right, that's a good start.
After first step, maybe, maybe. Yeah, if you do use it, let me know,
try not to be as cautious on making changes to the code and breaking stuff, but
good. Any other questions?
I think we're good. Thank you so much. Thank you.