All right, we're ready for the next talk.
We've got Kean Butler who's going to talk to us about the building performance critical
Python tools with Rust.
So thank you.
So yeah, so I'm here to talk about Rust in Python.
We add Clouds Miss.
We are a package management company.
So we download lots of bytes.
Everything goes through our core monolith, which is a Django app, which works for a lot
of the cases.
But we process 100, 110 million API requests a day, give or take a weekend or off, which
equates to several petabytes of requests.
When we don't have a cache hiss on our request, we have to send that back to our
monolith.
So that basically means everything goes through a Python application, which is slow, because
our Python application was written 10 years ago before Python had very fancy asing stuff.
It starts to show how slow it is over time.
So to scale our Python application, we have to take some interesting design choices and
we started putting services in front of us.
So we use and we use internet for some basic validation.
We use H.A. proxy for some queuing and some rooting of stuff.
And then we use U.F.D. for actually sending it off to our Django app.
As you can imagine, it's pretty hellish to look up a request if you're using something
like tracing and say, where was this request slow, what was it doing?
We also found that U.S.E. was basically a black box for us.
There was no data coming in our out of S. We couldn't tell why a request was blocked in
there.
What was it doing or if there was just anything wrong with us.
We also, we needed to look at and then yeah, so we just started throwing hardware out
of it and saying, okay, we'll need to just add more nodes, more replicas and more requests.
This unsurprisingly gets expensive and money isn't infinite as long as we're that one.
So we wanted to look at how could we make this faster.
And what could we like do to help out?
Of course, there's no chance we're ever going to do a full rewrite of our code base.
It's 10-year-old.
It's custom business logic and makes what protocol logic for 36 different package formats.
And I would like to say we understand this, but it's 10 years old.
We have cycle engineers over the last 10 years.
So there are edge parts of our code that people, I don't really know why we do.
We set that head up.
But we set it for a very good reason because it suits you unsatisf, everything explodes.
So we were looking for basically easy wins for performance without having to guess our
all of our engineers to learn a new language or restart everything from scratch.
And as I said, we're processing head-up by the day-to-day with 100,000 requests for
very large companies.
So we have to be reliable when we're actually doing this roll-out.
We can't just throw code at the wall and hope that it improves stuff.
I say that.
We did do that a time, so yeah.
So we kind of wanted to approach it with a good methodology of like, we'll measure
it, we'll identify the bottlenecks and look at what's out of there that we can start
to like, speed things up.
We're not trying to just pull in rust for the sake of pulling in rust because I think
it was said in the clickhouse talk, people like Russford and it's cool and I'm sure all
of you like Russ because you're in a room called the Russ Devroom, but we're actually
looking for ways to just make our code faster.
We found out that we could have done it with C or by rewriting our Python to be more efficient,
we wanted to do that as well.
So yeah, we had a long casting timeline for doing this, but it was basically just me,
and I was all I was doing, so a little bit of time for experiments.
Sorry, there we go, and this one, okay.
So we used open telemetry and we started the load testing.
We had a set of load tests written in Locust for those that don't know Locust is a load
testing tool that runs in Python.
We managed to set not push enough pressure through the Locust load testing to actually stress
our system mainly because we needed to throw more hardware at Locust to make it scale,
which then cost more money and more operation time.
I didn't have infinite operation time to be spinning up hundreds of thousands of
Locust workers to simulate hundreds of thousands of users, so I tried to cheat.
We looked at work, which was better, it's written in, I think, C, so it's much better at load testing,
but I ran into issues with it in that it was really good at just hitting an endpoint
and saying, here's a thousand requests, and then, but that's not really what a user does.
So we're not, we're not Wikipedia, so people don't just scrape our website and say,
I want this page a hundred thousand times, they go say, I want this package,
they then start saying, where is that package, they then, we then break that package up into millions of different
bytes and send them and send it to them, so we had very complicated users, so we need to simulate.
We found a project called Goose, it's rust based and it's modeled on Locust, so it effectively
let us rewrite all of our load tests and rust and then run them on much cheaper hardware
than our previous Locust and push much more requests through it.
So we start, then we start to actually see our system on the stress and get an idea of what we're broken and what we could improve on.
We found that, on surprise, we, everything was slow inside the application, it's all business logic,
we were waiting on third party requests to different back ends, database load, that kind of stuff.
We also just notice weird slowness, so with you, whiskey, we use a concept called green threads,
which is effectively when the Python would wait for IO, it goes off and does picks up another request and starts handling it.
But the request that it was waiting on would, wouldn't be able to restart until the other request finished.
So if you had a really quick request come in, that was then just, it could block waiting on a very slow request that came in right after it,
because it very needed to do a very quick database lookup.
Unfortunately, we do a lot of database lookups, something we're working on, but it's, we do, it's, we're basically constantly doing database lookups.
So long slow tasks could interrupt and cause insane latency spikes as we're trying to figure out what's happening.
We also noticed that a lot of our time with these very slow requests was spent in serialization.
So we send back very large JSON and XML payloads.
If you've ever looked at, say, the Python index file for like what all the packages are,
what we have to generate that constantly for our users, for their private registries, these can be massive serialization cost for us.
And then yeah, we had our complex proxy chain, which we didn't have rate visibility into,
like, we have metrics from HA proxy and traces from EngineX, but not the other way around.
So we can see how long you set in HA proxy, and we can see how long your request was between EngineX and our application,
and then it's kind of just guessing from there.
There wasn't like a thing where we said, oh, look, we've, we're clearly just spending a lot of time in this one database loop request or anything like that.
It was lots of little cuts here and there that we needed to like resolve and fix.
So we started looking at a couple of different things.
The ordering probably wrong is actually wrong, so I'm going to go through this in different order.
But I'm going to touch on these tools, Granyan, Warjason and Jason Schema RS.
Each solve different bottlenecks we had that were constantly kind of like being hit.
So yeah, I'll go through them all. So Warjason is a drop-in replacement for Jason serialization in Python.
It uses Pyotree under the hood. It's quicker by about six to 12 times that of the standard Jason library,
because I say drop-in, it's actually dropping really for more for the Python to one because it returns
bites rather than strings, and it affects both. It does cover the vast majority of these cases, so all that.
So we managed Django being the very handy framework it is, actually just had a plug-in that said,
do you need to serialize into Jason, and you can just switch out into Warjason,
so we just switched in that library and something we were faster.
This is also true of our Jason logger. We serialize all of our logs to Jason, that adds up
as you're like handling 110 million requests to all that time adds up, so we switched that out for
Orjason, and then we got on to the more annoying part. We call Jason load and Jason don't
true out our codebase. And with this one we had to go through every single call of it and switch it
over to Jason load, determine did we need bites or strings or could we get away or what could we
get away with? I would like to say that this would have been automatable with some like L.A.
But I never even thought about doing that. I just started, I just had a train ride and I sat down
and started hammering through it for two hours and then just hoped the test would catch everything.
Yeah, so yeah, so that's what it looked like, it's very simple, it's basically the exact same
interface, I said the type is slightly different. We it worked. We only had one issue, one customer,
noticed we changed this because they were parsing their response with a red jacks and bash.
And one of those, yeah, it was like, you kind of go, how, what are you doing there that they say that
this is broken? We looked and we told them have you tried JQ and they said, no, we'll try that.
Yeah, so it's also one of the fun little things. Or Jason doesn't have any white space in it,
so payloads are ever so slightly smaller, which is nice. Yeah, happy little outcome for that one.
We then looked at J. Jason Schema R.S., which is for like validating your payloads and stuff like that.
This was one where clearly an engineer long before me had found the rust library and said,
let me experiment with this. We were using it in about half the times we were doing
seem validation, the other half we were using Python one. Clearly it worked for us. I just
one day changed one import in half of our files and did nothing else and shifted. I didn't even
actually tell anyone I shipped us. I just did this one day in a part of a bug fix and it worked.
And we got some nice improvements, but like I said, I didn't actually test this one because
it was so slow, it was so small and we were already doing it, so I felt no need to actually
validate this. Then we got on to the big one. Graning. Graning. So Graning is Tokyo Hyper
Server 4 Python. It effectively runs a hyper event loop, hyper server that takes in HP requests
and farms them out to Python workers. It supports S, G, I, W, S, I, and a custom protocol called
R, S, I, which I haven't kind of challenged to experiment with. We use W, S, I, so that was pretty
good for us. I want to say that like we were, this was just a level of like, I was just googling,
I looked at our requests, and I said, this looks just like a hyper server. So I literally just googled
hyper for Python, and this was the first, first thing I found, and then I started reading the code.
We managed one of the reasons that was very good for us is it actually replaced everything in our
stack. It does TLS termination, it does request Qing, and it does the worker farming out of work. So
we were able, so the big thing that we wanted was to be able to say we could get more out of our
compute. We had, because all these services ran on the same box, if we could replace one thing
to do so, we could maybe get squeeze more cycles out of each request and each box.
The other reason we were looking at specifically rust tooling for this one is I'm a rust developer
in a Python shop. I wasn't enjoying, I don't enjoy writing Python, I do it for, because I
paid to do it, but there was a willingness that if I found code that needs to be fixed in rust,
I would be told I could fix it, I could help out, I could write features.
It also has an active developer who's constantly improving it and making things better for us.
So we moved on to it, and we did some low testing. This is just replacing U.S.G. with Glanyan.
We were able to push 32% more requests through the system with latency going down for the vast majority
of things. This is actually to do with that trace the threading model, again, because we weren't
getting those random weights. We were actually each request was going through and living as life cycle
rather than randomly getting these latent spikes. That's why we kind of started seeing,
we believe we saw the weird latency spikes on the P95s, because these are the real P95s rather than
the random P95s. So we had a more track traffic, and we didn't have any errors that we noticed at the
time. We then decided we could rip out everything, try with just gradient, granion, and no
engine X, no HM proxy. So that's the problem with the other one. We weren't really testing the
Qing, which is a core part of our infrastructure. If we can't queue requests, we can, they get
dropped in the floor, and then they get redriven, and then we get a tundering problem, it looks like
requests, that's a request, that we're not doing work. So when we did that, we actually saw a 2x
increase in speed. No, TrueX input in TruePost with a 50% reduction in latency. I honestly didn't
believe the numbers when I did this, so I had to rerun the tests a couple of times. So these numbers
are a little over multiple tests and average showers, but yeah, good numbers, and we were pretty happy.
So we begin testing it out in actual environments, real requests and all that kind of stuff.
We ran out of the staging, everything looked good. We then moved it to our internal environment,
everything looked good. We then rolled it out to one region then onto an X region, and things stopped
looking so good. Like I said, we have, I think it was on the final region, yeah, we things exploded.
So I was actually about to go off and have coffee. That's how good I thought it was going to be.
I was like, I'm done for a day kind of thing. We exploded the database when we hit the last region,
because we waited to the last region was our primary region, and what we didn't know is that
we didn't have connection pooling. And I do not know how we could have so many engineers,
not know we had connection pooling, but for those that don't know, Django, four, and lower,
doesn't have connection pooling. It just has connection reuse. So every thread had a dedicated connection.
And as we split up more threads, more services, we maxed out our primary database and
every and we couldn't connect to it and everything collapsed. The U.S.D. green threads,
because of how it handled the reuse of requests, effectively meant it kind of pretended we have
connection pooling. We looked out different ways to figure this out. We tried RDS proxy because we're
an Amazon shop, and we didn't have time to spin up a peach balancer. It's not rate. So maybe
we should try peach balancer and see if it's better. So we're just waiting for our Django five
of great before we can actually get this rolled out, because there is real connection pooling in
that, and we're pretty sure it's going to fix us. So we're not 100% finished with us, but we're
ready to go once we get our Django upgrade running. We're still running this in our internal
environments where our testing is, knowing that it's still working for us for our internal traffic.
We also run into some fun with HVP. I think I was chatting to people about this last night.
NGX is doing something weird to our traffic that fixes us. We have no idea what it is, but
package formats are really weird. Package managers are really weird. They do weird things constantly
that are not exactly right HVP. They'll follow 307s to look for packages to look for packages
at different locations like if you're hosting an S tree or on a different CDM because
you do it yourself. They'll go to that backend and pull it out and say, grant, I've got the
there's a file. But if you're a doctor layer and you say the body of 307 is 0,
doctor will then look for a body that's 0 at the other end. And if it doesn't have a body of 0,
it'll actually still download it, but it will encode it all the metadata of the image saying,
that body is 0, which Unsurprisingly doesn't work. So we're not, it also started randomly
our LB had another one where it because we were setting that body. We only noticed that the body
we were sending the body wrong, which caused our LB's health checks to randomly air out. That's
why we fixed it and why we corrupted a lot of users' dockers. Yay, I will always be willing to
complain about Docker for the end of time. We're not going to fix it in our edge. We'll just
do some magic things, but Hyper was doing things correctly, and next to those things. Weirdly,
we're not on a hunger server. We actually, while doing the RCA for discovered we had the exact same
bug when we attempted to use Unicorn for instead of UISCII. So it's not just Hyper, it seems to be
UISCII and EngineX just magically make this work. They shouldn't. It also broke our, one of the
nice things was we could be run on a user's laptop, UISCII. It's really painful to run on a user's laptop.
So we switched our dev environments for gradient, and it got to testing as we run them in
approach. Unfortunately, poke all our bookers. We chatted to the maintainer. He says,
he's not going to fix it. The reason is the model for the general Python debugger can't connect
through the rawst pilotry running S. There's no real way around this. We just had to switch back
to a normal whiskey thing. I have a lot of notes that I'm going to speed true because I think
there's a lot of interesting things. We spend a lot of time just tuning S. We got values to match
our H.A. proxy, workers, backwals, but about, if you want to talk more about that, I can talk
more about that. So we then were running it out, and we realized there was no metrics in gradient.
Like I said, we needed that visibility. So this is where I actually got that time to start writing
metrics. So we have, we've worked the project, we added in a metrics collector for Methias,
we started, because of the way it works, we needed to move metrics across each Python process
back to the main rules process to expose the metrics to the, to the Python server.
It works. We got it working. We created PR. We were super happy. That's, and this is the
fork we're running internally in our versions. We never got it merged because we only test on Linux
and Max. The maintainer wants Windows support, so he didn't get there.
Both saying that on Thursday or Friday, the maintainer took RPR and didn't merge it, but
implement his own one that matches what he wanted, and it's merged in, and it should be in part
of the next release. So you can get backpressure metrics, what's happening, how many of
how many requests are in the slide, how many Python processes, all that stuff. So we'll be switching
to this, I think, two weeks time on it, or if when it ships. So this is what we're running
now. Django, we're almost there. I have little time left, so I'm going to speed through. The
real win here was the company now has faith in Rust and our ability to use Rust. To the point
that we're actually rewriting more of our stack into Rust. So what you might think as we go,
okay cool, we'll use Pyotree and ship it to the monolith. No, we're just going to throw out
our Python edge network and rewrite our edge network and Rust. So we're starting a project now
to do all that, but none of that would be possible, and I don't think I would have that sign off
to do that. If we had improved that we can do things faster by just the small little improvements
in the stack in Rust. It really built that made that cultural difference and people had the faith.
So if you're wanting to use Rust and Python, look for the bottlenecks, the serialization, the Io,
those things they just are better and Rust, you'll find that the error handling is good,
comparatively, if you want to do it yourself, use Pyotree. I've done it for small side projects as well.
It generates really nice Python. You wrap the code and some decorators, you're just going
to generate some very simple Python objects that you is well typed and easy to use. If you want
people to adopt your thing, aim for API compatibility, if it's drop in, I'll probably just drop
throw it in and see if it breaks my tests. And if it doesn't, I'll ship it. And with one minutes of
despair, that's it. Thank you.
Have you got any questions?
For the recording? Just a quick question. Any chance you are going to blog or open source,
your Goose load tests, because the scientific method you took was pretty awesome.
Yeah, I would, I can't actually open source the tests because they're very specific and there's
some stuff in there that if I open source, I would get a little bit trouble, but I think just
go have a look at it. They have really great examples of how to write tests in the repository.
I would recommend it for load testing because you can compile it to a binary and then ship
that binary around the world and then stimulate load tests from anywhere in the world,
which is great when you have a CDN like us and need to actually valid do end to end tests.
I have a non-technical question, maybe a social question. So you said that the maintainer of
one of the projects you're uptreaming to, they rewrote your patch? Yes. I've seen this a few times.
How do you feel like, because then you're not maybe a co-author on it?
I, yeah, it was a little like, oh, I wanted to get around to this, but I didn't, it's his
project. It's not my project. I was, I was a little sad, but I kind of like, I want to feature more
than I want to be the contributor myself. And I had let it sit there on patch from his feedback
for months. Like, he wants to feature, I want to feature, I didn't have time to finish this.
That's the, at some fortunate part of it. You don't know what you can write. It's the first time.
It took maybe like two weeks to get back for review. And by that time, my work couldn't give
me more time to go back and review the features. Do you tell what's your, yeah, exactly.
Well, thank you very much. Thanks.