Hello everybody, I hope there is still enough oxygen in the room for you to focus on this
next talk.
I will be talking about the GIL or the global interpreter look and it is a fact on API
performance of it talking about the past, the present and the free threaded future of
Python.
My name is Ruben Hiels, I am a software engineer at Techwell, which is a skills intelligence
skill-up based in Kent and I also have a little side project flowed-up depth.
A quick refresher since it is quite crucial to the next to the following of this talk,
threats versus processes, a lot of you might already know everything about it, but a process
is an independent program with its own memory space which has isolated memory, which
is safer but also incurs higher overhead and they can communicate with each other over IPC.
Next to that, we have threats which is a lightweight execution unit contained within
these processes, they have the same shared memory space of the process, which is efficient
but can lead to some issues regarding data consistency and race conditions and they can
directly communicate via this shared data, the process can even either have one threat
or multiple threats.
So now let's get into it, the skill, what is it really?
Well, the name speaks for itself, it's a look that allows only one threats to execute
Python by code at a time, it's a look that gives the interpreter to a threat of the global
process.
Many seats as a core thing in Python, where everyone needs to work around, but it's
not actually a feature of Python, it's not the Python language can work without it, it's
implementation detail of C Python or forks of Python don't necessarily have this, and it is
there to protect its internal data structures and very specifically around the garbage collection
and reference counting.
Here we see, for example, a world where we didn't have something like the global interpreter
look, here you might have two threats, one thing to access the same variable, they both read
the current reference count, let's say four, let's say three in this case, they both
incremented, they both say it's four and write it.
Now in our states, the reference count is four, but a reality it should be five, and this
can obviously read too many issues.
Now with the skill we don't have this, since only one threat can execute at the same time,
and this will have to finish writing it before it can give its access to the next one,
and here we end up with a correct solution.
I even went back in time to find the very commits where the global interpreter look was introduced.
I might not have been born, but Hilo van Rosem was already writing Python in the good day of
August 4th, 1992. I even went to find the specific codes, it might be slightly too small,
but it's actually very simple, it's you launch a threat, you check, do you have the lock or
not, either you execute or you don't.
So you might ask yourself, why do we even have the president?
If only one can execute at the same time, we might as well just live in a single threaded world.
Well, the global interpreter look is the default, but applications are like the C Python implementation
can still choose to release this lock in specific cases.
One of these is Io, or input outputs like network, file reads, etc.
Especially back in day, we see like hard drives where way different clothes was still only spinning
so we had to read speeds 0.5, maybe 2 megabytes a second, network latency of hundreds of
milliseconds and bandwidth of 14 or 40 kilobytes per second. So this is a really significant
chunk of your application execution time and still today, so if you can have other threats
executing during this time, you still got a big speed up. Secondly is the hardware limits.
Like multi-treading wasn't necessarily a thing because this compact desk pull at the time,
one of the pretty top-of-the-line machines only had one tread, like multi-one core,
multi-core machines were only introduced way later and at least common.
There were in some high-performing context. And finally, one of the things that makes
spite and what it is today is its extensive ecosystem. And having a global interpreter look makes
it way easier to write the C extensions like for example an empire that so many of us rely on today.
So maybe we wouldn't have had this should not have been the case.
Now here you see all of these numbers already tell you it's years. Does anybody have a guess
what this might signify?
That's very true. These were all attempts to remove the bill. Even then I was lying in 2015 there
are two attempts. So it's not new that people want to get rid of this and really have the full
performance of the computer through its availability. And if you saw correctly it was only introduced
in 1992. It's 1996 and they are already attempting to remove this. But why were so many attempts
that so many attempts fail? Why wasn't a bill removed such a long time ago?
Well the initial attempt in 1996 they did this by having fine-grained locks. So instead of one
big look, every data object had its own look. And by doing this it did work. You could have multiple
threats executing at the same time. But the single trial performance was about three times worse.
So this was really not a sensible trade-off to make in the majority of the applications.
The same happened in 2007, 2015, they tried other things like atomic operations, etc.
But finally the years 2021 and it's beautiful. What happened here? A new fork of CPython was introduced
to the no-gill fork by Sam Gross, which was a software engineer at Microsoft at the time.
He introduced the concept of Bistreferin scouting. Bistreferin scouting is quite simple if you
think about it and it uses the property that while every threat can theoretically access every object,
realistically a lot of the variables will be only accessed by one threat. So there is quite a
lot of locality regarding the threat. And he abused this to say this variable is owned by this threat
and it can access it very quickly. And if other threats want to access it they can, but that will
incur performance penalty. He also introduced many other things like immortal objects,
dreadstif, memory allocator also developed by Microsoft Mimalok and many other small tweaks.
But this was really the core innovation and allowed it to be only have minimal over at I think
10 or 30% of the current implementation. And here is also after this it got officially accepted
as a Python enhancement proposal. And here it was I think set in stone that should something
be made with less than 10% over at or the current single threat implementation or yeah with the
gill then it would become mainstream Python. So how we go to here the gill it's not on the feature
it's a C Python implementation detail to protect these data structures many items failed mainly
due to performance penalties. And Python 3.13 finally introduced free-training and experimental
modes mainly through the innovations of Sandrose. And you always will have to make a trade-off,
you will have small single threats low down in order to enable this true parallelism.
Another question maybe, how do we actually use this? Quite simple, you add a T off your
Python version. This is a new standard if you add a T 3 since 3.14 you automatically get
the free-trained version and UV also has built in support for this. But realistically let's say
what does this do to your actual applications? Here you can see I think about the most basic
version of a multi-trained program, you just do work some basic CPU calculations over a set of
threats. Here in the normal non-free-trained version so with a global interpreter look enabled
we see an execution time of about 2.3 seconds and now in the free-trained version we see that
it goes down to 770 milliseconds which is about a 3x speed up. By the way all the benchmarks here are
executed on this MacBook M3 for reference. Okay let's now look at the single-trained
because we see okay there is real significant speed up possible with this new free-trained
modes but what is the penalty we pay because that's the reason it was not implemented for so long.
Well in the normal version we see that it's about 2.3 for seconds and then in the free-trained version
2.3 7 which is really only a 1.2% overhead in this case. Note that this will of course also
highly depend on your actual use case I've seen some people have seen around up to a 10%
increase but it has really come down a lot and that's also what this growth indicates this was
benchmark run by Miguel Greenberg and here he implemented a look across Python versions
about how did the performance change and what I want to highlight here is the difference in
performance penalty between the standard version and the free-trained version between 3.13 and
3.14. For example in the whole left bar with Linux 3.13 we see it's maybe 4 seconds of additional
time spent but in 3.14 it's yeah minimal not even a second so this really shows the hard work
the C Python developers have been doing to make this a reality. So what does it mean for CPU
bound applications it is now possible to have actual true CPU parallelism with about a 3x
speed up I found on some machines but of course the more of course you have the higher the possibility
for parallelism is it's also best for embarrassingly parallel workloads as you often call it
so fully independent calculations but this is not new to Python this has been the case for all
multi-trained applications. Single thread is now minimal for a lot of application
once again please test it out with your own workload before you push on thing to production and
everything starts breaking down. Thread over at matters like you can see I didn't explicitly highlight
this but between the multi and the single-trained version did the same things but there was a
performance penalty to be paid. Now for the API developers this was something I was personally
very interested in and I think many of us some might do of the real scientific calculations but
even they like a lot of scientific calculations is now done in NumPy but you often hear okay why are
even investing so much efforts into removing this global interpreter look if it doesn't necessarily
affect us that's much like it's all applications are Io bound anyways like CPU it's only like
for academic reasons that we might want to remove this so that's when I got curious. In current
day and age Python APIs scale by handling concurrent request of course with something like a
unicorn so they require parallel execution they can do their Io etc but at some point the
guild will block their thread based parallelism if they really get to the CPU bound work so
the current solution today is spawn multiple worker processes so these fully independent processes
no threats which operate fully independently of its owner on memory space and all of that but
processes have their trade-offs so we said it's a way more heavyweight thing than a threat so for one
processes don't share memory space let's say you have your multiple worker processes you
initialize your API it's about 500 megabytes you have to load in some data to be able to
surf things once you go up your processes you can see your memory requirements go through the roof
and your cloud build will go as well next to that data sharing is quite hard let's say if you
can just have a local dict which is your cache or something you can easily do that with threats
with processes way harder require serialization IPC etc as well as just being slower in general
but and one big but it doesn't have this global interpreter lot of content so does that make it
worth it or not and this is where we need it to do some benchmarks I tried quite a lot of configurations
both specifically to test retreading but also out of my own interest in a traditional way I looked at
the juni-core and traditional trend scaling I also took brainy into consideration which is a
web server similar to a juni-core but it's rust based and it offloads some of the network type
processing to restrates instead of doing it in the Python world next how does this work with things
like async processing a false API as well as g event and then finally also let's take
free-treading into account with threats and how does it compare to null type processing since what
the question is I really want to answer is with this free-treading enabled can free-treading match
the null type process performance or even exceeded but either way it will reduce the overhead
it encounters first quite a simple benchmark we'll be talking about CPU bound APIs what is
really is is we basically I took the do work function and put it behind an API and here we see
similar characteristics so we see like the maximum RPS it can reach with a global interpreter look
it's quite static across threats it's just CPU work so no parallelism to be found there
quite expected and then once we go once we go free-tread the notes then we see of course that
we can scale well very further and here we basically see the same characteristics we saw earlier
on the raw normal Python execution but to come back to my initial thing of the people saying
oh where I don't want anyway we're just waiting for this slow API we're just waiting for our
database well that's why I created more of a mixed work load so I simulated an API I looked at
some research usually most real production applications have around between a 70-30 or 90
10 split between IO waiting time and actual CPU work CPU work here is often parsing during
calculations some of some of that type of logic so we compared the different scaling strategies
and we varied the threats or worker count between 1 to 32 and we saw how it went the main
measure I wanted to see here was what is the maximum request per second we can realistically serve
as this is what often matters most with API performance and how efficient can you run your
API given a certain load so this is where we start this is the very baseline we see
junicorn with some G threats which is the basic trading implementation in junicorn
initially we see that it can is basically increasing linearly until it clearly reaches some
saturation point if you compare that to grainy and we see basically the same characteristics
except a grainy can go a bit further because it does some of this network IO stuff in the
restworld then you might ask yourself how does this compare to an async framework like a fast API
or any other ISGI framework well here we see that it basically caps up caps at the exact same
point which is what the which is where you see you or like the guild gets saturated and get that point
you're just always waiting for CPU or time on the CPU illustrated is a bit so let's say you have two
threats which is still in a linear scaling mode you just have some IO weights and some CPU time
but they can easily be interleaved and you don't really have any issues but let's say you now have four threats
now you can see like the actual CPU time it's getting quite overloaded I see there is already some
overlap you can hear that please but once you go to 16 threats you basically are still all the time
waiting for the guild and that's the saturation we see adding more threats doesn't matter at this point
at this point you're just waiting for the guild to do your real Python processing and handle the
request so how can we break through well as we set the traditional way to break through with these things
is true multi processing so once we add once we add our our workers we see that we can scale quite a lot
further this is once again you can see it's also flattening off which is similar to what we see
when your actual threats get saturated now this is where free trading comes into the picture this is
very one to see does free trading actually follow this line so first I tried it I run the unicorn
if free trade mode mode and it gets very close to this line the one thing you can clearly see
is that it breaks through this barrier of in this case about 400 requests per second which is where
the guild gives up or at least the guild becomes the main bottleneck of your API now if you compare
this to a gradient you might think oh there is a purple angle well it's exactly on top of it so
actually it matched the full free the full multi processing perfectly and this is how we really
can see that right now if you can run your application in free trade mode and you don't have any
extensions that depend on the guild still being there you can do this and you can just run
in free trade mode and it should just work a little bonus I added since I have not heard that
many things about it online it's a bit small but it's a library called G event it's something
I personally find quite interesting to use the concepts of greenlets to handle your IO so what is this
is you can use it in a fully WSGI application your jungle for example but it actually
monkey patches a lot of your IO to get to a similar performance so I think here this was just a WSGI
server one one trep I just enabled G event and it was able to get to 93% of a full ASGI implementation
which is quite impressive especially when you look at our jungle it's low you can do async stuff
async also has real downsides of like going full async modes and this gives 93% of the
advantages I will give a note here we did try this and sometimes like connection pools and stuff
like db connections pools once again try it please first locally in a secure way before you
push it to production so what is the thing what are the things I wanted to give which you
you're the global interpreter look creates a scaling ceiling regardless of your server choice you
can have the top of the line your your gradient very modern but at some point your CPU will get
saturated and every threat will be waiting for your lock async help with IO concurrency but still can
escape the CPU bank ill limits that's one thing I've seen many like mentioned a lot like
just go async and you can scale to the moon well real APIs do still have actual CPU work and that's
also what we see here free dreading enable this linear scaling for example in this case on this machine
which is the 2.5x improvement at 32 threats but for IO heavy APIs if you really are full IO heavy
and you don't want to go free treading yet or you still are depending on some libraries which don't
support it pacing and g event remain excellent choices also I was quite interesting to see
a modern web servers like gradient that are leveraging these threats if you're interested in
gradient we're currently running in a transaction and century that like the outlook in company you
might know they're also currently switching to it which might be interesting so as conclusions
I think the guild was the right trade-off in 1997 but the hardware and workloads has changed
so it's time to get rid of it free treading delivers true parallelisms without and many
memory overheads with 2.5x scaling for mixed workloads in production environments for new
treading for new projects consider free treading I know some libraries still have issues with it I think like
UV query for example like there are UV loop implementations still has some issues but for existing
projects that do depend on these async and multi-process work well enough but I do believe the
future of Python performance is multi-traded so I encourage you to try it out and check what
it works for you and let's make Python falcer so thank you very much for the very interesting talk
do we have any questions?
do the third party packages are there any requirements for them to meet updated to be able to work with a free
threading mode? So native Python applications should just work with it I think it's mainly
the compiles like actual C Python extensions that need to say hey we are actually
compatible and we do allow this to be used in free treading modes but usually when you install
the packages it will ever if you can't install it that's how you need to find it out
hey just how the gil was kind of like well they've been took a lot with like an implementation
detail of C Python um is there any Python is there anything for over the way that variables get
allocated to processes for reference collection in terms of or is that just an implementation detail
that there is no control over and it's just Python does its best or C Python does its best kind of
gets as to which thing is most likely to have access to a given variable quickly versus slowly
it's a very interesting question I personally didn't do any research into that I haven't seen
a syntax that you can manipulate like in which thread it gets allocated I think it's mainly where
does it get created probably but it's interesting that I said they don't have response for you
so one question we got was does enabling free treading modes require any more implementation on the
code site or is it a simple as changing your version meaning all packages all threaded packages
will be freed by default it's indeed you don't need to do any changes on the code site
it's when you yeah you need to install a new Python version and when installing a new Python version
you need to reinstall your packages anyways and then it will automatically install the correct
free treading version if those packages allow for it of course
then we see which high profile Python modules are not working with the guillotine I what would
prevent us from using 3.14T all the time I don't have an exhaustive list sadly like the one
I personally encountered it was I think you you've loop had some issues or at least 3.14 but it's
really something you have to try out for yourself I think it's most of the real high profile ones like
a numpy will definitely have evolved by now it's more or more so the lesser known one or the
deeper down in the dependency graph that will be affected
is there a future where the t-bills are not special but to default and what timeline with this require
the answer is I think yes but I don't remember the full timeline I think in the
in the prep where they consider it switching to the free treading modes they set when it would
be fully transferred but you you should you can look that up yourself insert the last question
when you enable this one to directly run into race conditions with your own code if it's not
explicitly protected against it so the Python protects you in the Python codes the main issue is
if you're writing sea extensions like a numpy and really high performance codes and that's where
it starts then you do actually need to think about this and see whether it works with this new version
as well but as long as you're writing pure Python you shouldn't need to care about this
thank you
thank you again
you