You
Here the mic but they're trying to record this.
So if you have 96 cores in your system and turn all those loose on a parallel GC effort,
they're all going to sit waiting for each other to get out of each other's way.
So it's much more efficient if you can take some small subset of those 96 cores,
put them on concurrent GC, let the others work in the cash for your service threads.
My focus today will be on getting low latency for Java and making the Java execution more frugal.
Several dimensions of that.
We want within Amazon for concurrent garbage collection to be the default configuration within a couple of years.
We want concurrent GC to run as efficiently as G1 GC and similar or even smaller heap sizes.
It's not that way today.
What we find is that G1 is by far the most prominent garbage collector inside of Amazon.
Many services over provision the memory so that they can separate the pause times.
So this is our effort, this is our focus, this is what we're working on.
And I'm going to give you the story of the challenges we face as the engineers trying to deliver on this vision.
So back in 2021, we were beginning to see adoption of Shenandoah inside Amazon for various services,
but by far the small limited minority of services.
Along the bottom here, you can barely see it, this is the G1 latency.
It's pretty good all the way up to say 290, almost 300 transactions per second.
When we apply Shenandoah to this workload, first of all, this is a low test program.
It fails the test at 200 transactions per second.
On the way to that destination, you see the latency is up there 32520 over 620 millisecond latency,
even though it's a positive garbage collector.
So what's happening there?
Shenandoah GC cannot keep up with the pace of allocation for this heavy high allocation workload.
And so even though we're not pausing for stop-to-world GC pauses, we have to pause so the garbage collector can catch up with the allocation demand of the service.
So this is what motivated our work on generational Shenandoah.
We needed the garbage collector to be more efficient.
We needed the frugality of deploying a pauses garbage collector to match that of G1 GC.
Let me just say, if you look at this chart and you run this same test with generational Shenandoah,
and with generational GC, you'll see that both of these are much closer to the bottom line here,
but still G1 has the better p50 latency.
And for many services, p50 is what they care most about.
But if you care about p99 or p59s, that's when you begin to see the benefits of a concurrent garbage collector.
And when I say, we want concurrent GC to be as efficient or as frugal as G1 GC.
We would like to be G1 GC across the whole line of p50 as well.
We're not there yet.
One of the challenges with a concurrent garbage collector is knowing when to start the garbage collection.
If you start too soon, you'll spend a lot of time doing many more GC cycles than necessary.
That will chew up CPU time.
It will detract from the CPU available for your service.
It will slow your p50 latency.
If you start too late, you're going to lose the race.
You're going to deplete or exhaust the free pool before it's been replenished.
So you want to predict when to start it, so you finish it exactly the right time and don't do too many GC cycles.
This problem goes away with G1.
They just run and tell the memory is exhausted.
Stop the world, collect the young.
They do have the problem with the old garbage, the collection of the old region.
The old generation, they use something called eye hop.
And now I'll show you what to stand for.
Initial, initial, keep occupancy percentage.
So you set this to 40% for example.
When the old generation gets up to 40%, it will start the GC.
Let me make the point that G1 GC has a constraint that concurrent GC does not have.
G1 GC has to keep the young generation small because it has to do a stop to the world of the young generation without violating the 200ml sec in or whatever
POS time maximum has been specified.
With concurrent garbage collectors, the young collection is also concurrent.
So we don't have that constraint that we have to finish it in 200ml seconds.
It's a concurrent phase, so we can make our young larger.
Shenandoah, if you study its performance characteristics, study where it performs well, where it begins to have weaknesses, it has these two attributes.
If the memory utilization is very low, for example, if you have a heap of 5% live memory, I would say low memory utilization, Shenandoah does extremely well.
That's because it's got 95% of the heap as run way to allocate while it's doing the garbage collection has a lot of time to do the garbage collection before it runs out of memory.
The other time Shenandoah does very well is if the allocation rate is very low.
So if the allocation rate is very low, you're going to have also a lot of time to get the garbage collection done before you run out because you're not allocating very fast.
So what we've done in the design of generational Shenandoah is we basically replicate the Shenandoah garbage collector twice.
We have it running once on the young generation, once on the old generation.
In the young generation, we have low utilization.
Most objects die young. Every time we do a young collection, very few objects are still alive in the young generation.
We copy those and we claim all the other garbage automatically as a by-product.
In the old generation, we have very low allocation rate, promotion rate, very few objects get promoted.
So Shenandoah works well on each generation independently and we run them concurrently.
We use something similar to the G1 heuristics in Shenandoah.
So we reclaim the garbage first when we choose which regions to garbage collect.
If there's a region has a lot of light memory, we just leave it aside.
We'll come back to that later after more objects have died in that region.
This also lets it be very efficient.
We do that both for the young regions and the old regions.
We trigger old generation marking under several conditions.
Very different from the way G1 does it with their eye hop heuristic.
We trigger if we detect that the old generation has become fragmented, if it's spread out all over the heap,
if it's infringing on areas of the heap that we would like to set aside to efficiently handle humongous objects.
We'll trigger an old generation collection.
We also trigger an old generation collection.
If the old generation has grown more than a specific percent since the last time we did old marking.
So every time we do old marking we say okay there's 10 gigabytes live.
If it gets up to 10 plus 10 gig plus 12 percent or about 12.
Yeah about 12 gigabytes we'll start another old-gen collection.
Also if there's a shortage of available memory.
Just realize this is very different from the way G1 behaves.
When a generation of Shenandoah is running correctly configured properly,
what you'll see is a interleaving of an increment of old generation collection,
an urgent preemption of that to do a young collection,
and then resume the old collection and then another increment of young collection.
The reason we structured this way is because it's very urgent whenever we have to do a young collection
to get that young collection done very quickly.
That's the higher priority activity.
It's urgent because the services cannot run until they get their allocations.
And so we need to replenish the free pool very quickly.
That's most important thing.
Get that young collection done quickly.
The old collection usually takes more time.
There's a lot higher density of live objects in the old regions.
So it takes a lot longer to mark old memory than it does to mark young memory.
And likewise it takes longer to evacuate or compress those old generation regions.
So this process goes on until you've finished marking the old.
Then we do very much like G1 does.
We go into something called young mixed evacuations.
So at the end of old marking, we identify regions that are candidates to be collected from the old regions.
And in subsequent young collections, we do the normal young garbage collection effort combined with some subset of the candidate old gen regions.
We use the same evacuation cycle that the young generation is using so that we can share the cost of the barriers that support concurrent evacuation.
We run as some number of those young mixed evacuations.
And then we've completely evacuated the candidates from the old generation.
And at some point in the future, we'll trigger another old gen collection.
So this is the general way it works.
How do we trigger young collections?
The way this has been done is the same way traditional Shenandoah does it.
We calculate headroom, which is our working safety buffer.
The headroom is determined by something called an allocation spike factor.
We reserve a certain percentage, maybe 5% of the young gen.
It says we'd like to stay away from the pending on that memory.
We'd like to try to get it garbage collection done before we use that last 5%.
But in case we mis-predicted, that gives us a little extra runway to finish the GC before we're all the way out of memory.
And then we get we factor in penalties.
So there's something in Shenandoah called a degenerated GC cycle.
Degeneration happens when the garbage collector is unable to free memory faster than the application allocates memory.
So the application says I want 100 megabytes.
The GC says sorry don't have it.
That causes us to do a degenerated cycle.
That means stop the world.
That's the way it's currently implemented.
Stop the world.
Let the GC catch up.
So it's a safe point.
Thousands of threads have to stop even though only one thread needed that memory.
But we do the stop the world.
We do the degenerated GC cycle and then we resume concurrent execution.
So those degenerated cycles are very undesirable.
If we ever have one, we penalize because the degenerated GC, so that it would be more aggressive
to stay with the next GC sooner so that it won't happen during the run out of memory.
Basically, we trigger the collection when we are at risk of run out of memory before the GC can catch.
I want to show you some experimental results that are based on this heuristic that I just described to you.
For full disclosure, I'm providing the details of the configuration.
I'm not going to go into all depth, but it's here in the slide if somebody wants those details.
To give you an overview of what all that gobbledy group means, it's about 2,000 running threads.
They're allocating at a rate of about 1.7 gigabytes per second.
There's about 11 gigabytes of live old-gen memory.
This runs a simulation for 20 total minutes.
And every 90 seconds, we rebuild half the database approximately.
Inside of this simulation, there's a customer database and there's a product database.
Every other time we rebuild the customers and then we rebuild the products and then we rebuild the customers.
In between those rebuild operations, we delete some customers, we add some customers, we delete some products, we add some products.
So that's what's happening.
Notice that this is an intermittent variable allocation rate.
Every time you rebuild the database, you get a spike of allocations causes us to mispredict.
It's intentionally designed to be a hard benchmark to run.
In this benchmark, I'm going to share results from G1GC, yes?
What did I do wrong?
Thank you for the alert.
I don't know what happened here.
I still have it all.
Okay, I thought I was asleep, I don't know.
So yeah, so I'm going to share the results of G1GC out of the box.
I'm comparing with some of the special configurations and experimental development branch of Gension.
I'm not going to share the GNGGC results right now.
But here are some of the results from this data.
And the first thing I want to point out, this is showing a response time.
Those are measured in micro seconds for various heap sizes of the same workload.
Remember it's got 11 gigabytes of old gen.
It's got allocation rate of 1.7 gigabytes per second.
And what's interesting to see is as the heap size shrinks,
the response time goes up.
That's what you would expect because the GC has to run more frequently.
It has to interfere more with the service.
There's a big spike here.
There's a quantum leap.
That corresponds to compressed oops optimization.
So what's happening here is in a 32 gigabyte,
and everything to the right of that, we do not have compressed oops.
And from 31 gigabytes down, we do have compressed oops.
So it's like two completely different worlds.
Your miles may vary.
It depends on your application, how many pointers are inside your data structures and so forth.
But it's very interesting to see that a 31 gigabyte heap
is running with almost the same efficiency as a 48 gigabyte heap.
And a 19 gigabyte heap is running with more efficiency than the 32 gigabyte heap,
because of compressed oops.
At the scale of Amazon, this represents hundreds of millions of dollars
to be able to move into that compressed oops range.
So that's something that's very important to us in terms of being efficient
in the way we deploy these systems.
A lot of data here shows the experiment we did.
The reason, I don't expect you to digest all the details,
but the reason I run this at all these different heap sizes is twofold.
The typical Amazon customer says, my objective, my need, my SLA requirement,
is to satisfy, for example, 50 millisecond response time at p99.9.
So we have to ask, well, how big of a heap do you need for that objective?
And you go over to the p99.9 column, and you say, hey, 50 milliseconds, that's easy.
We can do that even with a 19 gigabyte heap size.
But every customer is different.
The other thing that this shows us is that typical service will configure for
the intended design workload, but they get spikes.
Something happens in all of a sudden, they get twice as much traffic as they plan for.
So that's like shrinking the heap size in terms of how efficient it's going to operate.
So you want to make sure that it's scalable across the range of different heap size.
So that's why we also look at different heap size configurations.
This is a G1 comparison on the same workload, all the numbers are there.
What I'm showing you is how they compare.
Thank you, five minutes left.
So p50, p95, that's pink color means generational Shannon do what Shannon does is worse in latency at this end.
In the middle, generation of Shannon does a lot better.
In fact, like 94% better, if you go back just real quickly,
the three nines column, those are in the hundreds or thousands of microseconds,
whereas with Gen Shen, the three nines column was in the tens of microseconds.
So those are huge differences.
But at the p100, comes G1 is still better, bothersome to me.
bothersome to anyone that says, hey, I chose this because I wanted better latency.
So this motivated some changes to the way we do our triggering.
In particular, we've implemented, this is not yet fully integrated.
We have the branch PRs that are, some in draft, some ready to integrate,
but these are the changes.
Instead of sampling the allocation rate every 10 Hertz, look at it every 500 Hertz,
detect acceleration of allocation rates.
Don't assume it's an average that's going to stay constant.
If you see it's increasing from one sample to the next,
it's going to continue to increase.
Do your calculation of the assumption based on the rate of increase.
And finally, search number of GC threads.
If you do your computation, you say, we triggered now.
We're already destined to lose the race.
Let's search the GC threads.
When we do this, I'm kind of at the end of my time.
But you see this last P100 column has almost all switched to green and significantly green.
So we think we're making a very significant and worthwhile improvements.
We continue to work on that.
We've got a still an outlier down at the bottom.
It's not a severe outlier.
The numbers are reasonable.
So you can't see.
It's 339,000 microseconds versus, I'm sorry, I don't have it here.
But it's like 100 microseconds.
Next steps in our path.
Big plans for the year.
We're going to reduce number of safe points.
Currently, there's four.
We've actually reduced it to three safe points every time.
Gen Shen runs.
We're going to take it down to one safe point.
The others will be hand shakes.
We're replacing the out of memory during evacuation protocol.
Right now that degenerates.
We're going to do it a different way.
Then we can take away the degenerated cycles.
We're going to take away Shen into a pacing.
We're going to replace that with the GC worker thread search.
We're going to defragment the humongous memory concurrently so that we don't need any full GC cycles.
We can get rid of the full GC cycles.
And we're going to also prototype if we get it done.
A from space invariant instead of the two space invariant that's currently implemented.
That's going to give us higher throughput we believe.
And better scalability, especially to things like new market textures.
Gen Shen is now an open JDK 24.
It's not currently in production, but we are evaluating it under experimental production workloads.
Limited blast radius at this time.
We're planning later this year, middle of the year, to have a GA product that's fully supported for production use.
It is not in Coreto. It will be in Coreto 25.
It'll be in Coreto 24.
But you can get it in a nightly build of Coreto 21 on this website.
That's not our officially supported Coreto product, but it is a experimental release that you can toy with.
So I think we're okay for maybe some questions.
If there are any questions.
Okay, thank you.
Are there any questions?
Yes?
The overhead?
Okay, so the question is, how does the memory overhead compare between G1, Gc, and generational Shenandoah?
We have done all these experiments using the same heap size.
G1, for a given heap size, has a much more complicated and large, remembered set representation, much larger than ours.
So for any given heap size, choose G1, 30 gigabytes, G1 is going to consume a larger RSS than Gen Shen.
But our goal is to match the G1 sizes or even improve upon it because you can now do the young collections concurrently.
Any other questions?
Yes?
Amazon has a rule that we cannot say things like that.
You can see the GitHub repository which people are contributing.
There's a small handful.
Say again, handful?
Anything else?
Thank you very much.
Thank you.