So, good evening everyone. I am Master Toshmera and I am a principal software engineer
with Red Hat. Today I am going to talk about project
lighting and we will look at what we have done so far in this project and what are some
of the things that are coming up in this project.
So, I have been involved with project lighting for more than a year. Let us do a quick recap
of what this project is about. So, going of the lighting project is to improve the start-up
time, warm-up time and equipment of your job application and it aims to achieve this
by shifting computation. So, this is not a new pair and JVM has been using this technique
in various ways to improve its performance in. So, you can shift computation forward
in time or backward in time. So, GC where you know the GC is an example of shifting
computation forward in time, your object becomes dead at some point in time, but you delay
collecting that memory.
CDS archive it allows the JVM to pre-pass the class files and store in a format which
is easier for JVM to consume. So, that is an example of shifting computation backward
in time. So, instead of doing this processing at runtime, you are doing it in archived
generation time. So, so far this project in this project most of the
what has been has been improving the start-up time and warm-up time. So, before we look
at to what we have done so far, let us understand why the JVM is low to start.
This is because JVM performs or tends to be tends to do things lazyly, it performs
computations on demand as required. And the reason for this behavior is it makes the Java
play from highly dynamic which is good for Java developers, they get flexibility and
extensibility and it also benefits JVM in multiple ways. So, in case your application
behavior changes at runtime, it can add up to it quickly. It knows the exact hardware
and CPU that is running on. So, it can exploit their capabilities. And obviously, it
is doing the work which is needed to run the business logic. So, there is no wasted
of it here. But all of these benefits come at the expense of stretching your start-up
time. But is that how the things have to be? If you look at the from the Java developer
point of view, once your application is returned, most of the classes and their interconnections
are established. And JVM is repeating the activity of loading and linking those classes
in every run and they do not change over the from one run to another. So, here in lies
the opportunity, we can shift some of the computation that the JVM is doing in every run
and do them ahead of time. So, let us look at this a bit more deeper into details.
Let us say you have a hallwall job program and you want to execute it. You compile it using
your JVM compiler which generates your class file, it is a binary file. And the class
file not, it describes your the contents of, it describes the business logic of your class,
but it also establishes or it describes the connections that this class is going to have
with other classes in the JDK. For instance, in this example of a hallwall, it is using
a JDK class called system. It is accessing its static variable called out which is a
happens to be of type print stream that is calling a print allen method in the print stream
class passing some constant string. And the main method you can see there is accepting arguments
which are by JVM strings. So, as we write the JVM code, we are also describing how this
class depends on other classes in the system. And all of this information is represented symbolically
in the class file. And when the JVM wants to execute the byte codes, it needs to transform
this symbolic information into concrete addresses in its address space. And this process is called
resolution. As part of the resolution, the JVM if it finds that the required class is not
in its address space, it may load it from the disk. It would have to link that class again,
do the initialization. And as part of that is going to run more byte codes and do the resolution
again. So, this cycle repeats itself. So, the phases of loading, linking and resolution,
they are feeding into each other. And that results in creating a complex graph of classes.
And this process is repeated by the JVM on every run. So, let us understand this in terms of the
metadata model that the JVM creates during these activities. When the JVM gets the class file
by from the class loader, it creates an internal representation, which is called as instance class
here. In fact, all the structures here are from the JVM point of view. So, the instance
class is the central data structure here. And it has pointers to other structures, which represent
different information in the class file, like your constant pull the methods. So, if you are familiar
with the CDS archive, I just mentioned about it in my previous slide. So, this is the representation
that gets captured in the CDS archive. And when the JVM uses the CDS archive, it gets these
p connected graph of data structures. And that saves some of the startup cost.
So, once the loading is done, we need to link this class, we need to establish connections
with the other classes. And that happens on demand. So, when the interpreter starts executing the
byte codes, it comes across symbols, symbolic representation of classes, methods, fields,
called sites. And then, it asks the VM to do the resolution. As part of the resolution,
the VM also stores the resolution information in certain data structures. For example, the
constant pull cache and other structures hanging off it. And these structures, they act
like a resolution cache for the interpreter. Once these structures are filled up, the interpreter
doesn't need to ask the VM to do the resolution again. So, let us look at how these structures
are filled up. Let us take an example of field resolution. So, suppose your program is accessing
a couple of static fields, the bar and full. And this is represented, this would be represented
in your class while using foot static byte code. And the operand to put static byte codes is an
index into the constant pull, representing the field to be accessed. Now, when the JVM
loads this class, it re-rise the byte codes in such a way that replaces the constant pull
index within index in a structure called result field entries. So, before resolution, this is
this is what the, this is what the connections look like. So, you have a foot static field byte code.
It is referring to, referring to slots and the result field entries are in. And you will notice
these, the result field entry structure is empty. The only information it has is the index
into the constant pull, representing the field. So, when the interpreter starts executing the
byte codes, it comes across foot static and it observes that the, that the corresponding
result field entry structure is empty. So, it asks the VM to do the resolution.
So, where the VM will refer the constant pull entry. So, it goes out the class,
that contains the method, the name of the field, the class that contains the field, the name of the
field. And then it has to locate the field, locate the class in its address space. If the
class is not present, then it has to load it, link it and you know follow the follow the all the steps.
But in the case, it will, it will locate the address of the instance class and it will find the
field, present in that the offset of the field in that class and puts that information in the
result field entry structure. And now, it can jump back to the interpreter. The interpreter can
refer to the result field entry structure. It has the required information to locate the field,
the address of the field in the address space and it can access it and continue execution.
That is the another example. This time, we are trying to load some constant string constant.
This is represented in the byte code as LDC byte code and again the operand to the LDC byte code
is an index in the constant pull reference to the string. These LDC byte codes will then be,
will be rewritten by the JVM at in runtime. And it will replace the constant pull index with
an index in the result references. So, result references is nothing but a, but an area of
javaling objects. And as before, when the interpreter comes across LDC byte codes, it fetches
the corresponding object from the result from the result references array. It finds that it
is nulls. That means, it is not exact result. So, it has to call the, call the VM to do the
resolution. The VM will create a javaling string object corresponding to the string, being
that needs to be accessed. It will intern that string and put a reference to it in the javaling
object array. And from there onwards, the interpreter can continue execution.
The resolution of other entities like the methods, it happens in the same way just that the
data structures that are populated are different. So, there is a lot of back and forth happening
here between the interpreter and the VM during the initial execution of the byte codes.
Until the resolution information has been completely filled up. So, at the end of, at the end of
this whole process, what the JVM has done is created this big graph of data structure or the
metadata objects that we call it in hotspot jargon. And that describes not just the, not just
the class file, but it also describes its connections with the, with the other class file
at runtime. And this is done in every run of the java application. So, there is a fair bit of
repetition in the java activity during this phases to reach at this stage. And we can remove
this repetition. If we can somehow store this structure as is into a disk and then ask the JVM
to map it and you know it gets the, it gets this graph of, pre-built graph of data structure.
And this is similar to what CDS archive is doing right now, but we are going one step ahead.
We are talking about storing the classes in a link state in the archive.
So, that brings us to the JAP 43. This JAP is about storing, storing the
floor or being ahead of time, class loading and linking. Class loading, ahead of time, class
loading is already provided in some form by the current implementation of CDS archive. So,
we have built on top of that existing technology and we have added the capability to, to link
the, to store, to store the classes in a link state. And the way we achieve this is by,
by doing training run. So, the goal of the training run here is to touch as many classes,
as possible, as the application would do in a production run and actual run.
And the information captured in the training run is then stored in the configuration file at the
end of the training run. The JVM then reads this configuration file and emits a file called
AOT cache. And this is the file that contains all your classes in a pre-link state.
So, the JVM can adopt this AOT cache and can adopt these structures from the AOT cache.
And this happens, this adoption happens even before the main is called. So, by the time the
application starts exhibiting, your classes are already in a, in a link state. And you save
quite a bit of significant time during the runtime in the start of phase. So, it is worth mentioning
here that the JVM still retains the ability to link and class link and load and link classes
at runtime. So, in case if you are production run touches a class which is not in the AOT cache,
the JVM can still load and link the class at runtime. So, in a sense, AOT cache is not
putting a closed world constraint on your application. So, let us talk about couple of challenges
we faced. The first one comes from user defined notice. So, currently there is no mechanism in
the JVM to carry the identity of user defined notice across the runs. And secondly, they do not
have a well defined behavior. So, that makes difficult for the JVM to support user defined
orders in current scheme of things. Right now, we are only supporting the built in orders,
your boot loader, rate form loader and just system loader. The second challenge is in,
is in running class initializers ahead of time. So, class initializers are, they are very
messy, they are suppy in nature, they have, they can run random Java code. So, they are not pure,
they can have side effects. And they can also pull in dependency from the environment.
So, and the environment can vary from the training run to the, to the production run. So,
what we have done is, we have identified a very small set of JDK classes which are,
which are safe to initialize. And only those classes are initialize ahead of time.
And that is also done not from the performance point of view, but it came out as a necessity
when we try to do the pre-linking of invoke dynamic backwards. So, so far, all the classes
that, that, that are pre-linked in the, in, in the IoT cache, they still need to be initialized
and run time. So, I talked about what we have delivered so far. Let us quickly look at what's,
what's in the pipeline. So, now that we have a concept of training run and IoT cache,
we also want to capture the profiling information in the IoT cache. And we also want to do
compilations ahead of time and store them in the cache. And what kind of benefits we can achieve from
this, from these activities. So, let us understand the profiling and execution model a bit,
what, what the JVM is doing right now. So, there are three players in this picture.
There is an interpreter, there is a C1 compiler, which is a fast compiler, but it generates an
inferior code and the C2 compiler, which is your, which is an optimizing compiler and generates
the most performant code. The JVM employees here, compilation policy, that means
methods start executing in the interpreter and as it's executing, some of the methods which are
hot, that the JVM thinks they are worth compiling, they are moved to the, they are compiled by
the C1 compiler. And out of these methods, some are very hot and performant critical and they
get compiled by the C2 compiler. And these transitions happen based on the invocation count of the
method. So, invocation count is acting like a, like a filter to identify the performance
critical methods in the application and they get compiled by the C2 compiler.
So, talking about C2 compiler, it has to, to be able to do powerful optimizations and perform
the most optimal code. It needs insights about the runtime behavior of the method and this
information is captured by providing the method execution. So, the, the profiles and this
profiling is actually done by the C1 profile code and it can also be done by the interpreter in
some cases. So, profiling information that is captured doing by this, by this activity, it tells
us, it can tell us whether, whether a particular branch has been taken or not, how many times
a particular branch has been taken, what are the, what is the type of the receiver object at the
fall site and the type of the return values and the arguments. And all of this information allows
C2 to do something called speculative compilation. What it means is C2 assumes that the, that the
method is going to behave in certain way and it, and it optimizes the compiled code based on
that behavior and in case the assumption fails in the future of the method, then the JVM will
deoptimize the code. So, for instance, the C2 can omit out a code path, if the profiling
information tells it that this particular code path has never been taken so far.
Instead of that code path, what C2 does is it puts a deoptimization trap and the purpose of
that deoptimization deoptrap is it acts like a safety net. So, in case in future runs, if the
control does reach that control path, that code path, then the deoptrap will cause the execution,
will cause the execution of the method to continue in the interpreter. So, we look at how the,
how the method execution transitions from interpreter to C1 profile code to a C2 fully optimized
code. And this transition takes its own time and that is why you have warm up phase in your application.
This is the phase where you will observe lot of compilation activity until the, until the performance
critical methods, they, they are not compiled by C2, they are get compiled by the C2 compiler.
And at that stage, your application is performing at your peak level. And after that, you will observe
that the compilation activity goes down significantly. So, we can now understand what we stand to
gain by storing the profiling information and, you know, if you store the profiling,
capture the profiling information, the training run and store it into a UT cache and make it
available to the JDK or to the JDM at startup itself in the production run. Then it can actually
do the, come out of the interpreter quickly and do the, do the transition to the C2 combine code
much quickly. So, it has the potential of reducing your warm up phase of the application.
Also, if we store the compiled code generated during the training run in the UT cache, then
and make it available to the JDM during the startup phase itself. So, that it is able to use it
as early as possible. Then you are saving significant amount of CPU cycles which would have otherwise
we spent in compilation during the run time. And this can be used these free CPU cycles
can then be used by the, by the application threads to start up quickly. So, both of these techniques,
they have the potential to improve your startup time and warm up time.
As with the, as with the, at of time loading and linking feature, the JDM still it is the ability
to do just in time compilations. And this is very much needed because the behavior of the application
in the production run can be differ from the training run. If that happens, your UT profile data
may may be invalid or or the presumptions that were made during a UT compilation, they may not
hold true. So, in such cases in such scenarios, JDM still needs to de optimize the code and
you will still want to recompile the methods back to the C1, C2 level. And lastly, apart from
compiling the Java methods, the JDM also generates code for its interpreter. It generates small
snippets of code which we call as steps and blocks which have the compiling methods to do some
runtime activities like throwing exception, handling of exceptions and de optimization.
And then there are adapters which which allow the JDM to transition from interpreter to compile
code and vice versa. And all of these, all of these are generated at the startup phase.
So, although although the code, this code generation is very fast, it still takes a small bit
of your startup time. And as we started using the startup time using other optimizations,
the contribution of this of these will inflate. So, we are also planning or we are also working on
storing these, these one time generated code into the UT cache. That is all good when
vendors that arrive. So, as I mentioned, the add-of-time code, add-of-time class loading and
linking feature is already available in JDK 24. It has already been delivered in the main line.
The other two pieces, add-of-time method profiling and code compilation, they are expected to be
in JDK 25. But you do not have to wait until these releases, there is a light and a build
available. And for those who want to build who are adventures enough to build their own JDK,
you can compile from the light and repo. There is a pre-main branch which has all these features.
So, yeah, give this a try, try running your application and see what kind of improvements you can get.
That completes my talk. Thank you.