Thank you, everyone, for coming to this.
The objective of this talk is to give a few pointers and
Heidi on how you can maximize performance when using
GRPC and specifically the go implementation of GRPC.
A lot of people are already familiar with GRPC, but I will just make a few
basic introduction to reintroduce and re-explain what it is.
GRPC is an RPC protocol made by Google,
VATTEX is a team of first approach.
So you first need to define your service specification in a
Protome of fine.
You will list all of the end points that exist on your service.
All the message type that they take in input and that they output.
From this specification, you can then generate
Client and server code in any language that you want.
It means that with GRPC it is easy to have
Cross language support to have one client written in one language
talking to server and another language without having to do
much work.
GRPC is based on top of HTTP2 and it uses binary encoding for
the payloads.
It supports either unary RPC or streams.
So let's have a look at how it would work to do a unary
request.
I define in a Protophile my service that has a single endpoint
Create user.
It takes a specific message type in input.
Create user request and return the specific message type in
Output.
From this Protome of specification, I will be able to
Involve the Protome of the Protome of the Protome of the
Compiler to generate a associated go code.
And so it will generate a go script with all of the fields that
have defined in the Protome of the definition.
With this generated code, it is easy to implement
The server itself.
I just need to create a type that has a method.
Create user that has a signature where it takes my message
type in input and that returns the message type in Output.
The implementation here is very dumb.
It's just echoing back all of the field of the request in
the response.
And we have this implementation.
I can have a quick benchmark to see how efficient that is.
To do that, I use the standard go benchmark tooling.
And I create a benchmark function.
First, I will set up the server.
I will bind to a local response.
I will create a new JavaScript server.
Attach my implementation that I should previously
To it and ask it to set request.
On the second hand, I will create the client.
I attach to a specific local response and create a client
object from it.
With this client object, I can then benchmark the
act of calling client.create user on the client object.
And while this looks like just regular function call,
it is actually doing a remote procedure call and doing all
of the following steps.
It will first marshal the request object to transform it to a
byte array that is the wire representation of it.
Send that to the network connection.
The server will receive it.
You will get back again the request object.
Execute our handler implementation.
Which was the basic figure that I showed earlier.
Then it will get a response object that it could then marshal
to a wire representation.
Send that for the network.
The client will receive it.
And finally, we will unmark it to get a response object that
can be returned from the function.
And we can run the benchmark to have a rough ID of what is
the performance of basic implementation.
And it takes roughly 36 microsecond to do all of these steps.
And so the overhead of a grpc-pc-pc-pc seems to be around
36 microsecond because I get nothing interesting in my handler.
And it's actually not that slow, that's good.
But it's actually possible to reduce this time.
And if we look back at all of the steps that were taken in
or benchmark, a lot of them are about marshalling it
and marshalling object to their wire representation.
By default, what the grpc-library does is that it uses
a reflection.
It iterates over all of the public field of the goods
tracked and it will marshall them one by one.
And while this works, a reflection isn't the fastest thing
So it is possible to change the way
change the codec that is used to marshalling a marshall object
to use a different representation that may be faster.
To do that, grpc-go as a library exposes a codec interface
that has two method, marshall and a marshall.
And it is possible to pass a user-defined codec implementation
to do this transformation.
And it's not that hard and it's possible to make it faster.
And to do that, I will introduce the vtprotabuff plugin
that we'll help at doing the far further marshalling and marshalling.
This is an open source project and it's a protocol
with a compiler plugin.
So it takes your protocol definition
and it will generate additional code.
Using it is fairly straightforward when on your compiling setup
where you invoke the protocol buffer
specifically the go plugin and the grpc plugin
we just add one more to use the vtprotabuff codec.
The vtprotabuff plugin.
And when we generate the go code from our protocol,
it will generate an additional file called my service
and the score vtprotabuff.pb.go.
This file will contain additional method
on top of our ghost track that represent our protabuff object
and this method can be used to implement a custom codec
that was faster.
So we want to implement a codec that has a
method and a marshalling method and we will create
such type and we will delegate the actual marshalling
and a marshalling to a specific function
to a specific method of protabuff go types,
specifically marshalling vt and a marshalling vt.
These two methods are generated by the vtprotabuff plugin.
And they contain a hand-rolled implementation of the actual
marshalling and marshalling, but do not rely on reflection.
So it means that we don't have to use reflection
and that this implementation is way more friendly
to the in-liner and to the compiler.
Once we've defined this type that implement the codec interface,
we can tell the grpc library to actually use it.
This can be done at any time here with anchored in the
register codec or it can also be done when you create a grpc
server object or a client object.
And with this change we can run again or benchmark.
And while it used to take 33 microsecond to actually do the whole
herpc sequence it now takes around 33 microsecond.
To ensure that we have a real difference and it's not just
noises in the measurement, we can run the benchmark several times
with a dash content option and save the output in a file.
And then we can ask the bench start to compare the result that we got
and bench start bench start will do statistical analysis on
those results to confirm or deny that there is indeed a difference
in our two implementation.
And in this specifically it confirms that there is a statistically
significant difference and that with the vt prototype of
codec we use 12% less CPU time when actually performing
or benchmark.
So with very little code here we just had to enable the use of one plugin
and make a simple codec implementation.
We managed to reduce a bit overhead of the grpc ecosystem.
And there was no need to change or hinder implementation.
So this was for unary request.
Now let's have a look at grpc streams and specifically grpc streams
will send a larger amount of data.
In my service definition I have two rpcs,
put where we will send the stream from the client to the server
and get where we send the stream from the server to the client.
In grpc stream is just a sequence of prototype messages.
And specifically here what we consider to be a message in our stream
is just a chunk that has a single field which is a bite slice.
From this definition we can have we can write a basic implementation
of put on put we just iterate of the stream and
stream that we see on it up until the stream gets closed.
And on get we generate a random bite slice and we send that to the stream.
So we will work with chunks of 16 kilobytes and for total stream size of 16 megabytes.
And so we can have a look at how fast that basic implementation of grpc streams
would be. This time I will not choose the go benchmark set up.
I will have separate the client and the server and look at
system metrics coming from that explorer as well as go runtime metrics.
So when we run our grpc server implementation and we stress it with
a gats week. So sorry our test set up has two virtual machines.
One for the client one for server each of them have two virtual CPUs and they share
a five gigabit per second network link.
So when stressing the grpc server with gats we can see that we manage to saturate
this five gigabit per second network link and that to do that we can
see one from point three CPU core.
So that's good or basic knife grpc implementation can saturate
kind of high-performance kind of high speed network link but it costs some CPU.
And if we compare with another project for example kd which is a popular
ktp 2.5 server with an in go we can see that kd is able to saturate
the same network link but for all these 0.2 CPU core.
So there is a lot of this dependency between what our grpc implementation
does and what kd which is optimized does.
And so even though grpc is a bit of more involved protocol
by an HTTP 2 it would still be nice if we are able to reduce the CPU
that we consume to do so.
And the problem is even worse when we try the put stream when we put
data to the server so when the server receives data it needs to consume
1.60 CPU core to saturate this network link.
So the question is where does where is the CPU consume what is our
CPU doing on this workload to consume that much power.
And the good thing about go is that answering this question is fairly straightforward.
You can just run the CPU profine on your rolling jrpc server to get an answer.
And if you do that you get this kind of frame graph.
There is one big tower in the middle which is actually doing the c-school
to read data.
So while it can be optimized I will not have a look at that today.
This is kernel CPU time.
But I will never have a look at other towers and specifically this one.
They all are related to garbage collection or memory allocation.
And indeed if we look at go runtime metrics you can see that
or jrpc server while under load allocates at a rate of 11 gigabit per second of
that time.
When we are serving 5 gigabit per second of throughput we are actually
internally allocated 11 gigabits of second per second of that time.
And due to this higher location rate we need to run gc very often.
During our stress test the gc was running between 4 and 5 times per second.
This is also metrics coming from this growing time.
And so while the go gc is kind or efficient and fast if you are in it several
hundred times per second when it starts to add up and to be a significant amount of CPU
type.
So now the question is where does this memory allocation come from and can we
reduce it?
Once again in go it is very straightforward to have an answer on where does it come from.
We can just run a memory profile on our running jrpc server and see where it comes from.
And specifically it seems to come from a single function inside the jrpc library called jrpc.
The jrpc that we see has two parts.
Receive and decompress that reads data from the network connection and put it in a
byte buffer.
So it reads one part of a message is located on the heap and puts the wire representation
on that place.
And then on the second part it unmashing.
So it tries to turn this wire representation to a good track.
So specifically when for the receive and decompress part by default every time a
prototype of message is received by the jrpc library it
evaluates a byte size and it writes that's wire format to it.
And so for every message that is received by the jrpc library we need to
heap allocate.
It would be nice if there was a way to tell the jrpc library to actually not heap
allocate for every new message that is being received but actually try to reuse data
internally to reuse memory.
And it's actually something that is possible.
You can configure the jrpc go library to do so.
If you are using jrpc go with a version that is lower than 1.66 there is an
experimental option that can be used when creating a server or a client.
If you use experimental dots receive buffer pull you will tell the jrpc library
to try to use a memory buffer pull internally when receiving new messages.
If you are using a recent enough version of jrpc go which I would strongly advise then
you actually don't have to do anything and it is now the new behavior by default.
With this memory pull we don't have to heap allocate for every new message that
is being received by the jrpc library.
This memory can be reused across different messages.
And so if we were to run again or benchmark setup you can see that our allocation rate is lower.
We only allocate at around 5 gigabit per second compared to the 11 gigabit per second that we had previously.
And as a result we consume less CPU time when serving the same amount of traffic.
So this was one half of the answer.
The other half we would need to look at how to reduce memory allocation when we unmasher.
In our basic implementation what we did was calling stream dot receive in a loop
up until the jrpc stream gets closed.
And what stream dot receive does is that it will return new heap allocated chunk
that has a single data field that is a byte slice that will contain what we send through the network.
Every time we call stream dot receive we heap allocate a chunk.
And we also need to heap allocate when we unmasher.
We need to heap allocate the data.
The slice that contains the data relevant payload.
We need to heap allocate it so that we can link it to the data field.
And it is possible to make a difference implementation of puts that do not have this issue.
To do that we will stop using the jrpc api stream dot receive.
And we will rather use the api stream dot receive message.
The big difference is that stream dot receive message takes a chunk as argument and will unmasher to this thing.
If we are able to pass a chunk that has a data field that is a byte slice of length zero
but with free capacity then we can unmasher to this free capacity without having to do heap allocation.
And to do that we will pull our chunk messages by using some VT protocol helpers.
When we get a chunk from the VT protocol pool there are two options.
I have of the pool was empty and we heap allocated new chunk.
The data field is the nil slice, length zero and capacity equals zero.
So when we unmasher we need to create a new slice and attach it to the data field which is exactly what was happening before.
Or if we get lucky when we get a chunk from the pool it is possible that the data field is a slice of land zero
but with some extra free capacity that was previously allocated.
And if it is the case when we unmasher to this slice we can just write it directly to the free capacity
without having to ask the runtime to do a new heap allocation.
And so with this thing we don't have to make a new heap allocation for every message that we receive on the stream.
And so by running again or benchmark we can see that with this change we have reduced dramatically the allocation of our GAPC server.
We now only allocate around 20 to 30 megabit per second of data.
One we use to allocate several gigabit per second.
Due to this change the gc rate is way more appropriate.
It only runs around one per second.
And so with that the CPU consumed when saturating the network is actually around 0.7 CPU core.
It's used to be at 1.6 so with this simple change we more than reduced and divided by 2 the amount of CPU that is spent on the GAPC in terms.
We can now have a look at the get workload.
That's the first from roughly the same issue.
On the get workload so when there is a stream from the server to the client it allocates a lot of the memory.
It allocates around 5 gigabit per second when we serve 5 gigabit per second.
And as we run this year lot it consumes a lot of CPU and it's the same issue as previously.
But this time the memory allocation comes from a different place.
All of the memory allocation comes when calling the marshalling to transform or go struct to their way of presentation and to send them free network.
The reason for that can be found in the API that GAPC library uses for the codec.
Specifically, marshalling takes a good struct and returns a byte slice.
As we may have shown in the talk previously returning a byte slice means that you need to high-pallocate it.
There is no way for the color and the color of this API to cooperate to prevent memory allocation.
So every time we need to marshalling object through this interface we need to keep allocate as much memory as it's why your representation is.
Thankfully GAPC recently introduced a codec V2 interface.
It's kind of the same thing as a codec interface.
There is still a marshalling method but this time rather than using a standard byte slice it uses a specific GAPC internal type.
called Mem.BufferSlices.
And what a Mem.BufferSlices is that it's basically a reference count at byte slice but the GAPC library can reuse and pull internally so that it does not have to e-pallocate it new stuff every time.
And so we can make a codec that implements this codec V2 interface.
Once again we have the help of the V2 prototype helpers.
To implement marshalling we will first ask what would be the size of the wire representation of this object.
We will then get a piece of memory from an internal GAPC memory buffer pool that can hold this wire representation.
Then we can just marshalling to the slice and return it as wrapped in a mem.BufferSlices type.
And this means that the GAPC library can now send this piece of slice of memory that contains the representation of a ghost truck.
And when it is done it can just decrement a reference count and if it switches zero put it back into the pool that can be so that it can be reused later.
So with this change we don't have to make a heap allocation every time we call marshalling.
And so it means that we can run or benchmark again and we can see that once again our allocation rate has been dramatically reduced.
We only allocate a few dozen megabyte per second when serving this gigabit per second load does the GC rate is way lower and we consume less CPU.
So as a small summary we've seen that it's possible to make a few changes to how you use the GAPC library to reduce CPU consumption.
For you now I request using the VT product of the codec helps you to remove the default implementation that is based on reflection and by doing so you can save around 10% of CPU time.
For streams on egress streams of stream moving out of the GAPC implementation you can reduce memory allocation and this CPU consumption by a significant factor by just using a recent GAPC version and having a codec VT implementation that do not hit allocate for every call.
Ingress stream you can also reduce memory allocation and CPU usage by using either a recent GAPC version or enabling an experimental option all the ones.
And if you want to go further you can change the handler implementation to pull the message that you receive from the stream.
Apart from this very last point all of the everything that I explained previously do not require you to change the actual GAPC implementation of your handler.
You just need to change a few lines of code on your combining setup on when you would how you generate GAPC from protocol and have one good codec implementation.
So with very little amount of work and code you can get a significant amount of CPU reduction.
Thank you for listening we have a few minutes I think for question guys.
I have space for one very short question.