So I'll new go for us. We are going to it using Go. And we have some very nice performance
things to do. I find it Alex Ober, Alex is actually also managing a deaf room today
and he just left his completely busy room to talk about how to use P going Go, so an extra
hard of much room. Thank you. So I guess can we can start. Before the tour, I wanted to
ask how many of you actually know what profile guide the optimization is. Great, wonderful.
I like to skip through slides. So the few words about me, I used to be a C++ engineer
right now. I'm a Rust engineer. Keep in mind, I'm not a Go engineer at all. I wrote several
lines of gore and I'm having that. But I think my point of view will be interesting because
PGO in Go is kind of special compared to let's say, a more mature PGO from C++ in
the Rust world. It's kind of special. Regarding my interest, I'm interested in compilers
stuff. I was hacking LVM. I prefer more LVM stuff compared to GCC. I'm also in PGO
offer, this is a resource that's trying to collect as much possible information about PGO
mostly from C++ in the Rust world. But I've covered it as a Go ecosystem too. And I'm
a software performance developer organizer this year. So let's start. A bit fury about PGO,
but we can skip it. Great. So we have a Head of Time compilation model. It's pretty simple.
So our score is binary output. It's executed in target machine with regular one. So unfortunately,
we have a just-and-time compilation model too. So it's a Java V8 stuff. So they have
a possibility to not simply compile to some kind of byte code, but also execute with the byte
content target machine. So just in time, it has a really huge advantage here because they can collect
runtime statistics on a target machine. How our code is executed, which parts, which functions
come frequently, et cetera. And fortunately, in the Head of Time world, we don't have such a privilege.
We need this information because we have a lot of optimizations, which are dependent
can be improved by providing to them around time statistics. The most important one is in Lightning
is important in every compiler, including the Go one. And the solution profile guided optimizations
so we just collect runtime statistics on how it's called PGR profile,
pass the profile to the compiler, use the profile during the compilation phase.
And it's it, our binary is faster, if an asterisk, because it's not actually true for all cases.
It's first we need to understand we can optimize if PGR for now only CPU intensive stuff
and stuff that is not already optimized. Whereas some developments,
really fresh developments, less than a week ago, about applying PGR for GPU stuff, device GPU
stuff, but it's highly unstable and available only in LLVM. And okay, our binary will be pretty fast.
So, if you want to know more about PGR internals in the Go compiler, in the main Go compiler,
highly recommend the stored from Go for a coin UK. And I highly recommend to ask questions to
this guy on GitHub, in the Go repository, because actually it was the only person from the
Go compiler, the team who answered my PGR related questions well from my point of view.
So, why should we, we should care about PGR? Most of these benchmarks are not go unfortunately,
unfortunately for you. But it's the reason because it's simply much easier to find
ready benchmarks for more mature PGR systems. Let's see it. And most of these benchmarks actually
all of them, I did myself. So, I'm pretty sure. You can expect almost similar results from the
PGR applying PGR in the Go ecosystem, but keep in mind one thing. Go compiler, implement,
less less aggressive optimization compared to C, C++, RUSTF, or GCC, or LLVM,
a consistency, because if you've seen at least once the amount of optimization passes in LLVM,
you understand. So, there are a lot, really, a lot of optimization since I told LLM.
So, let's talk about PGR in Go. In the game, it was implemented firstly in a 1.20,
previous 1.21, it's globally available. Much, much later compared to C and C++ world.
It was implemented by Google and Google also is the offer of the same thing PGR, which is
frequently called as autofidio or simply PGR. It's just same thing PGR, nothing special, that's it.
Google implemented this kind of PGR for exactly one reason, because they wanted to collect
PGR profiles directly from production environment. The regular ones, the default one PGR mode,
instrumentation mode, gives you an ability to collect much, okay, more precise PGR profiles,
but in the cost of much, much higher instrumentation costs on the run time.
If you hear that you cannot run on production instrumented binaries, let's not completely
true because they actually answer it depends, it depends on your requirements, etc. But the
go engineer decided you don't need it, and simply PGR will be enough.
And the consequence of that, that Go compiler misses a lot of interesting PGR modes,
instrumentation, instrumentation kinds. For example, Varra, a lot of instrumentation flavors
before translating to IR, after in-lining phase, combining sampling and instrumentation at once,
trying to fill gaps in the PGR profile by passing it to some machine learning models.
And one of the fancies PGR kinds is a temporal PGR that optimizes not a run time
speed of your application, but a start-up time code start. It was implemented by a meta for
mobile devices. It's not available for Go. So unfortunately, only the main Go compiler supports PGR,
LLGo, the LLVM-based compiler for Go doesn't support PGR at all, and even
there is no any request to this. I forgot to report it to the upstream. And if you hear about
TINIGO, so subset of Go for Embedded, I post it TINIGO also doesn't support PGR.
I left a request, I guess, one year ago also in the upstream and the developer said, yeah,
great idea, give it a try. Okay, that's it. So, I don't care much, sorry. So, and another interesting
detail that Go compiler reuse the PPR off ecosystem for gasoline PGR profiles, it's a huge
difference compared to C++ world because they implemented custom profiles that are not
compatible between compilers, you need special tooling, which is called out video from Google,
this tooling is not so good, it's not so easy to build on your machine, it requires specific
version of LLVM, you cannot build it on the latest LLVM version where I lot of stuff. So,
from this point of view, PGR in Go is much better.
Actually, you can try, so if you are not happy with TPR off format, for some reason, for example,
you have your own proprietary open source profiling stuff profiling a system which was developed
before PPR off, you can try to convert your profiles into the PPR off compatible format,
but it nuances. Whereas a artificial documentation by the way Go documentation about PGR,
I would say one of the most user-friendly in the ecosystem not only in Go, but actually for
in the world, PGR ecosystem because for C++ world PGR, the documentation is terrible.
It was terrible right now, it's simply bad. So, and Go developers really tried and they did a good
job, trying to not only describe what PGR is, that's what we usually have, but how to use
in practice, and a huge kudos to them really. And the second interesting detail,
it's what designed PGR mainly for service like workloads. For example, if you have a binary,
if you buy your HTTP handlers, they are running continuously and you collect profiles from a PGR
from time to time, if some frequency. In C++ world, the instrumentation was the first PGR mode,
and it didn't have such limitations, so usually you got a PGR profile at the exit of your program.
You can highly customize this behavior, but it's pretty undocumented. You need to read a lot of
DM sources, et cetera. Compared to Go, once again, Go, you can just
at a package profile, HTTP, and you will get HTTP provided profiles all simply
right and mainly, like a gather and dump somewhere to the file. So, let's talk about interesting
possible PGR issues that you can meet in your PGR journey in Go. So, at first,
mismatched and outdated PGR profiles, official documentation says that, but they choose,
they work very carefully. So, that if your profile is kind of outdated because
new code is added, so it will not be covered by a PGR existing PGR profile, or some code
that is allergic covered by PGR profile is changed. So, your profile.
Okay, backup, yep, backup is fine.
Yep, yeah, a couple of last engineers here. So, Go, Go implementation of PGR is pretty stable
and sustainable to this changes. However, I still recommend you to avoid problems if the
stale profiles just try to regenerate them as frequently as possible and that's it. You will
eliminate a really big amount of potential problems with possible regression and performance,
et cetera. Please don't even try to play with these things because it's really hard to back them.
I would say so. And another interesting thing, C++ ecosystem gives you an ability.
When you provide a really outdated profile to the compiler, you will get at least from LVM
a system, you will get a lot of warnings mismatched profiles, something like that.
Unfortunately, such functionality is not available for the Go compiler yet, there is a request
on the screen, but it's not implemented and there is no activity. So, be careful.
And regarding performance regressions, there is an interesting note in the
documentation that you should not expect performance regressions from the wrong PGR profile.
I was kind of curious about such a bold statement, I would say. And of course, it was a lie.
Because there is an issue in the upstream, when a person provided a PGR profile to the workload,
and the resulting binary was performing worse than before PGR.
And I would say it's kind of a problem for the whole PGR ecosystem, not only for Gore, for one
really annoying reason. If you met such a thing, what do you do?
You can try to try to regenerate PGR profile, okay, let's say you did try it and it didn't help.
What else? You as a good engineer are going to the upstream and report it, and they will try to
so they need a reproduction, they need your court, minimal viable product type, they will need
probably your profile or better benchmarking profile gathering scenario. You will need to provide it
once again, it's a very difficult job. And even if all of this information highly likely,
your issue will not be resolved. Because they have much, they have a lot of enough work to implement
it, they go compiler and debugging performance regressions from PGR in your specific case
is not a very fun job to do. We have the same problem in LVM and GCC, I reported such bugs in
these ecosystems and I got exactly the same answer, so silence, so that's it.
And debugging yourself is a really challenging thing, and I wanted to highlight one thing regarding
the one-shot utilities like CLIs, for example, log, generator, etc. I, so remember the service
like or end of PGR. So if you have one-shot utility, for example, you start in it,
it's running, for example, for 10 seconds, and it's finished. How to collect a PGR profiles
from this workload? Your utility doesn't have a HTP handler, it's not a service, it's like
a one-time utility. Unfortunately, I was, I proposed an idea about some kind of automation,
regarding dumping a PGR profile like counters, internal accounts, etc. at the end of the program,
how it's done in the C++ ecosystem. This idea was rejected, and the main motivation
of all the links, the public, of course, in GitHub, the main motivation is files I understand,
they don't actually care as much about such workloads. I cannot blame them because we have
a lot of us writing a lot of you. Writing goes services and go, not like utilities, but anyway,
if you want to dump PGR profiles, you will need to implement it by yourself. For example,
once again, example from the documentation, you will just need to use the Proof ecosystem,
and write some code manually. Compare it to Rust ecosystem, for example, in Rust,
there is a really great,
you can get a copy of it because as you know, it will also form an app as it grandsons.
To begin with, this recommendation mandatory, so for example, the pres yours there.
in variable variables, and possible compiler switches.
Unfortunately, this use case is ugly in every ecosystem,
because it's multi-linked language build.
Multi-language build, you will usually have a build system,
which is oriented only into one language.
Cargo in Rust, whatever in C and C++.
They have a bunch of them.
So whatever, it's always ugly.
Unfortunately, no build system can really automate well.
This use case for actually,
a lot of interesting reasons.
For example, so how to implement it manually.
You will need to pass manually to the C compiler.
For example, if you see dependency,
all corresponding pj of x.
And here you go.
Different pjomots.
You will need to read the old documentation,
regarding pj about your gcc or client compiler.
And remember, this documentation is not that good,
especially in gcc part.
You will need to choose the right pjomot,
because we are a lot of them, especially in LLVM.
And you can have really interesting combinations.
For example, sampling pj of as a go part,
and instrumentation pj of as a C part.
And of course, pj of profiles are not compatible at all
between compilers.
Gcc or LLVM use their own format,
and pprof is not compatible at all.
You will need to get this file from the C compiler,
prepare it in the right way.
You will need to learn right tools,
especially for gcc, they are not a three-wheel.
And pass in the right way to the compiler once again
to the C compiler.
Of course, all of these things should be done via
CGo environment variables.
I wouldn't say it's impossible to do,
but it's steady of styles to do.
It's manual labor.
And believe me, a lot of likes,
it could be a problem.
And if you want to do it like a reproducible way,
like write a script, believe me,
such a script will be really ugly.
For example, if you're trying to package
writing a resipe for a go application
for native dependency,
and you want to write a routine with
optimizes with pj of all the application
during the build process of the package from this resipe.
As for example, a regular use case for pj
optimised application, for example,
for RSI or for Clank, how they are packaged
on the distributions, especially
performance-arranted distributions.
Regarding pj of the go-sego,
I actually tried to find an information.
And I found only one topic at the RAST forum,
and one person actually asked exactly this use case.
And as you see, not so many people were interested
in the topic, so automatically closed
where not interested.
So that's it. That's all information.
So you will meet a lot of interesting
traps on your journey.
So reproducibility and pj,
that's really a huge topic.
For example, when I was talking
about any person like maintainer from distribution,
it's their first question.
How to make reproducibility when pj is enabled?
So here we are actually two dedicated cases.
The first one reproducibility saved pj of profile.
You dump a profile, committed to a VCS.
For example, go once again, did a great job.
There is a standard default dot pj.
We don't have such option in c++,
and we really suffer a bit from that.
And reproducibility pj of profile generation.
So the first case, just saved profile,
but please keep in mind, it can be stale.
We need to regenerate, some frequency, et cetera, et cetera.
Like outdated profiles.
The second case regarding reproducible pj of profile generation
is impossible to achieve.
Don't even try, because it's required
deterministic execution of your application.
And it's, so if you're talking about huge applications,
it's not a viable option to achieve.
So just don't waste your time.
On it.
Regarding saved pj of profile, there is another thing.
I met, for example, when you find an open source project,
for example, this project is file d, file dot d,
that's like a lock processor written in go or buy a zone.
And I found this project, and I found default pj of,
that means that project is optimized by pj of,
and I asked this question.
How did you collect this pj of profile?
The answer was science.
So when you try to build this application,
this profile will be automatically applied
by the go compiler, because it's default dot pj of,
but you don't know the scenario
where this profile was gathered.
There is no information to get it,
except for asking maintainers.
Actually, I found this guy on telegram chat
in some local group, and asked,
if I should remember, asking directly,
and he said about this scenario, et cetera, et cetera,
but no public available information on the internet yet.
Keep it in mind when you see any open source project
with provided pj of profile.
So it's scale, a bunch of additional issues.
You need to collect profiles from hundreds of services
from thousands of machines.
I need to implement proper gathering,
symbolizing store, cleaning routines,
for all of this pj of profiles,
because you will have thousands of them.
If you are talking, especially about large fleets,
like for large big tech stuff.
And of course, you need to make this professor,
you're a robot, a robot, et cetera, et cetera.
Tracking for all of his profiles,
skew, and how they are outdated,
for sure services, raising arrivals,
but we're talking about actually
a pretty strong enterprise here, et cetera.
You don't want to implement all of this thing
from this scratch.
Luckily, we have several solutions.
Let's not our way using pj of manually
at such a scale.
It's not a viable option, believe me.
There is open source solution.
Go or again, that one is parka, parka.dev.
It's open source one.
And another one is solution from Yandex.
Yandex perforated was an open source
one exactly one year ago,
one day before the previous was them.
And there are actually possibilities
to extend our open source solutions,
like Grafana Pyros, copetanlas,
or Fireing Platform.
I opened request to them,
regarding extending their functionality
for doing pjore silenced in both cases.
So, of course, if you're a large enough like Google,
or whatever big tech you can write,
you own profiler, and that's a viable option.
For example, Google, Google White Profiler,
a zone vision at a zone,
and many, many other systems at profiler
is very, very low data,
also has a linear one, probably ready to do it.
So, however, if you decide to use
with solutions, with solutions that kind complicated,
to admin, because it's a large scale.
Stuff, you need to deploy several parts,
you need to,
admin different database,
you need to monitor them, etc,
for example, object storage,
free open source stress, probably safe,
so if you produce, it's not far.
Fun job to admin it,
that's the same actually picture,
but for young expert for it,
once again, postgres, click house,
it's really compatible object store.
Okay, that's not that fun,
but you need to do it.
If you want to use it.
Let's quickly compare,
because I don't have much time,
for minutes.
So, both projects are alive, great.
Unfortunately,
here for it are written in C++,
because the index uses many C++,
Spark is written in Go.
So, we will decide to extend,
a Spark is your way to go,
please don't type C++.
Licenses are both fine,
apashlicness.2.
The commutation, actually,
I would say, both could be improved,
but Spark doesn't cover the SPGO part,
of the functionality at all,
only one commit in the git commit history,
where they said,
we added end to end test
to cover the SPGO functionality.
Let's see,
young decks cover a sampling PGO,
very well, because they use it.
Payed support.
If you're interested,
if you're afraid to know
and they are not interested,
I'm pretty sure,
Spark, yes, they provided.
PGO support for GORE,
it's an important part.
Spark supports it.
It's written documentation.
Perforator supports it,
but in theory,
because they didn't try to use it in production yet,
because most of the service of written in C++
and C++ is extensively optimized by PGORE
from perforator, go,
not yet, but we want to achieve it.
PGO supports other languages,
Spark, no,
where is an issue in the upstream,
no activity yet,
if I say, no, we don't plan to implement it.
Perforator, yes,
and they are interested in it a lot,
especially in native languages,
because once again,
most of the service of written in C++,
and the way you will support
it definitely in a good way.
So, in the last word,
one minute, about PLO,
what is it?
Perforator supports it,
but with nuances,
and Spark doesn't.
PLO, it's actually a PGO on steroids.
It's an optimized binary,
even after PGORE,
the most important optimization
is the re-ording function inside the binary,
for making your CPU instruction cache,
because we'll be less cache-missa,
instruction cache-missa's during the execution.
Available, open source tools at the moment,
LLV and both the main one,
Google Propeller,
is a second one,
until finally out,
optimizer,
archive, to rest in peace.
So, a performance impact
from using PLO for GORE,
is really huge.
You can check,
Huawei implemented this functionality,
but unfortunately, they have for you bad news.
Upstream,
didn't agree to accept
this change into the goal-linker,
because it requires some additional
immaterial occasions and changes
since some goal-linker,
and Huawei guys just decided
they are not motivated enough
to push it into the upstream,
and that's it.
So, go a system,
doesn't have a PLO,
C++ has.
So, summary,
PGORE is great,
local PGORE optimizations.
That's it. Thank you.
Thank you.