I'm going to do the introduction to this might now.
We are still in the deep diving go track and you heard we talk about garbage collection
a lot which reminds you remember if you have garbage you want to collect the garbage
bin is there.
We are a garbage collected language I don't want to see any garbage at the end of the day
only if you do reference counting.
But we are going to talk about going easy on memory and Samar here is an amazing person
to talk about this well with a very cute graphic here explain us how we can treat our
memory of our servers and our computers way better run for plus.
Thanks guys thank you I'm really happy being here.
So my name is Subagip I am a software injury and a full-time dead of three monsters
basically I contribute to open source as much as I can but these days it's a bit hard.
I'm mostly interested in the observability, distributed systems and databases I have spent
like all my life coding basically I'm a systems guy before go I was just writing profilers
in Python as a sea extension but we as a company decided to implement a continuous
profiling ingestion pipeline and go length we turned out to be an excellent idea.
So my main motivation for this is for this talk is that there is tons of material around
like how garbage collection works like what types of garbage collection collectors are
there but there's less emphasis on writing garbage collection and weird code which means
like I feel it like when you're writing code you will know like what kind of tricks you
have in your bag like what should you do to write more efficient code in terms of garbage
collection and even though the topic is very language-agnostic like you mostly see the same
things in Java as well.
I aim to be practical as possible and let me first start with some real work data so this
is one of my favorite conferences and Pekka he is the CEO too so it is a horizontally scalable
SQLite database and mentions about like avoiding dynamic memory management is the only
thing that matters for low latest it's kind of a bolt statement but I think really it's
after some point it becomes true because you optimize all the low hanging fruits in your
code and maybe like do lots of optimizations in your algorithm and data structures but
then what your left is just memory management and you need to do much better in that area
as well and yeah this talk is from brands like previous talks if you I'm not sure if you
can see everything but view works is horizontally scalable database and it is like if you
can some of these numbers you will see that half of the time is actually spent on garbage
collection and this is from a production system and one other example is from this talk and
receive video and this is I think this is a perfect put and if you see the first function it
basically takes like 40% of the time and it's just runtime gc bar it's basically the marking
process marking gorytine of the garbage collection and if you want some more this is from our
ingestion continues profiling ingestion pipeline here is the like half of the time is actually
the handler code the ingestion code and you can see that 25% of the time is actually spent on
garbage collection as well and this is a recent data dog blog again this is a metric database
from from day by the way the the code that I showed you from Uber was M3db which was their
metric database as well and this is from this is a recent blog post from the data dog they switched
to us because 30% of resources on garbage collection and like there is this gc friendly libraries
there are lots of them like go or 0 log go reflect all of them are just advertised there's
themselves as being zero allocation and that kind of stuff and the people just download
it which means that there's there's a really neat for this and yeah I don't language I'm not sure
if you have ever heard the term like that spiral that that's I that was a term I heard a lot
in java there are that was you guys sitting behind me not myself but the that spiral means
that your application is basically doing nothing useful just trying to keep up with the pace
of the allocations so I'm going to have a very overview of very I'm going very fast here
to just mention a bit about like how memory works in go lang basically we have two types
stack and heap and the both are on memory of course and the stack is basically pre allocated memory
dynamically grows like it basically starts with one megabyte by default for a go routine and then
it can dynamically grow it is fashion because it is when you eloquate it's just incrementing
and decrementing a pointers so it's very fast because it's known the access patterns are very
known and the faster access because you most of the time when you enter a function you access
variables most of the time the same stack will use that means that your by nature your code
your stack memory will be cash friendly because it will be cashed in L1 L2 whatever and it's
managed by the compiler in go which means whenever you compile the binary the stack memory will be
known and the heap is dynamic allocated and it's managed by garbage collection the only thing I would
like to say is that whenever a variable escapes to heap you will you can see it as a pressure
to garbage collection and one more thing I would like really would like to mention is that like
it is the stack is cash friendly so it's mostly in L1 so you're mostly accessing the same
variables again and again which puts them closer to the CPU that's how CPUs work and if you look
at the numbers you can see that accessing RAM is much much worse than just being in the cash
so and these numbers are usually hidden you cannot see them in your CPU profiles whatsoever
so what I'm trying to say is that do your best to be in stack if you've wild writing code if
there's a possibility you can put that variable in the stack please put it in the stack
and to move before moving forward into the tricks I also would like to mention a little bit
about like understanding garbage collection so how garbage collection works before that I just want to
say that it is a very complex software and it is probably do inherent wide range requirements
of all the applications it need to support the allocation rate of an application can be very high
the volume can be very high there might be few goretings like there might be millions of goretings
running in your application there's fragmentation issues it needs to consider and there's also
the pacing like it needs to basically say no to say no to allocations when the time comes
and it needs to do it in a minimum later otherwise people just switch to other languages like
rust so yeah and how garbage collection works it basically works in three phases this is again
like two thousand feet overview of this so I don't want to shame the people work on this one
because it is like an extremely complex software so in the first phase this is called the initial
mark phase there is this stop the world so this is the stop the world well let's stop the world
means is that it basically stops your all your application no nothing runs at this period this
is done because it needs to get a consistent snapshot of the variables that point to the heap
so it basically stops the application nothing is running it then works over the stack of your
goretings try to identify which which ones are pointers there's lots of optimizations on
in doing so but just I'm just saying in this initial phase it is stopping the world so effectively
it it means a lot for the for the latency and in the second phase it can then concurrently because
in the initial phase it eject it basically enabled some right barriers so what right barriers mean
is that in this concurrent mark phase when the right barriers are in place what you can do is you can
basically get updates or rights happen to those pointers and then you can understand you can
basically detect them you can think a white right barrier is a callback to your pointer change
so this this all happens concurrently and after like marking all the reachable objects in the heap
unreachable ones are just sweep which is the third phase of the garbage collection
what I would like to just highlight is that it is extremely not extremely but very unproductive it
might be very unpredictable like this because of the stop-to-world you basically stop the
application and the garbage collection is also very dependent to your application because it is
linear with the number of pointers you have in the heap so and it acts very differently on pressure
whenever you are limit in the pressure it acts differently it basically kicks in just more
and more and one thing that's overlooked is that CPU cash flushes that I just talk about before
garbage collection it is like this everything is green and everything works perfectly but after
garbage collection it is just empty because if you think about it garbage collection is kind of a
software because it basically traverses all heap it is kind of very cash unfriendly if you think about it
so to wrap up the garbage collection can high impact we see that real work data makes it up
40-50% usually goes unnoticed and it can be unpredictable so let's keep garbage collection happy
okay I shamelessly copy and paste this from Brian's talk hello brand Brian and this is from
Histalk that like reuse reduce recycle this is an MRI and my motor phrase that is basically
just to eliminate waste and I think it's a very nice analogy for this one so let's start with
reduce basically so by the way reduce is not all about right reducing allocations or pointers it is
it is reducing size like like reducing any size of your memory always have compounding benefits
like I was a piting guy and I know that they have done for example from 2002 to 2007 to
2003 that 13 they have shrinks the base object base everything is an object in piting by the way and
basically they shrink the size by a 95% and they see like 60% on runtime without doing any optimization
that's how you benefit from cash locality and that kind of stuff so mechanical sympathy is very
important yeah I have written a simple blog post about it yeah so the first thing I would like
to mention is like the stack versus heap so whenever you return something some reference type or
pointer it returns it basically escapes to heap so you should try to be if it is possible you should
try to try to design your APIs such that you accept it instead of returning it like the
reader interface is a good example and I saw this example from Jacob Stalkier he mentions that
whenever you return it it means that the allocation is already done it escapes to heap but if you
accept it there is a chance that you can allocate this on stack and whenever you call the function
it might still be in the stack there is there is no rule on that so returning escapes to heap but
calling does not and yeah closure variables are tricky just if you if you use closure variables
just be cautious about them they might escape to heap. Interface in generating escapes to heap as
well because compiler doesn't know the type that's the size there's slower on hotpets please
prefer using concrete types so the the most important thing I think is this line here in
the Stalk is that avoiding pointers so whenever you write I was mentioning some mindset before
the Stalk I think it is this whenever you write the pointer you use a pointer just try to be
mindful about it because it basically means more garbage collection pressure it is linear with the
pointers and if you don't use pointers it can skip entire regions without pointers I'm talking
about the garbage collection it is more cash friendly to to be to not to use pointer by the way
and compiler generates extra checks like like if you panic for example you need to have some checks
in the compiled output and sometimes these pointers can get unnoticed like I didn't know before
this talk the time that time for example has a pointer inside so if you for example have a slice of
time objects they basically contain a pointer which means garbage collection pressure
yeah string time that time all contain pointers in self so careful maps with reference types
slice values string keys and many more so one one one technique that there is being used is that
if you have like this kind of struct like basically two integers region and tenon tidy instead
of just formatting them to a string and using this string as a key please just use the struct as a key
because you will be avoiding pointers for free and this this one is also interesting for me as
well I don't know if you remember that's just a Swiss map talk from Brian that just a previous
thought he was measuring the buckets like so if you allocate like a struct value or key
more than 128 bytes this means that the map implementation needs to allocate more memory and use
pointer instead of inlining the value inside the bucket but if you if you are smaller than
this special value you will be inline integral so there is no extra allocation you can
test it yourself just benchmark the code with the allocations you can see that it is different so
I will just head up for this one yeah one one more thing is that as guys coming from sea
world like me there is this myth of copying is expensive but if you think about it like in terms
of CPU it is it is a myth because basically like copying a 64 bytes is just the same as copying
a pointer because basically CPU and RAM just operates in cash line rate yeah I mean
prefer to use non pointer versions of data structures like linked list you would be amazed
like how many data structures can be just implemented without using pointers this one might not
be avoiding pointers but I just wanted to mention because this bit me in production there is a big
peeper of payload for we we used in our ingestion pipeline and we were cashing a small
struct inside it which was not a pointer if you do if you do this this thing I mean the big
struct will not get the allocated it is I mean it is obvious but just keep in mind while doing
these kind of things because we get in in the in the head and yeah it took us lots of time we
our application was out of memory like crazy so basically avoid holding references inside large
objects and the remember zero kitchen libraries use them they're awesome there is lots of them
and if you wonder how they work the main trick is just pre-allocating memory and use integer
indexes to reference objects so it's just basically another way of avoiding pointers that's all
and reusing okay reuse I think it should start with singtapool definitely because it's the basic
tool singtapool is again coming from a serial singtapool is not a fearless like you cannot just
put something and wait for it to be present whenever you request it it is different because
the values when when you put them in the singtapool they can be garbage collected and so the main
trick here is that if you would like to use the singtapool you should have allocations that you would
like to use reuse between two garbage collections cycles basically I say too because whenever you put
something into the singtapool it stays one more time it's persist to the garbage collection
cycle one more time and one more thing is that it is defined and singt this one yeah it is very
simple thing but I mean if you think about it sometimes you don't see it so it's not in
memory that pull or something like that it's singtapool because it is very very optimized for
concurrent access there is no looks inside the implementation of the singtapool there is a very good
blockpost about it from the pictorial matrix I just read and basically what it does is under the
hood in the runtime it has a go routine caches basically uses very small caches but it's somehow
managed to do it without using any mutex or like yeah it is very useful but a bit misunderstood so
you need to understand very well one of the tricky things that be careful with returning non-pointers
why because so if you look at the example in the in the red here if you return a slice header
and return this one like that what happens is that when you put it put the buffer as a slice
the as the put accepts an interface there is a conversion so the the buffer as the buffer slice
other escapes to heat which means that you have another allocation unnecessary allocation so
so the the reason you basically use singtapool is that reusing not allocating but you are
allocating more so this is exactly the opposite and that is why you optimize for allocations but
introduce one for for the next cycle and yeah this might be hard to have a support on production
and that's why we have this static check and please just use static checks in your code as well
and sorry like if you read the static check when passing a value there is not a pointer to a function
the value need to be placed on the heat which means additional allocation and this is the reason
and I'm not going to too much detail on this one this is very simple I guess like if you have any
chance of like you allocate basically a slice of numbers and if you have any chance of reusing it please
reuse you can just point to the start of the slice by this trick like a pan trick and please again if
you just allocate a big or small it doesn't matter a map or slice please a pre-allocate because
if you let the dynamic if you let your slice or map to dynamically grow it it might lead to
fragmentation as well and yeah strings that build their bytes but buffer like all the stuff
no they're optimized for in not making no intermediate allocations as well oops okay recycle
so for me recycle is all about like tuning the garbage collection tuning the parameters
and if you look at it like I really think that Golank has Golank team has done great job at just
abstracting all these garbage collection details into two basic configuration parameters
one of them is the Golk gc which means that okay like if you set it to 100 person for
example what it means that I have this the current heap it is x bytes whenever it becomes like it
doubles it becomes one person more sorry 100 person more just trigger the gc that's how it that
way what it does and the go map limit is basically saying that okay it is the it is the hard limit
for the go run time do not do not get this threshold but it is also a nice thing it is also
a soft limit for the operating system because you would like to avoid out of memory errors because
whenever out of memory occurs it might end up very weirdly this gives you a chance to set a
proper value without out of memory and the go run time can act be like agree and by the way
when garbage collection near this limits it may be near this go map limit it acts more aggressively
try to reduce there is a nice tool that show you can just select go gc memory and play with it
okay two solar reef I have a few minutes left so this is overview for reason I think go is
unparalleled when it comes to observability it is like in the runtime you have lots and lots of tools
I'm not going very detail on them there's tons of material I just would like to mention like
profiling memory you can basically use heap life objects size to debug memory leaks you can use
allocation count on per function and line level by the way line level is also very interesting as
well you can basically observe allocation frequency and maybe you think that pull and this is again
like an older I shame this I was really really lazy to just do it myself I used Ryan Stalk again
but this is line profiling I'm still amazed by it you can do line profiling and it shows you
per line allocation information and the escape analysis again try to be I said try to be on the
stack not heap so this is your tool to do it if you run this tool it basically compiles and
basically says you I will be on the stack or not so you can use this tool to decide if you're
on the stack sometimes it's not easy to spot and one tool that I'd like to mention because I
feel it's kind of underrated execution trace is one of them I feel that it is it is the most
cinematic visualization what I mean by that is that it offers the most realistic view on what happens
inside your application because it shows the time and it shows the important events happening
in your application it can be look it look acquire it can be garbage collection whatever like
you will see a timeline of events happening and you can also see the garbage collection
pressure and phases and the latency like stop the world every every phase that I just mentioned
you can see and visualize it and yeah you can debug look condition issues as well and it is
safe on production kind of safe why I say that because we with one that 21 overhead drops the
one to two percent of time thanks to kudos kudos to Felix Gaysander for who works on these
kind of topics basically they optimize the stack online encode to drop it from 10 percent so it is
kind of safe to use on production so yeah this is how it looks if you have haven't seen already
and for me there's one more M1 environment where I will that you said like this but it is for me
like kind of a CLI way of doing execution tracer I just would like to mention it
okay just I would like to just wrap up reduce preferred stack over here if you can avoid pointers
make it a habit I mean there's there's no way you can avoid pointers I'm just saying be in the
mindset of that they put pressure on garbage collection use sparingly I mean use interface and
generic sparingly sing pull is your friend but understand it very well reuse pre-allocate maps
slices whenever possible and usable observability tools I think anytime spent on using observability
tools is just a well time spent I mean profile benchmark execution tracer and one bonus one I see
the morning talk from Martin that about memory just I was very I didn't decide since one hour ago
I should put this or not but memory just like before this one arena was proposed which is to just
a way of deciding you're like implementing your own stack in some way like basically you you say that
I don't want to involve garbage collection anyway in this namespace and regions are awesome and
they're better than arena because arena just found to be less ergonomic in the in the previous
implementation but I don't know it I think it will be good so it and it was minimal garbage
collection impact as well with that thank you very much