Thank you very much. Hello, my name is Yoran Hadlan and you can find me as Jay Herland
on Github, Macedon, LinkedIn, elsewhere. I'm an Norwegian, but I'm currently living in
Delsen, the Netherlands and I work as a developer productivity engineer at Twig, which is part
of the larger modus creates of a consultancy. We help our clients solve our problems with
open source software and we contribute back to the open source community whenever possible.
Today I will talk to you about a somewhat obscure topic that lies between the dark arts
of the compiler tool chains, debuggers and executable binary formats. For the average developer,
I suspect these areas are often associated with a certain amount of black magic. Certainly,
I know that I had worked for many years as a C and C++ developer without ever encountering
today's topic as a possibility. I should start by saying that I have not invented any of
the features or things that I'm presenting here. I'm merely stumbled onto this while trying
to improve debug builds at a client. And I ended up writing a blog post about it to shed some
light on what's going on here and this talk is based on that blog post. So what am I talking
about today? I will give you a practical introduction to debug fission, also known as split
dwarf where dwarf is the name for the standard debugging data format used on Linux and most
of the unique systems. This talk is limited to L files on Linux and we will be using GCC
and the Google linker. LLVM plan works with debug fission 2 and there are also other modern
linkers that support the things cover here. A related topic that I will not cover here
is that I'll compress the debug symbols, but there are some links here if you're interested.
So let's quickly go through the basics. What are debug symbols? In short, given the classical
world program, it's a difference between this and this. On the left hand side, we build
with default compiler options and on the right hand side, we pass G to GCC in order to build
with debug symbols. On the left hand side, the debugger is not too helpful, although it knows
where the main function is and we can set a breakpoint on it, it is not able to give us
much more help. On the right hand side, wherever GDB can tell us what the main function
where it's located, online for in a load of CPP and we can even get source code listings.
The debug symbols are what provide these helpful links or references that map the machine code
currently being executed back into the higher level source code concepts like variables and
functions with file and line numbers. However, debug symbols take a lot of space. In the
build without debug symbols, our toy executable is only about 8 kilobytes large, but the debug
symbols add almost 300% to the original size. So what's actually being added to the executable?
We can use redelf with sections to look at our two executables and we will see that a bunch
of new sections that are called debug something have been added to the bigger executable. These
sections contain these debug symbols that the debugger can use to provide extra help when debugging.
So what's the problem with these debug symbols? Well, they're not only take a couple of
space, they also take extra time, time to generate the debug symbols in the first place,
but also as we'll see, time to copy them from the initial object files through the
intermediate build artifacts and into the final executables. Not to mention that this extra
duplication throughout the build process means that enabling the events symbols in a build
can dramatically increase the total space needed. In the end, you need to consider the benefit
of the debug symbols versus our cost. In other words, how often do you actually need the
debug symbols versus how much does it cost to build with them? And very often a project will
choose to build without the debug symbols by default and then it's after the developer to request
the build with the debug symbols when needed. But then also in many cases, you don't know
that you need the debug symbols until after the build is complete and the release has been deployed.
So let's continue by looking at stripped executables briefly here. Stripping executables is often
done as part of a release build in order to make the smallest possible release or effect.
We can see that applying stripped or executable with the box symbols creates a stripped executable
that is even smaller than the executable built with default options. But the result is completely
unusable in the debugger, of course. This time GDB doesn't even know that there is a main function,
and if I want to set a breakpoint, I need to use hardcode addresses. But GDB does have a trick
up its sleeve because if we have access to the unstrict version of the same executable,
then we can supply the unstrict executable to GDB with the symbol file command.
GDB will now use that unstrict executable to basically make sense of the stripped executable
that is running on the CPU. And we get our debugging conference back.
We can take this one step further and add a debug link into the stripped executable that references
the file that contains the debug symbols. In this case the debug link only adds 96 bytes
to the stripped executable and when this debug link is present, GDB will automatically follow
it to find the debug symbols and we get the debugging we want.
So maybe the solution is to build with the debug symbols and then keep both the unstrict and
the stripped executable. We can then distribute the stripped executable and as long as we can retrieve
the corresponding unstrict executable when we need to debug, we can get the best of both worlds.
This does sound like a good idea first. We get the smallest possible release or effect and as long
as we don't lose track of the corresponding unstrict executable, we also get the debug
is help when we need it. However, it's not completely without drawbacks. In order to have both
the unstrict and stripped version of the same executable, we must first build the unstrict version.
And as we saw before, building everything with the debug symbols is very expensive.
Then only at the end of the build process can we strict executable and make the final release
artifacts. This ends up on the critical path in our build graph. For any change we make to the source code,
we must now rebuild the unstrict executable and then strip it. If we end up not actually using these
debug symbols for most builds, then it can be very hard to justify this build time cost.
So to make this trade of more interesting, we should try to decrease the overall cost of building
with debug symbols. So what we really want to do here can be summed up in two questions. Number one,
can we somehow split off debug symbols while we are compiling? And number two, can we still
keep the separate debug symbols around for debug and time? So let's finally talk about debug
As compile time, you can supply the g split worth option and now in addition to the regular
object file, you will also get an accompanying dot to DWO file. Compare to the original object file,
we can see that around two thirds of the data has been moved into the DWO file and one third remains
in the new object file. Together, the total overhead is about 4% over the old original object file.
We can look close to the new object file with the debug dump and it shows that are now references
from this object file to sections inside the accompanying DWO file. And these are even loaded or
markedly by read-off. So it seems that the g split worth option has moved most of the debugging
information from the object file into the separate DWO file and put in references instead.
When it comes time to link the final executable, the linker now has to handle an object that is
only one third of the original size. This ultimately results in small executables and considerably
foster link times. So let's go ahead with linking. There are no special link options needed
for debug vision per se, but of course we do need to use a linker that supports it. Here we are using
gold. This new hello dot split executable is equivalent to the previous unscript executable.
The only difference is that it is based on an object that was built with g split worth.
Indeed, if we run read-off debug dump on the executable, we can see that the linker has carried
forward references to the DWO file that was created as compile time. Although not strictly necessary
for debug vision, there is a useful link option that is worth mentioning here and that is gdb index.
It is not easy to find good documentation on this option, but it's main objective seems to be
to speed of gdb when loading the executable and it's debug symbols. In effect trading link time
for debugging time. When we use this, we see that we actually save over 8 kilobytes in the final executable.
If we look at what happened to the L sections, we find that three of the debug sections in the
executable have been replaced by a much smaller gdb index section. I'm guessing here and I'm sure
someone in the room can tell me later what's going on, but I'm guessing that there was some
duplication between the remaining debug information in the executable and in the DWO file.
And with this option, we are able to eliminate this duplication at link time.
So let's take a closer look at the DWO file that we generated. Here we have only one
but in a larger project, you would naturally have one DWO file per compilation unit.
And a given executable might have anywhere between a few to several thousand compilation units.
Can we somehow consolidate this into a better debug package so that we don't have to carry all of these
DWO files around with us? Yes we can. There is a tool called DWO DWP that works like a linker
but only for debug information. The best way to run this is with the EXEC option, DWP will then look
at the DWO references in the given executable and consolidate all those DWO files into a single DWP package.
We can then remove all the DWO files and when we run gdb on the executable, gdb is clever enough
to find the DWP package with the same base name as the executable and debugging still just works
automatically. Finally, we have accomplished debug vision. What we have at this point is an executable
that we can easily debug and it's only slightly larger than a strict executable.
Most of the DWO information is now in a separate DWO package that we only need to have available
when we need to run the DWO. And we achieve this by splitting off the DWO information already
at compile time and linking it separately from the main executable. Now let's move on to
how we can enable DWO vision in higher level build systems. After all, very few of us build
our projects by invoking the compiler and linker directly. We'll look at CMake first and then
briefly at Basel. So here is a minimal CMake configuration for our toy project. We declare
minimum CMake version, a project name and we add our executable and the sources it is based on.
When we build this with CMake, we will quickly discover the CMake by default builds without
debugging symbols and it also uses a default linker that does not support debug vision.
So we fix those two things by adding FU's LD equals gold as a link option and by telling CMake
that we want to use the debug build type. The next thing we need to do is to pass the options needed
for debug vision. This tells CMake to pass the same compiler and linker flags that we so previously.
Finally, we get to the trigger part. How do we tell CMake how to link together the DWO files
into DWP package? There is currently no built-in logic to do this in CMake but here is a workaround.
The workaround is explaining more detail in the blog post but in short, we're extending the
default executable action in CMake with a custom command that generates the debug package that
accompanies each executable. There is an open issue on the CMake blog tracker requesting better
support for debug vision and someone actually added a comment there referencing my blog post.
In that comment there refer to this as a hack it were around and kind of fragile and I cannot
say that I actually disagree with them. Having a free support for this in CMake would make things
much smoother. Let's move on to Basel. Here the story is fortunately a lot simpler. Basel supports
debug vision since version 6 and it works as long as the underlying tool chain advertises this
per object debug info feature. With this in place you would then pass fish and equals yes either
via the command line or in your Basel RC configuration file. The DWP package is built separately
but by requesting the executable target name but with a dot DWP appended.
Then I've done a small experiment here. I have to admit that this was done on the by building the
LLDM project. I know this is the GCC DRUM sorry for that but this was the one project that I found
online where it was easy to generate a comparison. They have a flag in their build configuration to enable
this and I'm basically doing three builds a release build we know the DWP symbol that's our baseline.
A DWP build regular build with DWP symbols and then efficient build with DWP symbols and DWP
vision enabled. The blog post has the absolute numbers and much more details but I will only do a
rough comparison here. The darker blue shows the cost of the DWP build without the above vision.
The lighter blue is with DWP vision enabled. For overall build times the DWP build takes almost 40
percent longer than the release build and the vision only takes 25 percent longer. If we include the
install time, things look even worse for the DWP build and I didn't really understand this is first
because the install face of LLDM is almost purely copying files from A to B so what is it takes
so much longer with a DWP build. It is actually purely down to the size of the artifacts. A DWP build
takes 15 times as much space as a release build and you'll notice that I had to resort to a
logarithmic scale here to fit it into the slide. Now the DWP symbols do take a lot of space but most
of this enormous overhead is not due to single copy of the DWP symbols rather the DWP symbols are
copied from object files into intermediate archives and from there into refined executables. When we enable
the vision, that size goes down a lot to five times and the numbers are similar for the
install artifacts. Anyway what can we conclude from this comparison? DWP symbols take a lot of
space of course but the real space overhead has listed with DWP symbols and worth to do with
the amount of times things are copied. When we use DWP vision we can eliminate much of this
duplication and simply by reducing the waste space we can also improve the build time significantly.
So with that we reached the larger conclusion of this talk. DWP vision can save you both time
in space at the cost of some added complexity especially if DWP vision is not already supported in
your build system. Is it worth it? It depends as always. If you complete DWP vision to building
an unscript executable and then distributing the strip version, the strip plus unscriptable
offer the smallest release size. No matter what you do the executable with DWP vision will be
somewhat larger. But the strip plus unscript will also be the least efficient way to do your build.
First you will make a full DWP build and then you must strip that in order to generate release.
So if you struggle with this overhead then looking at the above vision can be a really valuable
investment. So that's it for me. Thank you for listening. If you think that this talk while
there should have been a blog post then you're up to the right. Here's the blog post.
You can find everything in this talk plus even more details, side quests and links to more resources
that I couldn't fit into here. More generally on the tweak bug you can also read about all the
all the things that triggers are up to and if you want to learn more about tweeter mode is great
or links for those. If you want to reach me I'm Jay Holland on GitHub master on LinkedIn and elsewhere
and with that you can always find me afterwards if you have questions or want to discuss more.