Alright, hello everyone. Thanks for coming. My name is David Branstill and this is
the talk on why our Android builds so damn slow. So just a little bit about me. I spent
about 10 years working at Google on the Android operating system. I was an engineer on the Android
runtime team working on optimizations in the Java stack and Android security else
the technical lead for the hypervisor and virtualization framework projects and now it's
sort of the dev we work with device manufacturers who use Android in their product and we try
to optimize their workflows so they're more efficient. And the number one complaint we keep
hearing from the ecosystem is Android builds and checkouts are just crazy crazy slow. Just
to give you some context. I mean you just heard from Stefan how huge the Android codebase
is checking it out, downloading the source code to your computer easily takes between 15 and 20
minutes. Most of this is just because it's so damn huge. But the other factor is that the
OSP mirror can be quite slow and a good prototype is to use the authenticated access because
that's going to get around some of the usage quota. Once you have the source code you want to
build it that typically takes around two hours or roughly four and a half dollars depending
on where you get your machines from and again this comes down to the fact that it's so damn
huge. There's about 250,000 build steps that the machine needs to go through and obviously if you
spend more on your hardware it's going to go faster but the costs are also going to go up so
there's a trade-off that you have to balance. And then as you develop you're going to go through
incremental builds and those can take 10 minutes, 30 minutes, it all depends on what you've
changed and how much of the codebase needs to be rebuilt at that point. But there's also another
problem with Android and that's the fact that it doesn't have a good enforcement of the dependency
declarations. So sometimes you end up changing something and it doesn't get correctly rebuilt and
it uses an older version of the build artifact. So at that point most engineers just wipe the
build directory and start from scratch and it's another two hours to get back to a working state.
I'm going to be quoting some numbers here. I'm already M just to make it reproducible. I picked
a virtual machine on Google Cloud for the 2VCPU 120 gigs of RAM and VMSSD costs about $2
an hour and all of the numbers are for the latest Android 16 QPR 2 release and it's building the
AOSP cutoffish ARM 64 phone target. So if you want to try it, sorry.
Now, okay I'll speak louder. So if you want to reproduce these numbers you should be able to buy
by creating a similar environment. So historically the answer to how are we going to fix this
has been that Android will eventually move to Bazel. Bazel's the build system that's used in
the Google Monetary power. It has been open sourced. It has all these great properties of
permeticity, reproducibility. It has build and test caching. It scales really well.
So the original plan from AOSP has been to gradually migrate from the old make files to Bazel.
A bunch of steps. Step one, develop a build system. It's now called soon, which uses the Bazel file format,
but it's backwards compatible with make. Step two, gradually convert all your make files to these
new BP files. You saw that in Stefan's chart as well. And then start gradually building some
parts of the system with Bazel and gradually phase out soon and the original make.
Step four, profit. I get all the benefits from Bazel and the associated infrastructure.
The effort culminated around Android 14 early 2023 when most of the AOSP codebase has been migrated
to soon. Give or take some device config files. But the adoption downstream has remained
a lot lower, especially in BSPs. You still see a lot of make files.
But in early 2023, we started seeing Bazel builds on CI. Android.com. So some parts of the system
started to be built with Bazel. They're all looked great. And then in typical Google fashion,
we got a notification that Bazel isn't a supported build system anymore. The migration has been
halted in late 2023 and outside of building the Android kernel soon is the main build system from now.
So that kind of reason isn't a bind. We're in a state where the migration hasn't been
completed. We have soon, but soon isn't Bazel. It doesn't have those great properties of
permeticity and reproducibility. It's sort of tries. There are some mechanisms to
sanitize the path force. Certain tool change to be used. There's a sandbox for generals. But
under the hood, it's still compatible with make. And you can do anything with a make file. We can
bypass all these great sandboxing mechanisms. So the reason why Bazel builds are fast is that
it builds on these properties and uses them in a really clever way. But because soon doesn't have
doesn't have those properties, you can't really do that. So in Bazel, all inputs and outputs
are declared upfront. You don't have that with soon. It's easy to send box tasks in force,
dependencies, capture results. You can't do that with soon. It's easy to upload
computation to another machine. If you have a build farm, especially in a large organization that
can be really helpful, you can't do that with soon. So the question is, where in this state,
what can we actually do? How much of these sorts of features can we get with the build system that
Android currently uses? And we'll probably use for the foreseeable future. I'm also going to
reference a talk from Chris Simmons at embedded OSS 2020 free where he deep dives into
how soon and all the other components work. I'm not going to go through that here. I'm just
going to tell you the important bits. But if you're interested, go and check out that video. But in short,
the way that the build system works is that we have two kinds of build files. Android BP,
those are the ones that are very similar to Bazel's StarLark language. And these are just templates
and all the build logic actually lives in Go. And then we have the legacy Android MK files
that are still used in a bunch of places. And the way that the build works is that you have
four main components. Soon UI is the top level process. It drives the whole thing. It also renders
the progress bar. And then you have soon underscore build, which takes the blueprint files and
compiles them down to Ninja and Kati, which does the same thing for make files. And that's how the
two systems can coexist. They both compile down to the same low level format and they can reference
each other if necessary. And this process, if you've ever built Android, is really annoying because
depending on how fast your SSD is, this can take anywhere between two and seven minutes. And every
time you touch one of these files, you have to go through the process again. The reason why it's taking
so long is because it loads everything into RAM. There's a lot of analysis. It uses about
40 gigs of RAM and then writes about seven gigabytes of data on disk. So if you're changing
the config files, the build files often, you're going to be running into this over and over. But once
this is done, you get a set of the build. Ninja files and you execute Ninja and then just goes through
roughly 250,000 individual build commands and executes them with as much parallel as it gets.
That's the bulk of the build. And if you're ever trying to debug while your builds are slow,
one indispensable tool is looking at the build traces. This is a file that's generated during the
build. It's in out slash build.trace.gz. And you can take this file and open it in any
perf viewer. The one I like is Profito.dev. You can just upload the file there and it gives you this
beautiful UI where you can see everything that's happening. And we can see those stages here.
So this red one that's building the blueprint files, this is building make files that on this
machine took two and a half minutes and the rest is Ninja compiling all the little bit and pieces
that you actually need for the for the system. And before we dive into caching to some quick
tips for how to make it run faster. So obviously, nobody has unlimited budgets. We have to make
some trade-offs. So what's a good trade-off in terms of hardware? For the CPU, more cores is always
better. You can get so much parallelism out of that main Ninja step that if you can get more cores,
get more cores. 24, 32 is a good baseline, but 6428 is just going to make it go so much faster.
More recent architectures are usually better, like they're always better, but it's not as much
of a win as just raw parallelism. RAM, you want about two gigabytes per logical core. So if you have
32 cores, you want 64 gigs of RAM, but that's typically the minimum. There are some spikes, so you
do want to have a swap roughly the same size, just to be able to absorb those spikes every now and
then otherwise, the next might decide to kill your kill your belt. But the most important one is
storage. Low latency is the most important aspect in a belt. It doesn't matter how fast is the
transfer speed, latency is the king. And the reason is because the working directory has about four
million files and the median size is about 300 bytes. So these are millions and millions of
teeny tiny files and you just need to be able to create them on the file system as quickly as you can.
It's the same with the soon underscore belt face. It keeps scanning the code base
and scans about one million paths, but only reads a handful of those files.
So again, just how quickly your SSD can respond is the most important aspect here.
Right. So back to caching and re-couping some of the features that Basil has, but soon doesn't.
So if you just want a local cache, there's the good old C cache. Everybody's heard of that.
And Android has supported pretty much since the very beginning. At some point there was a pre-built
binary in the code base. That's now gone, but you can still bring your own. If you just install
C cache in Ubuntu and you point Android at that binary, it's going to work just fine.
The way you set it up is you need these three environment variables, the top of the terminal.
UC cache equals one, just enables it. C cache, C cache, dear is the place where you store the data
and C cache, exact is the binary itself of C cache. And then C cache, dash capital M,
sets the maximum size of the cache. 20 gigs is enough for AOSP 16 and then you just run M to build
Android and at the end it's going to tell you how long it took. If you remember in the beginning
I said the baseline was two hours, it's about our 57. In this case C with C cache fully populated,
it finished in 47 minutes. So we spent 20 gigs of local storage and we get 58% improvement in
build times. And if we print the statistics from C cache, it will tell us that it had a cache hit
126,000 targets out of a total of 244,000. So it's pretty much exactly one half of the whole
build is C and C++ that can be cached with C cache. So this is a really low effort thing that you can
do and it's going to make a huge difference. Sorry, no. But the state of the art in AOSP in terms
of build caching is a component called RE client. This is similar to C cache in the sense that it's
a wrapper around a compiler, but in this case it connects to bezels RE infrastructure.
And compared to C cache which is only for C and C++, RE client has support for a lot more
different kinds of compilers and tools. So there's client, there's a linker, javasy, metallava,
RE, D8, sine APK, and a bunch of smaller tools. And our client can take those and make them
cacheable. So for this you need an RBE backend. Backend has two kinds of services. The first one
is that it's a distributed cache. So clients can upload results of their computation. And then
either they themselves later or somebody else can download the same result later on. And in a
larger organization there's also the support for remote executions. So you might have a build farm and
you can submit those individual jobs to the build farm and get the results back. There are a lot of
different backend solutions. This is a standard protocol. The open source ones all have very similar
build barn build farm and build grid. They all are under Apache 2.0. And if you go to the official
bezel website there's there's a list of other ones and commercial ones and so on. So how does
RE client work? It has this command called RE wrapper. You prefix your commands with that and
Android would do it for you. And RE wrapper its job is to create this data structure called action.
It's part of the standard of the protocol. And that describes what is about to be executed.
But here we hit the difference between bezel and zoom. Because in bezel you have all the information
available upfront. It's all statically declared and you can find it immediately. Whereas RE wrapper needs
to do a bit of work to fill in the gap. So for the command and environment that's statically available
if it parses the command line that's being executed it can figure out which compiles being used
with our domain source files. So it can populate some information about the inputs. But for
example to figure out which header files will be included by the compiler laid on it needs to run
the preprocessor. So it's a bit of work to get to that information. And finally those input files
need a hash. So it needs to hash those files before it can port can talk to the back end.
Once it has the action struct it sends it to the back end the back end depending on how it's
one figure it will either execute there or look in the cache. And it gives back another data structure
code action result. That's the same thing but it describes the outputs of that particular action.
And it's just an exchange you know you do this a few hundred thousand times a build and that's how
that's how you that's how the caching works. So to set this up on your own machine
uh first you need a RBE back end I'm not gonna go through that here. It depends on that particular
project and it's quite well documented. But on the client site and Android you need to create
a config file. There's an example in build soon docs RBE that JSON really well hidden and you need to
populate things like the IP address credentials then you select the two chains that you want to use this for
and unfortunately I haven't been able to find one place where they would just have a list.
But if you go to code search cs.android.com and you just search for RBE underscore you'll find all the
strengths. And finally you need to select the exact caching strategy so it's what happens on a cache
and there are three options remote local fallback so execute remotely fallback to local if it fails
local that's the one I'm using in the example which is no remote execution just treat the cache as a
distributed cache nothing else. And finally my favorite racing which means you you tell the remote
server to execute but you also do it locally and whoever gets their first that result gets used.
And once you have this config file then you just need to again set some environment variables
to point Android at that config file. So the way this looks is two variables at the top the first one
is the directory the other one is the name of the file minus the.json for some reason. And with that you can
run m first time you need to populate the cache second time it's going to be fully populated
prints the stats at the end. So from this we can see that there were 156,900 local executions so that's
the number of times that the wrapper was invoked and of those 144,700 were cache hits and that's out of
247,000 targets it for some reason adds 3,000 targets. So this is 64% coverage of the build and the build
completed in 30 minutes 25 seconds which is 73% faster compared to the baseline. And this is a clear
improvement over C cache but to me it's actually quite disappointing because we've gone through all this
work of you know a standard protocol distributed cache support for 10 different two chains and all we
get out of it is another 14% of coverage and another 15% speed up. So it kind of shows you that there's
a limit to how much you can do with compiler wrapping and this general approach of retrofitting the
RBE protocol to AOSP. But this is the state of the art this is the best that you can do with
AOSP today. So if that what's next what can we actually do better here? Well ideally we'll get better
coverage less local pre-processing but that's tricky because this is a heterogeneous code base
every project is a little different it has a million different programming languages. This is
probably not unique to Android the Octo is very similar in this regard. So it's some sort of generic
mechanism for isolating and monitoring those individual build commands so that we can we can
move them around cache them find the find the inputs and outputs. It will also nice to speed up the
Ninja file generation like we said it's somewhere between two and seven minutes you run into it
every now and then ideally the algorithm itself would be faster and anecdotally I feel like in the
past couple of releases it's got a little better but it's also quite you know ram intensive and
I'll have a so there's probably a limit to how much how much you can do there. The other option here is
extending caching to this early stage to cache the result of zoom build and cate so that
if you're an organization somebody else has gone through this or if you're switching between branches
and you have a local cache you don't have to go through this every single time and you will have
notice that we haven't really talked about checkouts and that's that's also still a problem especially
in CI where you don't have that persistent state or if you're an engineer who keeps switching between
Android branches because you need to cherry pick fixes into different releases and so on.
In that setup you keep going through that checkout over and over and it's also quite painful
but again going back to Stefan's talk actually about 50% of the disk usage is coming from pre-built
and pre-built have all sorts of like versions of client and they're built for Linux and Darwin
and really what you need during the build is one version for one architecture so a lot of that
time during the checkout is actually completely wasted and even in the source code you don't always
rebuild everything you don't need all the definitions for all the devices you don't need cts when you're
just building the operating system itself so there's there's a lot that we're downloading for
no good reason there actually used to be a solution for this get vfs is a Microsoft project
for a virtual file system forget which would allow us to do you know on demand downloading of
of these files but unfortunately the project has been deprecated so that's also
also kind of stuck in that sense and that's the end of my talk these are the questions that
we're asking ourselves in search of the death and we were having an announcement if you want to see
demo of how we do checkouts in 30 seconds and built in five minutes then we have a video for you
and yeah if you if you're interested in what we're doing we're hiring so that's an email
and yeah that's the end thank you
um do any questions yeah go ahead
I have heard of a bfs I don't think I'm allowed to talk about that sorry go ahead
are the questions yes problems when you have one version of a base file in the cache and they're
building using our cache a newer version of a base and some break in the question you'll
build so I've had that experience uh put them uh so I don't think I've understood
you mentioned that the song doesn't properly drag all the inputs right everything that
influences our content so you have never had bridges using our new cache right you have passed
based on the wrongs for those but you're building newer source but they're okay so I'll just
repeat the question uh it was this our client ever run into the issue of not tracking the dependencies
properly uh and I have not seen that um it's and that's because it's spending that effort
trying to preprocess the uh the local inputs to really uh to really figure out what the compiler
will need later on are you only using a pure google baseline when you're doing the testing
yes uh in this case pure aosp um yeah I mean though the support for if you look at our
client it was built originally for chromium and then later extended for aosp but it's literally aosp
like if if you have a vendor code base which uses a compiler flag that isn't used in aosp
then it will not know how how it works and if it affects the build so it would just say sorry
I can't I can't catch this time's up um all right thank you so much