Perfect. Thanks, everyone, for coming. My name is Sherbanko, I'm from UNESCO. This is an unofficial
guide to trunk stable. For those of you that have not heard of trunk stable before, this
is a new Android development model that you're going to learn a bit more about in the next
few slides. So, just before we start, a little bit about myself. I've worked on Android
for close 13 years now. Almost half of that time, I've worked on Android from a
term. I've contributed significantly to porting Android to 264rv8, so 64 bit. I've
also pmd technology called R memory safety. This is basically a hardware technology that
helps with catching buffer overflows and use after freeze. And then with Google, I was the
pmd for a bunch of technologies that you've heard about in some of the previous slides. I
led Android virtualization. I led Android runtime and Android native teams. I've also led Android
regulation and in the past few years I've been working a lot on regulation and longevity.
This is basically Android's initiative to keep devices around for longer.
Last year I left to start source of dev together with David who's here in the audience and
also a game who's a vector where we're basically building an operating system for the operating
system kind of. We're trying to make it easier for some of you folks to be able to develop
but also to maintain those code bases for a long time. We're putting all the learnings
with a gather across the years to build this and we're hiding. I'll tell you more about that
towards the end. So just the last thing, please take this as an unofficial guide. I've been
deeply involved with a lot of the decisions we'll discuss here but I'm no longer a Googler. So
please take this as the unofficial guide and if we discuss it before many things are changing
quite rapidly so I might want to check the code base even if I claim something's here in
in this deck. So just a quick overview of what's trunk stable. So trunk stable is basically
a new upstream Android development process where all the releases are cut from a single main
development branch and that branch must remain stable at all times. So it must build and pass
a set of tests. This basically requires that all new features are developed behind flags
so that on a case by case basis those flags could be enabled or disabled to maintain code base
stability. You'll see in a second why this model was very much needed but basically Google
advertises this as streamlining development and improving quality. So probably the best way to
describe why the new model works and why it's so useful is to take a look at the old development model
and to try to understand what wasn't working in the old development model. So if you look at it
from a timeline perspective the old development model kind of looked like this. You had an Android
version being released then soon after that Android version would be released or actually internally
even before the Android version would get publicly released we would start working on the next
Android version then you would have portion of code freeze and this is also known internally
at Google as Dr. Now you would need to get approvals many approvals to get patches margin to the
code base and then the next version would basically get released. So behind the scenes
this model kind of looked like this. Folks like me, pms would go off and write pardes and
figure out what we're building and why we're building and what's the minimum viable version of that
and then together with the engineering teams we would start working on those features
and then almost a week or two weeks before code freeze held would break loose literally
like everything would be merged into the main course. We would all be dog footing our devices
and those devices would stop working and that period over their kind of correlates with the time
my wife would get very mad at me because my phone would almost never work. But the important
thing here to highlight is that because the timeline was somewhat fixed that meant that we had
to compromise on a few things and some of those features might not have been
ready to be released or would have required a longer period of time. So from a quality perspective
if we look at the same thing it kind of looked like this. We would release an Android version
and in the background we would still be eagerly working on stabilizing that code base
we would heat some sort of stabilization point after releasing the code base and then again
we would do the same thing where the quality would drop when we start merging all the
features. Just to put this in perspective I think the Android engineering organization today is
about 4,000 engineers at Google we would work very closely with some of the ecosystem partners
so think about similar numbers from the lead associate vendors and then think about the
charting that would go on in the code base I think Android AOSP it's about 120 million lines of code
today. Android OSP as you folks know is not actually the full Android as you can imagine that
the full Android is even bigger than that and I'm sure that some of your internal code bases are
very big as well so imagine the amount of children that would go on there and how difficult
they would be to get this on. So just sort of like recap if you think about this model
at this model choose is a fixed timeline kind of fixed set of features and then kind of
doesn't prioritize quality and this model has worked actually for many years early versions of
Android didn't have these problems it probably became really significant in Android 12 if you
search Android 12 online there many articles including the words should that say well this was
in the greatest release and was primarily because of that reason. So by comparison this is the new
Android development model from a timeline perspective. So instead of having one release you have
four releases and throughout the year every quarter you have a new release this is unlike the
oak all QPR model you're used to the all QPR model would primarily be bug fixes this is actually
features all the time features get released obviously bug bug fixes and security fixes and then
you have these two distinct releases so you have Q2 which is a major SDK release that comes a lot
earlier then all the Android would come would get released so you get probably Android around Q3
now you get it around Q2 because that basically expedites all the development towards products
in Q4 which is the peak of product sales so that gives folks one more quarter to work on that
Q2 has API changes so new features developers could use Q2 also has behavior changes so
changes in the API surface that developers would use and also not mentioned here but Q1 or Q2
have sort of like the hard changes so if you're a SOC vendor you probably just have to focus on one of
those that will be your major release and obviously you need to iterate with the others but those
will be one the hard changes happen and then Q4 is effectively think of it as developer preview
it's APIs and features no behavior changes but developers would have sort of like a minor version
that they could target in their application so that they provide earlier feedback towards the next year
towards Q2 so for this model you're obviously going to have features that take a lot longer
than a quarter to build most of the Android features take a lot longer than a quarter to build
and you would have features behind flags only the features that are ready to be released would be
turned on by the flagging system and everything else would remain turned off and you basically
have every release would probably have a slightly different set of features enabled a slightly different
set of flags where probably every feature would inherit from the previous feature and other
few more on top of it so from a quality perspective assuming you've got like the right
system is in place this kind of looks like this and so you'd be able to quickly revert features
that don't work turn off flags and things like that actually if you follow closely the Android code
base some of the systems are already present in the OSP codebase you would see flags turn off automatically
by an agent not an AI agent but like an agent that actually does something that's useful
and that agent would basically monitor that the codebase is stable or if it's not stable
it would try to bisect and identify the flags that have been turned on and then turn them automatically off
so from if you look at this model this model obviously chooses features in quality prioritized
as a quality which is very important and then kind of has a more granular release mechanism
which gives developers more opportunities to release the features that make the Android codebase
great so if you put them side by side the all development model and the new development model
kind of looks like this the all development model was branch based even though
David corrected me on this actually internally you don't have that many branches but you have a lot of
code that's flying around so the all development model you could still say that's branch based
teams work separately and then towards that called for this everything would be merged back in
you have a yearly release cycle those QPRs actually are not that meaningful or didn't
used to be that meaningful in the past and then it's relatively hard to maintain codebase
quality and you get only one SDK release per year by comparison the new development model is flag based
there's a quarter release cadence and field it's easier to maintain codebase quality and as I said
you could already see the benefits of this in RSP and then you get a few more as the key releases
per year which is also important because what the old model was missing was feedback from developers
on API usage in the applications and so on and this model enables that so if you look at it from a
downstream OS fork and you can think here of a downstream OS fork sort of like an OEM OS fork
but you could also think about automotive for example or other verticals those are OS forks as well
the old model had a bunch of problems around sort of like rebasing and testing cycles
some of the bigger OEM so I had a pleasure to work with across the years for them this yearly
rebase would be a significant productivity hit like basically the whole team would would have to
rebase a massive codebase it would be difficult to identify issues I really stopped stream
actually we've heard this a lot one going the details but you can imagine it's finding the needle in the
haystack and by comparison the new development model in theory makes it easier to rebase and test continuously
easier to identify upstream issues easier to bisect and that would hopefully reduce the time
I dropped it a codebase I think the last thing I'll say on this slide is that
this is primarily for android properties or android phones but you will see some of these
trickling down to the other verticals so if you're doing automotive where or anything else
android related sooner or later this would be coming your way so you might be asking yourself
weren't went to use facts I'll say by saying most big modern codebase is used flag based systems
there's almost nothing out there of the size of android and actually even order magnitude smaller
that doesn't use flag based system google play gms core facebook tick talk all of those codebase
flag based releases so you should use a flag wherever your change might lead to codebase
instability and that most likely is for everything so all major features need to be behind
this complex bags API changes behavior changes need to be behind flags the place you would not use
flags would probably be comments small changes or where the flag already exists and I think
this kind of requires a lot of change in the engineering mindset it's probably better to
start from a place where you add flags everywhere rather than not flag flags everywhere
so there are two types of flags that you could use in android there's runtime control flags
something that's called aconfig this are basically used in cc++ rust and java
you basically declare the flag in a dot aconfig file and then you could check that the
runtime if the flag is set or not and then there are two types of flags read only flags
these flags are basically the flags they would have for the release
once you set the values as read only compilers would actually understand that
some portion of code is is that code and would eliminate that and optimize the code for you
but you also have read right flags and this is what we use during development
and actually this is how you do a lot of at scale AB testing so you'd have one dog food
population with the flag set one one dog food population with the flags that do often you would see
what are the benefits of that change in feature and then at scale with the with a right
of system behind it this helps you out you also have built-in control flags and these are used
for low level firmware changes the Android virtualization framework uses this for example
because it has changes in pre-built code and other changes that require build system changes
I want to leave more about flags but you can take a look here how to use a flags that's a
pretty comprehensive guide for that and then for build flags I didn't find the right documentation
but if you look inside a build folder and just to get a lot minus p and take a look for the
change that adds a build flag it's pretty self-explanatory so there are a few release flavors
and if you look here I basically start a cuttlefish target and then I vote this command
list releases which would basically give me the releases that are available for
that target a USB current that's basically mapped to the latest release which in this case is
AP4A AP3A and AP2A are releases that came before it then you have trunk staging which is basically
a development configuration you could take a look at trunk staging as like an indication of what
might come up in the next release and then something that's not public is this next release that
Google has internally which describes sort of like the dog food population and also what I
wanted to show you is if you go in in the build output I could take a look at what are the flags
enabled for your particular target and what's really neat about that is that you also have a
description for all of those flags so for folks that were downstream of Android this is actually
quite helpful some of those descriptions are not set up until the flag actually gets publicly released
but you could you could have an idea of like what the next Android might look like
similarly if you look at the AIC on feed flags if you go and pick a new Android phone connected to
ADB and then run this command that command would tell you what flags you have on that device
if you actually run this command you'll see that there are many many pages of flags
there's flags used for everything right now this is just a short shortening of that
and as you can see those flags can have different default values they could be read only they could
be read right and then you also have a name space the name space basically says where you could read
that flag from and so on I'm not going into other details here but as I said some of these flags
are serve could be server controlled so you can think of a system where you could actually control
some of these flags server side so that you do those AB populations and stuff like that and
that's actually what the big companies that do Android scale do here so from a life cycle perspective
it kind of looks like this you have a feature flag you start developing before behind a feature flag
you enable it locally for testing the new enable it dock for the team wise and then you enable it
in the next release and then as I said automated systems could actually go ahead and
disable that and that could also be downstream automated systems that monitor stability for
whatever use case you might have so you might be thinking about this one well that's cool but
when when I get that well there's actually a lot of regulation coming in that says you need to get
security updates and always upgrades there's regulation coming in for all the verticals
in 20th of June this year in Europe all smartphone and tablets placed on the market
would have to support devices with five years of OS and security updates after the point of
placement point of placement for some devices could actually mean seven years of updates
and there's some companies that have already made some announcements in that direction
but there's something that explicitly says that features or OS updates need to be released within
six months of public release so I'm not a lawyer but I'll let you interpret that
and you have a FAQ here that talks more about it and gives you a link to the law
so that's it thanks I'll also have a quick shameless flag will be posting this in air
speed devs but we're hiding if you want to work on something that's very exciting
sort of like an operating system for an operating system we use rust we use all sorts of
kind of technologies we use a brand-new lake and a bunch of other cool things come chat to us
or send the email sharebundant source to dev I will also be working with
former Android team members so that should be super exciting thank you