This next session is going to look at project maturity through a very specific lens,
release votes in the Apache incubator, from the outside it can seem pretty procedural,
but over time it reveals a lot about readiness, quality and shared responsibility.
Please welcome Justin McLean.
Welcome everyone, I am Justin McLean and I am from Sydney, Australia.
This is my first Boston, so it's good to see everyone here and it's great to see so many smart
and passionate people in the open source space.
So this talk, why I'm talking about the Apache incubator, this is not a talk about the ASF.
You're going to have some practical stuff that you can take away based on this,
and this talk is about shipping code and about how the process around shipping code
can either amplify stress or reduce it.
So again, this talk is not about ASF policy, it's about lived experience
and it's also based on actual data.
Now, this is where we're going to find out what this works for not.
So I recently looked at a decade of releases and this was not just for one project,
it was from a total of 33,000 emails, 1,600 vote release threads and 160 projects.
So that's a lot of data.
What was interesting about that is that there's a lot, the 160 projects were quite diverse.
There were some very small projects, there were some large projects,
there were written in different languages, they had different communities behind them.
And some of those projects, I didn't exclude any projects either, so some of those projects didn't make it.
Some of them failed, some of them retired because there was no community interest in maintaining them anymore.
And that's sort of what matters because we had this wide set of data, which to draw things from.
It's not working.
So the other interesting thing that came out from the data is that release processes are not neutral.
They are released will actually quietly answer some important questions for your contributors.
And expectations learned through experience of people's involvement with a release rather than your documentation.
And the problem that the incubator had a decade ago was that we had a whole set of rules around releases.
And the way that those rules were conveyed to contributors and people working on these projects was that it was learning more by punishment than it was by iteration.
And these releases will say up here, this releases in code power and what I mean by that is that when you come to a release and you're voting on it,
it means that you find out who can say no, who can hold up the release, who can block progress on it, who can who needs to justify themselves in order to carry this release forward.
And also what, you know, effort that people need to go back, go for, and also how the feedback on the releases delivered and all of this matters.
And if the process feels like a peak or, as I said, punishment, people are going to maybe not make a release or they're going to delay making a release, do they think it's good enough?
But if it feels, you know, clear and survivable, then people are going to try and actually do this. So your release process actually shapes behavior well beyond just legal policy or, you know, other policy compliance.
And that was one thing that was very, very clear from from this data.
And as I said, about a decade ago, the releases through the incubator were technically correct, they followed policy, but they were socially damaging.
And the reason why that happened, and this wasn't malicious, this just came about by accident.
It was, um, first off, there was a lot of checkbox policing. So, you know, a release must have these things, there's a whole one checkboxes, people would go through and check everything off, and if it didn't have something, they said, okay, that's broken, we got to do it again.
There was also a lot of rules that were not documented. These rules lived in people's heads that was tribal knowledge.
And so it was very easy to make mistakes. People could come along and, and thought they've done the right thing, but didn't really realize this.
And the only documentation that existed was the mailing list archives, which is really hard to go through and read all the information from those and try and work out what, in some cases, these unwritten rules were.
So, the review feedback often became the place where the rules were learned, and it wasn't something you could actually prepare for, which makes things difficult.
And as I said before, this wasn't malicious, it's just how the way things evolved, and the decade before this one that I looked at was probably even more chaotic, and because that was when the rules were actually being formed.
So, there was lots of arguments about what, you know, should it, shouldn't be, and how religious it should be made and all the rest.
So, if we go back to this point in time, so the project's first release would 90% of the time would probably fail.
And there would usually get a large list of issues, and they would have to be fixed, and then it would take several weeks for those to be fixed, before another release candidate was tried again.
And these days, about 75% of first releases actually pass.
It actually has been a dramatic improvement over that decade.
And if there are some issues, they generally minor, and they get fixed fairly quickly.
Now, one is really important to understand here is that the rules themselves didn't change.
The rules are still the same as what we had 10 years ago.
It was the experience of the people involved that has changed.
And what happened was, it was the same rules just a completely different learning experience.
And what the main change was in review behavior, and the shift didn't happen automatically, it happened over time.
I think there's some reasons for why this happened, but the data doesn't tell me that.
And mentors on projects shifted from saying, that's wrong.
You need to fix that, to say, well, that's not quite right.
And here's why it's not right, because it doesn't follow this principle that we have, and this is how you can fix it.
And so saying why something matters, how to fix it, and to give some reassurance to the project,
that things are actually on a good path, and that you're next release is going to be there,
is a much better way of approaching this than just saying, you haven't done this, start again.
And again, it's the exact same rules, and just a completely different learning experience.
And it makes that means that people will no longer afraid to be involved in making releases, and more people will willing to try.
So over that time, once people's fear, or concerns are removed, they are much more likely to make the releases they're behavior changed.
Projects attempted releases earlier, they iterated faster, and they recovered more quickly from any mistakes.
There's been some other changes over that decade that some of them have been complementary, but this is mean the main changes that has an impact.
Other minor changes, like tooling, for example, to automatically find most of these issues in a release, have certainly helped,
but that's not the whole story. It basically happened because shipping code and creating a release became psychologically safer.
The other interesting thing that came out from this is that releases are not just not artifacts,
but releases are actually onboarding material for your community.
And this applies to every release, not just the first one.
Each release teaches what done means, how strict you have to be, to follow your rules.
Guide loans, whatever you have, and with a mistakes are fatal, will easily recover from.
And the first release is probably doubly important because that's six expectations for future releases.
And it's still the case though, because you get new contributors that are being involved in your project, that every release is going to be an onboarding experience for the people who are involved with that release,
and the people who are watching as well, not just the people who are actually creating the release.
Now at this point, I should probably point out that not all open source software does releases the same way that ASF does, but the same still applies.
If your project doesn't do a formal release, they're going to have GitHub tags, they're going to have packages that they make in some way.
They're going to have container images, or even a stable API that's called a release.
So every single one of those things can be seen as a way of onboarding new contributors to your project.
The other thing that came out from the state of, as I said, 160 projects, was there's a huge impact with the first release.
The first release gives a project momentum, and that means more people get involved.
And looking at the entire day there, it's the clearest indicator of a project's trajectory, whether or not they're going to succeed or fail.
And if a project makes a release early, in the day that I was looking at, we're talking six months within six months, then they attract contributors, form governments, make new releases, quicker and quicker, and generally they survive and become a top level project.
Projects that don't make a release quickly and leave it, you know, say for a year, or so forth, tend to stay small, they tend to stagnate, and sometimes they don't succeed.
And this was a really clear outcome.
And over the decade, usually took the medium time for a release was about seven months, initially, and it's now got down to four or five months.
So we've seen a great improvement in that as well.
So one thing I should make clear here though, is that earlier releases don't imply lower standards.
In these cases here, the quality bar didn't drop, it just became more visible.
And early on, when people come up, early on the project attempted releases, that there were mostly common technical failures, like, you know, issues with licenses and notice files and trademark issues and so forth.
Now, with those sort of manual privileges have declined, and have declined for a number reasons, one is that there's some automated tooling in place.
The other is that there is better documentation that exists.
Another factor, and I haven't actually done any research into this, we have a course set of mentors that the mental, the mental, multiple projects, and they've gained experience over time.
So I think that has definitely helped as well, but more importantly, there's a shared understanding amongst all the people involved, so they know what to do.
And that has come up, you know, it's taken 20 years to get this point.
It didn't happen overnight, that's for sure.
And most of the issues that we run in today, and not issues that tooling or CI or AI can find, it's the sort of issues that require human judgment.
And that's basically things like licensing intent or providence of code or things along those lines.
So the automation and tooling is removed some mechanical noise, but it's just gotten rid of the low hanging fruit, and we still need humans in loop to actually create these and their judgment and mentoring certainly helps that.
So the other thing is that people tend to follow momentum, so I was talking about this first release effect, and that attracts people to your project, that attracts new contributors.
More road maps that attract contributors, it's actual progress that does that.
And one other impact that we could see in the data is that projects that released early also tended to add new contributors to their project much earlier.
So again, a decade ago, it generally took a year for some projects to add new contributors to their project.
Now it's under six months, and that has a huge impact.
So not only, if others people outside your projects, seeing that contributors have been added to the project, they're going to stay around for longer because they think, oh, maybe that could happen to me as well.
It also means that the earlier that people are being added to a project, it removes some risks.
And those risks include things like single Martina projects, so you have more people involved. It also helps with vendor neutrality.
Because you've got more people involved, they're probably employed by multiple vendors, and it means that no one is setting the direction of the project, and it's a shared effort.
The other thing is that it's really easy to have a look at a project and look at its commits, and see what's going on there, that commits can lie.
Particularly now, it's a whole lot of automated tooling, the pen button and all sorts of other things that will mess that statistic up.
So it's actually, you can very easy look at release cadence and treat that as a measure of community health.
If a project is releasing on a regular cycle, that's great. If you start to see gaps, that could be early warning signs of coordination problems will burn out or governance issues, fragile ownership.
So if you actually want to look at an open source project and see whether it's healthy, just look to see whether it has made releases and how often it makes releases and how often it adds new contributors.
And that's going to give you a really good idea about how healthy your project is.
So in those decades as well, that decade I should say, the governance didn't get lighter, it was still there.
It was just a little quieter, because those early events, crisis, maybe in some cases didn't happen.
There were fewer escalations. So there were people who had a more a better shared knowledge of how to make a release and it was better documented.
So there were fewer confused projects and they actually made more releases and more releases got shipped.
So the fact that there wasn't, there's now less escalation than there was before, doesn't mean there's less oversight.
But it just means that the oversight is happening earlier and more gently and the projects have a better understanding of what needs to be done.
So what to take back to your project?
So as I was saying, you know, I've been looking at this from an ASF lens, but this is applicable to open source in general.
So you should treat making a release, we're shipping software, however you do it as onboarding, both through existing contributors and for new potential new contributors as well.
And you need to make those expectations around that release, explicit document what your release process is.
So you know, these are the things that must happen, these are things that are nice to have.
You know, if we make a mistake here, we need to, we must fix it in this release or we can wait until the next release as long as there's incremental improvements that that go on.
So you don't want to rely on people learning the rules during a release review.
You want to take checklists and examples and templates and anything that helps reduce that anxiety and, you know, shorten feedback loops.
You want to also, when something does go wrong, explain why it is an issue, don't just say it's wrong.
Suggest how to fix it, point to the underlying value or reason to why why this is something that is important.
Also say whether it actually is a blocker for this release and whether or not it can be fixed in the next one or at a later time.
The other thing you can do is track time to ship and time to first new person in new projects.
That's going to give you a really good indicator of their health.
And I think these four things here, none of these are specific to the ASF, they will apply to all open source projects.
But the real reason that we've got here is that releases are actually community design.
They're not just legal compliance or how to follow policy.
They have a much larger impact on your community that you may think.
And the question isn't whether you enforce standards, it's whether you let people feel safe enough to be able to meet them.
And the right process you have around your releases is going to lower confusion and fear that people have.
And when that goes away, people are going to stay, they're going to learn and they're going to contribute more.
So that's it for me, do I have any questions?
Anyone questions?
Thank you.
Have a good rest of your day.
And if you have any questions for me, come find me around the conference if you can and I'll be happy to answer them.
All right, folks, we have about five minutes.