Hello everyone. I'm Elizabeth I am the lead Pakistan developer for Rhino Linux. This is my
first time at Boston and I'm really excited to be here and to give this talk. Thank you everyone
for coming here and everyone on the live stream and we're all really excited and honored to be
the first talk of the day in the community room.
Hey everyone, I'm Adam. I'm Vfanda and desktop lead for Rhino Linux. I'm so excited to be back here
at Boston giving another talk. We aren't here today to talk about the technical side of Rhino
Linux but rather the personal side. Some of the challenges we face as a team of young developers
and hopefully we'll have some lessons to take away from this but before that there's still one more
person who needs to introduce themselves. And I'm Warren and I'm the systems lead for Rhino Linux.
This is now my third year in a row giving a talk at Boston and like the others I'm quite excited to be
back on stage. As Adam said this is going to talk about the technical side instead we want to
retrace the human side of Rhino Linux and how we learn to build a team of young developers
as well as what this project has continually taught us along the way about maturity, communication,
and sustainability. We recognize that our talk may not apply universally to broader burnout
and developer culture rather as the demonstration of our specific case and how we have worked
to overcome it. However we do still hope that the lessons we have learned along the way can
impart translates to that broader scope. Before we fully begin I'd like us to give a brief overview
about who we are as developers and how we ended up in this space together.
Yeah so my background stems from full stack web development. I was 12 when I decided my first
website and that began my journey tinkering in and around the web space. Eventually I made the
pivot from solely for an end development to full stack and continue to learn and try to master
my skills in that space and then definitely that seems to be impossible with a new framework
releasing monthly. It was around 2020 but I developed serious aspirations to move into full stack
development as a career and began working towards that goal ever since.
Coincidentally 2020 was all to be ever I discovered Linux which kick started another obsession of mine.
I began to play around as much as I could with Linux in my spare time and my exploits eventually
led to me creating my very first distribution. The eventual predecessor to what became minor Linux
and for more on that history you should read the hour-fost them talk from last year QR code on screen.
So I got into Linux in 2013 when I was 8 when my dad got me a Raspberry Pi. I wanted to play
Minecraft like everyone else in my grade and I didn't really know what a computer was and it
technically is a computer so I used it. Eventually I started writing scripts in Sonic Pi which is how I
got into programming in the first place. And then around 2018 I got a full desktop computer that I
continued customizing and rising as much as I could. And then in 2020 I was bored one day and how
most projects start with being bored. I just decided to write something so I decided I was
going to write a package manager in bash or at the time I called it a pseudo-package manager
which eventually turned into Pakistan. And then I came to Rhino Linux through working with Adam
on his predecessor distribution rolling Rhino remix and when it came time to deprecate that in
time for Rhino Linux, Pakistan formed the core package manager of it as a source of distribution
upgrades with our Rhino meta packages. And finally I was born with a keyboard in my hands.
My father's professor at MIT and had me as a test subject from very early on. He created the
education oriented programming language star logo TNG alongside Mitchell Resnick which influenced
Mitchell to invent the far more familiar block based language scratch. And so as such I was one of
the very first kid users of scratch and it massively influenced how I view an interact with code.
Much of my coding regardless of the language feels very much block based under the hood.
I built my first PC with my father at around five or six years old and gave my first
talk at a conference with him at 10. And because of the environment that I grew up in, I've been using
Linux from a majority of my life and have always seen it as just another operating system.
And that first custom build ran Linux Mint and like Elizabeth, I've worked on many
iterations of the Raspberry Pi. As such and as I have mentioned in my previous talks at
Phastem, I have a very long term affinity for Ubuntu. And I've always been a tinker, finding
great joy and putting software where it shouldn't necessarily be. And this drive is what initially
brought me to Rhino Linux, where at the time I had been working on porting Ubuntu touched the
pine phone pro. So from the moment we announced Rhino Linux to our initial release 10 months later,
we worked at an incredibly fast pace developing most of our applications, workflows and roles
very early on as a scope of Rhino Linux expanded rapidly. And we mistook our speed for progress
because we were getting the job done. So we didn't really bother to check up on each other or
document our work. And as such communication between each other was minimal, since we didn't end up
crossing the boundaries of our core roles that much. So we could only assume that everyone else was
doing the work and just continuing on a fit. And the car was technically on the road, but
it would end going in the right direction. So we didn't really bother to check if anyone actually
had their hands on the wheel. We were all students and teenagers at the time with no real
responsibilities. And with a lot of free time on our hands, we kept saying yes to more bold ideas.
We were cruising on borrowed time until the moment came when we actually had to manage our work
life balance with Rhino. And retrospect, this was one of our biggest pitfalls as we designed our
process season pacing around the environments that we were living in at the time as students.
We didn't know how batter system was and how on maintainable the pace we adopted would actually
become. We had no reference for what a functional development team looked like and no real plans
besides the present. We just had to ship the damn thing and deal with the followed another time.
This system worked for previous smaller projects that we had worked on like-pack stolen
rolling Rhino remix. But those were no-in there for scope of Rhino Linux. I was working
style should have changed when we began maintaining an entire series Linux distribution. But it didn't
early signs of problems began to arise in that communication between each other and there should have
been an immediate rev flag. But we kept ignoring it until about eventual moment when it would
become a much larger problem and only then will we be forced to deal with it.
As the project began to slow down paths for honeymoon phase, concerns began to arise about
our pace and the future of our development. Team Dynamics rapidly shifted as a couple of our initial
large contributors primarily around our applications development dropped off or left the project
entirely. They're missing presence in that space is still felt to this day and we had to pick up
the work ourselves. This has only put more pressure on the team and we struggled to figure out how
to sustainably expand our ranks again. The structure we built created a lot of roles I need to be
fulfilled and we could not take on all of them but we had to or else we would have to drop certain
things that people had come to expect from our district. It was around this time that our team
disciplines surrounding release planning steps and our use of canbambles faded away. We got into
a burnout feedback loop. Overwork, withdrawal, resentment, repeat. This feedback loop then became
the basis for how we operated and worked together for a long while and admittedly it's been
quite a struggle to break out of even today as we stand on this stage talking about it we still
find ourselves stuck in this cycle. So the burnout feedback loop it first starts with overwork.
As a team we've consistently taken on more work than we can handle at any given point and this is
always been a source of struggle for us and leads people to first overwork themselves to meet
the internal pressure that we put on ourselves and subsequently withdraw as they burn themselves out.
Typically once this occurs another team member has come out of their withdrawal period whether
by volition or necessity and is ready to take on that burnout instead. Being an active maintainer
while everyone else is contributing less can be a frustrating and isolating experience. Especially
if you or other members of the team have never experienced burnout before up until this point.
Resentment can begin to develop and over time it will continue to grow until eventually someone
snaps and that's exactly what happened. We had our first of many team arguments and it was neither
civil nor pretty. Calls remarks and threats of resignation got thrown around very recklessly and in
the aftermath a lot of us ended up withdrawing entirely for a period of time. It's easy to become
very heated in the moment and taking a pause before applying probably would have made all of the
difference. In the end the arguing didn't solve anything it only deepened the riff between us as a
team heard our feelings and kept boiling that resentment. Moving on from our first few major
arguments we decided to attempt a few changes to our dynamics to boost our team back to the
productive and capable force that it once was as well as making sure that we began to foster a
healthier development environment. First we put up a blog post talking about some of the difficulties
that we faced and explained to our users that we would be taking a step back to focus on ourselves
for the time being. We wrote a code of conduct compartmentalized our roles and tried to take on a
just try to work philosophy. Title of this slide is resolution attempts because despite our
tries we ultimately felt flat on our faces once again. We had hopes that bringing to light our
issues would inspire others to join our team but this was not the outcome. We were desperately
crying out to the world to help us while failing to listen to ourselves and what we had written.
Admittedly though the tiny break we took did end up benefiting a short term with some of those
benefits being almost immediately recognizable. We were talking more compassionately with each other
and team morale was higher than it previously was. Everything else though, unfortunately,
fell short. We never made any faithful attempt to follow our newly published code of conduct and we
compartmentalized our roles and built up our silos of independent knowledge yet again. Truly believing
that if we specialize in our respective areas further, then there would be less chance of friction
as we were communicating less. What was an attempt to go back to how things were was crucially
missing two parts. One we weren't the same people that started this distro and two our roles were
not the same as they originally were either. Officially we had our own distinct roles but we never
really stuck to them so when we finally decided to try and only work in those official roles
we found ourselves stuck in a place that we were before. Our roles were incredibly lopsided
and work and authority and we were simply entrenching ourselves further. Finally, I would just try to
work mindset to addressing but not have some logic behind it, also very we fall. With the hope
that once you get into the group of things you'll begin to find for work much easier and more manageable.
We thought that it would be a wake up call and that just tried to work would be reason enough to
get motivated and get stuff done. This definitely did not work and in fact probably caused us to burn
ourselves out faster in the end. We had sacrificed long-term gains for short-term wins when we
desperately need both. We could not find this over-bully that we were hoping for.
So after a couple months of working like this we decided something needed to change. So it actually
ended up working for us. We can boil it down to a philosophy of three words, allocate,
collaborate and communicate. It became important for us to actually allocate pieces of work rather than
just saying this needs doing and expecting someone on the team to do it. By doing this we could then
set ourselves up to be in a better place to come together and then collaborate on that task together
while each doing our own part. We have found that when we have instituted regular team meetings
we see huge benefit not just in our productiveness but also in our communication. In addition
Adam and I found last year that seeing each other in person at Phastem massively improved our
relationship and so we began trying to emulate a similar atmosphere when we can't see each other
directly face to face. It's very easy to lose your temper and insult someone who only ever appears
as a little icon on a screen and it makes it just a bit more difficult when you know that person
on a much more personal level and meet regularly with them. We focused more on becoming actual
friends and recognizing when we were being rude to one another and in consequence we began to
care much more about each other's thoughts and feelings. One additional thing we have noticed
has helped us is communicating our issues to those who have been in similar positions to us
previously. For Aaron he had his dad to turn to for advice relating to maintain about now
and team dynamics. For me I had a colleague up my company who gave me some great practical advice.
Reaching out to those with previous experience has been a vital piece to personal growth.
For my name including the some stage it can be a struggle to work with in the team if there is
no personal element to it and I feel like our previous mountains of dysfunctional communication
and consistent arguing contributed heavily to my previous burnout. This will culminate it into
us increasing our collaboration as a team instead. Previously we expected team members to work
individually rather than as a collective and this mindset has consistently hindered us or let
the stored releases of multiple of our intended deliverables as work has often dependent on a
soul maintainer. We realized that coming together to work on tasks rather than having individuals
work on separate issues led to faster outcomes and the overall workload becoming more bearable.
Figuring out what really worked for us took a lot of introspection on Aaron. It became apparent
that we had to take a step back and analyze the areas that we were following short. We had to
acknowledge that we do not scale well and had to develop better systems for this. Over time we
eventually realized that we are not good at saying no and consistently take on more than we can handle
at any given point. We repeatedly make promises to both ourselves and our community that we do
not grasp the full scope of, often providing unrealistic and unreasonable timelines for ourselves.
We have a lot of grand and bold ideas that we want to implement but we simply can't deliver on
all of them even if we want to say as to everything.
Aaron is often a perfectionist and has had to realise when he is getting too controlling
as well as learn how to relinquish some of that control. Guilty as charge.
Multiple times this level of overbearing has pushed others on the team to consider leaving
projects entirely. Although part of this problem does stem from the overreliance on
Aaron as a primary developer in many areas, this is what many will refer to as verbose factor
where in a hypothetical situation this person would get hit by a bus. The rest of a team would
not be able to continue as before. The overreliance on Aaron in many areas has contributed to
his personal burnout feedback loop. In other areas we had to recognise that the rules we originally
carved out for ourselves weren't necessarily the right fit for us. For instance Adam had been
considered the primary lead developer of rhinolynics with a large portion of that decision
making pointed upwards towards him. But this wasn't sustainable for a couple reasons.
Firstly he actually gets the majority of his enjoyment from website and desktop development.
And secondly it overalied on just one person to make all of the decisions. Even when that person
was no longer in charge of most of the development which were areas that had mostly been
relegated to Aaron. Despite our arguments and frustrations at the end of the day we have
grown to care about each other a lot. Over the years we have gotten to know each other on a deeper
more personal level and have developed close friendships with one another. We have come to provide
a friendly ear whenever one of us wants to open up about issues that are affecting us personally.
We have found common interest between each other outside of development and within development
we began to provide advice, test and contribute to each other's side projects. Over time we have
come to respect one another as members of the team and recognise the individual contributions
each team member provides to the making of this district. With that we have managed to form
lasting friendships that have been able to hold steady on the rocky road of Linux maintainer
ship. Larger groups have also reached out to us for partnerships. And while this has put an
increased amount of pressure on us as a small team is kept as pushing forward and that's
us know that we are still in large part moving in the right direction. Achieving recognition
from projects that we have a huge deal of respect and admiration for has continued to motivate us
even through our periods of extended bone out. And furthermore we love our distro and what it stands
for is the only rolling release boom to base distribution which we recognise provides great value
to developers who may use it for testing ground for upcoming boom to releases. So this isn't
a distribution we're making for any other reason than our genuine love and joy for it.
Going back to a state of before we believe that the three pillars of alicate collaborate and
communicate are essential. So looking back here some of the things that we've learned along the way.
Constant burnout and dipping out is not a sign of responsibility rather it is typically one
of dysfunction highlighting underlying issues but it is not failure and should never be taken as such.
Reflecting on what is resulting in burnout particularly with each other is a crucial step.
Fearing out for problems that are affecting your team and its dynamics can take time.
It's taken us while over two years to figure out what the problems are and to begin making
some progress towards solving them. Despite this we still find it hard to avoid about how
it's from time to time. It's easy to lose sight of yourself and your own needs and to put the end
user before yourself but at the end of the day the developer matters most and the project can wait.
Explicit expectations for each other matter immensely. It should never be assumed that everyone
is on the same page. Be willing to step out of your comfort zones and learn more in depth about
others areas within the project. However this relies on an important aspect. Those working in
each of those specialized areas need to be documenting their work or will be incredibly difficult
for others to pick up. Team project planning should be structured and them frequently.
Early on this was one of our biggest strengths as a team but eventually we began to neglect it.
Project planning is crucial for two reasons. One is that's for your expectations for the
upcoming release and two is that it assists with team cohesion and allocation of work.
On that note when planning every team should expect that as members will have less time
available than originally anticipated in ideal situations and so you should account for that in your planning.
And finally when you're starting a project that you truly care about and not some toy project
it's okay to stop at the beginning and to get to know those around you to communicate as much as
humanly possible and to not immediately set up roles and responsibilities because those should come naturally.
Finally we realize that no adults in the room accept us now. We had to step up to our own play.
We recognize that we still have a long way to go and that this is a substantial learning process.
In the grand scheme of things we are still quite young as developers and so with Rhino Linux as
a project having just recently passed out with free year anniversary of beginning work on it.
We still have yet to solve all of our problems and we are still figuring out how to manage no
longer having the time that we once did. We are still learning how to grow up. To that end,
however there is no endpoint to growth regardless of the age of a project and its developers
every project continues to grow up and team building is an eternal process.
We have seen it and surely you have too. We are not the first ones who have had issues with scaling
and we certainly won't be the last. Ideally older and more mature projects can be given guidance
to younger ones. Older projects have already suffered through the scaling problem so they are
great resources to look up to and reach out to for advice. As is the topic for this room,
there are several other talks today revolving around tackling burnout and sustainability.
And unlike close source and corporate development work,
bosses extremely personal to the individuals working on it where people often have very specific
roles and by and large the boss world is built through passion projects where that individualism
is on full display for better or for worse. So we take a lot of pride in our work and many times
when lacking pressure from the professional world it arises internally. In some, if you want to
be systems maintainers, you have to maintain your own systems too. As always, thank you for coming to our talk.
And now we have some time for questions.
It's not really a question because I think you answer all my questions to be honest.
It's hugely inspirational what you're doing. So just keep doing what you're doing, share with others
because it's true passion and it's coming through. Thank you very much.
I think so the talk was very inspiring. How we all prioritizing tasks and future releases?
What's driving that process?
Yeah, so typically we all get together. We like, or instead, we instituted regular team meetings
and that's typically where we decide what we prioritize. We've started recently picking back up
and using command boards, again, which I think is great. I'm a big advocate for them. So yeah.
The can-bem boards particularly allow us to decide what is a high priority, what can we save for
the next releases. It gives a very visual layout to all of the things that we're working on.
And we can also organize it by which areas of the distribution they're particularly involved in.
Thank you. I just want to say I really enjoyed that talk. He's really inspirational and it's
really enjoyed the majority of the show. I've learned a lot as well from myself, but what I can take away from work.
So I just want to ask what's the next step, what's next for Rhino?
We still have plans to try to scale ourselves back up again. We know that we are here to stay for
for a good time and we are still very much passionate about continuing the project.
We do want to continue to try to figure out how we can expand our team in more ways, but we have learned
some unproductive ways how to do that. So now we're going to aim for more productive
manners of that, which we think starts a lot with documenting.
Yeah. Any more questions?
Thank you.