And so now for the first real talk of the day, out, which is Victor, which is not
boo, with amazing, who we're talking about monoliths and very old monoliths almost as
all does this death room, which I'm very curious to hear about, because I'm also dealing
with very old codes around with a blast.
Welcome to my talk and modularizing a 10-year monolith.
My name is Victor Libaslowski.
I've been in tech for over 25 years and I'm currently a principal software engineer at
fleet device management at fleet, we make endpoint telemetry for corporate security
teams and device management for corporate IT teams.
With my current job, it took me roughly a year to figure out the code base, get familiar
with a programming language, and understand our business domain.
So around that time, I started to notice a consistent issue whenever working on our service
package.
The service package was the biggest one in our code base, and whenever I was recompiling a test
in this package, it took roughly 30 seconds.
So if I modified one character in a string, it took roughly 30 seconds to recompile the
package and its unit tests.
So I did a little digging, the gold programming language is known for fast compile times.
However, the linking of all the packages was dominating the compile.
From this analysis, they usually was obvious.
The service package was just too big and had too many dependencies.
I asked myself a simple question, what am I actually trying to do here?
I'm not trying to fight the build system or wrestle with a giant package.
I'm just trying to write good software.
And it shouldn't take 30 seconds of self therapy every time I change a string literal.
So if I just want to write good software, what needs to change?
I want a code base that feels light, understandable, and fast.
One that helps me think instead of slowing me down.
And I realized the only way to get there was to make the code base itself smaller.
So we tried modularizing.
Next, let's discuss our first attempt of modularization, whereas you might guess things
didn't quite go as smooth as we hoped.
Let's talk about our code base.
Fleet device management is primarily MIT license open source.
We also maintain a small set of advanced features in the clearly separated directory
under a different license.
Our code base is roughly 10 years old with over 500,000 lines of go back and code.
This is a much simplified picture of the core of our server code.
We have a single huge go package for the API layer.
It's our service package.
It includes controllers and the service containing our business logic.
The API layer calls the persistence layer.
The persistence layer is another huge go package.
It constructs the SQL queries and executes them against our my SQL database.
We also have an in memory cache, a React front end, a CLI, and a bunch of other components
which are not pictured here.
So at one of our engineering old hands meetings, I gave a presentation.
I described how we could scale our code base to be a more modular architecture.
The core idea of my presentation was that when working on a new major feature, we could
create a new module of vertical slice.
This new vertical slice would mirror the structure of the legacy code.
We would actually organize it exactly like the legacy code that way engineers would still
roughly know where things are.
This actually looks exactly like a microservice but still within a single compiled binary.
My presentation seemed to land and I felt confident enough to try it on a real feature.
So shortly afterwards, we started working on a new feature and this new feature seemed
like an ideal candidate to try the new modularization approach.
The feature was Android MDM, mobile device management.
I created the new structure for the Android support.
I created new packages and a dedicated directory.
Most of the work was actually on tangling several common parts so they could be reused
between the legacy code and the new Android feature.
I put up the PR, the pull request, and it just sat there.
Our guidance is to review and merge PRs within 24 hours but no one wanted to touch this
one.
In retrospect, I should have done a better job communicating to the team that I was actually
going to do what I proposed in my earlier presentation.
Some engineers felt surprised by the changes.
They didn't feel like they had a voice in their approach, they didn't have an opportunity
to speak up.
So facing this resistance, I'd backtracked.
I decided to take a step back and unsplit the my SQL layer so all requests would still
go to the same my SQL schema.
I also wrote two ADRs, architectural decision records, want to split the API layer and want
to split the persistence layer for this Android feature.
The ADR to split the API layer was approved but the ADR to split the persistence layer
was rejected.
Engineers voiced their concerns, their feedback was consistent.
This felt risky, confusing, and too big to do all at once.
So I reverted the persistence layer changes, then you API layer now talk to the same
large-share persistent package as the rest of the code base.
And at that point, I moved off the feature and delivery pressure to cover, without sustained
architectural ownership, the Android code slowly seat back into the legacy system.
So what do we have now?
The picture roughly shows what we have, certainly at a high level, the code base doesn't
appear very modular.
Arguably, it's even more complex and harder to maintain than it was before.
In practice, there's no clear separation between the legacy code and the Android feature.
So next, let's discuss what we learned from our first attempt.
I keep rediscovering that talking to people takes time and there's no shortcut.
Real-time, meetings, follow-ups, side conversations, and the bigger the organization,
the slower that loop gets.
We also can't rely only on who speaks up first, early voices are often smart and well-intentioned,
but they don't represent everyone.
To build real consensus, we have to actively solicit feedback, especially in one of the
ones and smaller forums.
To succeed, we need strong commitment, both from engineers and managers, to the architecture
and to the specific boundaries we're proposing.
Without that commitment, engineers naturally revert to old patterns and architectural compromises
creep back in, and without management support, short-term delivery pressure wins over a long-term
structure.
Here's a small example.
When I said modules, I meant architectural modules, cohesive code with clear boundaries,
but some people thought I meant go modules.
Nobody was wrong, we just weren't using the same language and it took a face-to-face conversation
to realize that.
We saw the hard part was the architecture, it turned out the hard part were the conversations.
Let's move on to the second lesson.
What is an architectural change anyway?
That sounds like a straightforward question, but in practice, it turns out to be one of
the most political questions in engineering.
Here's what we saw in reality.
Social changes in our code base rarely showed up as architecture work.
Instead, they went in as part of feature work.
If you framed your changes part of feature work and included it in a PR with other feature
changes, it moved, even if it broke existing architectural patterns.
After all, we needed to get all the feature changes in quickly, features of what make money
for the company.
But if you framed your changes as a standalone architectural improvement, it's stalled.
It's stalled because it was prioritized behind feature work.
It's stalled because engineers wanted to have bigger discussions around it, and it's
stalled because it wasn't providing immediate value to the company.
So what's the main lesson here about architectural changes?
If you don't define how architecture is allowed to change, it changes anyway and often it
changes behind your back.
Let's move on to the third lesson.
One thing we're missing was clear ownership.
In a modular system, not every API endpoint belongs cleanly to a single service.
Some endpoints mostly orchestrate work across multiple services, and this was true for
us because the new Android feature still needed to talk to legacy code.
The mistake was leaving that orchestration on owned.
Every API point still needs a clear home, a specific module and a specific team that's
responsible for it.
One of the endpoints is just glue, someone has to own that glue.
So where are we at this point?
The first attempt didn't quite succeed.
We still have the pain points of a large, tightly-coupled code base, but hopefully we learned
a few things.
Now we have experience.
So where to go from here?
At this point, I realized the importance of the people part in architecture.
And I felt like I couldn't convince the company on my own.
I still didn't feel like an architecture expert.
The first attempt was kind of a mess.
So why would other engineers take my next proposal seriously and not see it as just another crazy
idea?
I needed someone else's help, I needed something like an appeal to authority, I needed
to find someone else that had already done this transition.
And I needed to analyze the way they approached it.
Unfortunately, I did not find any large open-source go-projects making a transition to a
module on the left, so as far as I know, where the first large open-source go-project
doing this.
Yay, us.
The next best reference was GitLab.
GitLab runs a Ruby on Rails model list, which so they decided to break up into a modular
model list.
They documented the decision publicly, and it turned my proposal from a crazy idea into
a proven pattern.
We weren't trying to invent something new.
We were trying to follow a trail that was already there.
Next, how to start?
This time there was no obvious new feature to anchor the modularization work.
So we had several internal discussions with the engineering team.
We aligned on the main-driven design and bounded context.
But the hard question was where to draw that first boundary.
I looked at our code organization, had a few conversations with my AI friend, and proposed
several ways to slice the model list.
As you can imagine, this process is a lot to get your head around, trying to tease
a part of a 10-year-old system into self-contained modules.
I felt like I was barely scratching the surface.
I felt like I was truly missing a lot of the details and edge cases.
There was a lot of unknowns starting a modularization effort with anything beyond a bare
minimum seemed risky.
Although all the engineers generally agreed that we wanted to do an incremental approach,
we ultimately agreed on activity audit.
This context is about recording things that had already happened in our system.
For example, recording when a new user was created, when a new host enrolled, when
a key configuration was changed, and a ton of other activities.
Next, I wrote up a detailed architectural decision record and put it up for review by the
team.
Here's the link in case someone can't wait to dive into the details.
Again, this ADR was framed as a pilot.
This is the first bounded context we were creating before a bigger rollout.
Several times, I wonder where this whole thing was going to go anywhere.
Sometimes I felt like everything was on board, other times I felt like I kept running
into a brick wall.
I also felt I was deep in the weeds and details of this re-architecture.
I didn't quite see the full picture myself, and I had a suspicion that other engineers
didn't either.
I needed to convince myself and others why we were doing this.
I needed an argument, and I needed to use it not just once, but every chance I got.
So when I talk to the team, I roughly framed it like this.
Every successful engineering org eventually hits the same wall.
The monolith grows beyond what humans can safely understand.
At that point, teams start colliding to each other's changes, ownership gets fuzzy, and
small changes become risky.
We're already seeing that, and that's not the failure of the people, it's a failure of
the structure.
Modularization isn't about process or slowing anyone down, it's about creating clear
boundaries and ownership, so teams can move fast with confidence as the system continues
to grow.
All right, now let's talk about the actual architecture.
The high level idea is almost the same as where we started with the first attempt, then
you feature or new bonded context will be in its own module with its own controller, service,
and persistent layers.
The controllers and service will be able to call methods on other services in other bounded
contexts.
However, they will not be able to access the persistence layer of another bounded context.
Here's the directory structure, directly out of the bounded context, the service, and
my SQL directories match the API layer and the persistence layer from the previous diagram.
The integration test down here should be able to fully test this bounded context without
requiring parts from the rest of the application.
Using this bounded context with other bounded contexts will be done at a higher level integration
test.
I can talk about the decisions that went into this for a while, so catch me after the session
if you'd like to dive deeper.
Now let's talk about the decision not to split the database schema.
When people here module a monolith, they often assume that means multiple databases.
We deliberately did not do that, we kept the single schema.
Our problems were database scale or load, they were unclear ownership, broad service
layers, and code that was hard to reason about.
Splitting the database wouldn't have fixed that, it just would have added operation
and complexity and new failure modes.
Splitting a database is one of the most expensive architectural decisions you can make, and
it freezes boundaries.
We were still learning where those boundaries should be.
So modular, so we modularized the code first, preserved optionality, and decided with
only split the database later if scaling actually required it.
Now everywhere we turn, we heard recommendations to use ports and adapters.
However, a lot of our business logic lives in SQL, so pretending the database is just
an adapter doesn't make sense for us.
Ports and adapters doesn't work for our system as a whole, but one idea for mid-fits
extremely well.
That idea bounded context do not share the main models directly.
When you cross-context communication must go through an explicit own contract, in practice
that means explicit module boundaries.
Now, earlier I talked about how we ended up with large packages.
When a shared common package is used for cross-context communication, it creates a specific
problem.
When two bounded contexts share the same common types, they are no longer independent,
a change in one context silently changes the behavior of the other, at that point the
boundary only exists in our heads, not in the code.
So for cross-context communication, shared packages aren't reuse instead they're hidden
coupling.
That's why we require explicit own contracts at context boundaries.
When one bounded context needs something from another, it doesn't reach into its internals,
instead the providing context exposes a small explicit interface along with the types
that belong to that interface.
Those interfaces and types are owned by the provider, they live with the bounded context
and they evolve on its terms.
Other contexts can depend on the contracts but not on the implementation.
This is what makes the boundary real in code not just in our heads.
Now sometimes a new bounded context still needs data or behavior that lives in legacy code.
In those cases, we don't let the new code talk to legacy directly, instead we introduce
an anti-corruption layer or ACL.
The ACL is the only place that understands both worlds.
It translates legacy concepts, types and quirks into something that new contexts can work
with safely.
Notice that this ACL package is outside the new bounded context.
It keeps legacy semantics from leaking into the new code and then gives us a clear
scene we can replace or delete later.
The goal of the ACL isn't elegance, it's containment.
Now let's continue with the story.
Before we change the new code, we had one big issue.
By and if this was going to work, we didn't want this to feel like something bigger
implemented and we just had to live it.
Our software engineers needed to feel like the end result was theirs.
Not something imposed, not something slipped in quietly.
Something they actually owned.
We left the ADR open for discussion for about four weeks.
This was longer than any ADR we'd done before.
We wanted time for people to read it, question it, disagree with it and sit with it.
The goal wasn't past the approval, the goal was shared understanding.
Next we made a deliberate decision about reviews.
Every pull request related to this ADR would be reviewed by the four tech leads from
each product group.
They did two things.
Of course, if make sure they were a genuinely unbored, second it meant their teams were
represented, not surprised later.
Surprise is a great for birthdays, not for architecture.
Architecture stopped being something decided by one person and started being something carried
by the org.
We also didn't want the implementation to be rushed.
The work followed our normal design process and we made progress visible to everyone.
Visibility turned a risky architectural shift into a series of small understandable steps.
Now at this point, it may seem like we were putting a lot of process in place.
It probably sounds like we were saying, we need to have everything figured out before we
start.
And to be honest, that felt a little backwards.
This first bounded context was supposed to be a pilot, not a well-engineered masterpiece.
And a pilot is supposed to discover things, which means sometimes taking a step backward,
only to backtrack when things aren't working.
If we already had all the answers, if we were able to lay out all the steps from this
for this refactoring top-to-bottom, then we wouldn't need a pilot, right?
This means we needed to do a POC, a proof of concept, to discover things that we didn't
know about.
The goal of the POC was to expose hidden coupling and understand which boundaries were
actually possible.
So I started with a proof of concept, not a full implementation, just a scaffold, the idea
was simple.
I made the basic shape of a bounded context.
That means using some shared utilities for HTTP, some middleware, basic my SQL access,
nothing fancy, just enough structure for a dummy, hello world, and point.
Then I ran a dependency analysis and that's where things got interesting.
That simple scaffold was pulling in half our monolith, not because the domain needed it,
this hello world domain of the scaffold didn't mean anything yet, but because all of
these utility packages did, the glue code was the monolith.
The things we thought were harmless helpers were actually acting like dependency magnets.
And there was another issue hiding in plain sight, a God package.
Anytime someone needed a new type, I did nowhere, belonged, it went into this fleet package
at the bottom.
Over time that large fleet package picked up its own dependencies, which were not shown
on the previous slide, which meant everything that imported it, picked those up too,
and that's made circular dependencies incredibly easy to create and incredibly hard to
reason about.
That fleet package existed to make things easier, but it ended up doing the opposite.
At that point it became clear we couldn't just extract a bounded context right away,
we were missing a layer, before we could pull anything out, we needed a platform layer,
code that handled infrastructure concerns, HTTP helpers, database wiring, middleware,
without dragging the main assumptions along with it.
We needed to differentiate between two types of code.
The main code should express business rules and platform code should make those rules possible
to run.
Before we could modularize the domain, we had to modularize the foundation.
Another issue we had to solve from our first attempt was that even the good documentation,
even the ADRs, even the buy-in, nothing stops coupling from sneaking back in, especially
in a large code base, and especially nowadays when changes aren't only written by humans,
but also by AI coding agents that optimize for make it work and not preserve architecture.
Architecture that lives in people's heads doesn't scale, and it doesn't survive time.
That's why we realized we needed architectural checks.
Sometimes known as fitness functions.
Automated checks that run continuously check that don't care who wrote the code, human or AI.
Specifically, we needed checks to enforce boundaries, which packages can depend on which,
what the platform layers around allow to import, what a bounded context is not allow to
import, because once those rules are explicit, they stop being simply opinions.
They become executable.
Now all of that tooling matters, but let's circle back to the core problem we're trying to solve.
Looking back, the real constraint wasn't go, it wasn't my sequel, and it wasn't even the monolith.
The real constraint was human attention.
How much of a system, one person can hold in their head and how confidently they can make a change.
Once a system grows beyond that limit, progress slows, no matter how good the engineers are.
We often talk about architecture as it's as if it's for computers.
But computers don't care how big your code base is, humans do.
Architecture is really an interface for people.
It defines what you need to understand, what you can safely ignore, and what you're allowed to change.
When that interface is unclear, every change feels risky.
When it's clear, people move faster, even in a large system.
For a long time, I thought the hard part was architecture or code or tooling.
It was the hardest part was alignment, shared language, shared understanding, and shared ownership.
Modularization isn't really about modules, it's about respecting the limits of human attention,
and designing systems people can understand, trust, and change without fear.
Tools matter, patterns matter, but architecture lasts only when it's owned by more than one person.
So before I wrap up, I want to pause in these.
These are some of the teams using fleet introduction today.
They're very different organizations with very different constraints.
We learned a lot from working with customers like these, and many of the lessons I talk about
come from operating in this kind of scale under real conditions.
That's it for this talk.
If this story resonated with you, I'd love to hear about your own.
Problems are rarely neat, even when the code bases are.
Here's a few links about fleet. We are hiring go developers right now,
and you can find some links about me.
And thanks for listening.
Thank you very much.