So our next speaker is Colin Dean, he's going to talk about MacPython Devix,
so it's the red green refactor of one, with an old tool, to be honest, I have no clue
what he's going to talk about, so give him a warm welcome.
Thank you folks, appreciate you, and thank you for coming.
So I'm Colin Dean, I'm a software engineer, about 27 years, and now for about the last six years working
on a major retailer in the US, to most of my experience in the past six years is in fact.
So supporting team of data scientists, doing all kinds of deep learning,
G-learning, A-I, and the vocation data engineer.
I started out in PHP early on in 1999 or so, spent some time in the Ruby,
and Java communities learning how to program and how to get better program.
And then worked for about eight years in scholar before I found myself working in a little bit in roughly three
person time, and then came back to Python in 2021, having not touched it since 2005.
So this is a story of how I came back into Python and tooling and built some tooling in Python
that suit my needs, having worked in many other businesses.
I'm here from Pittsburgh, mentioned call it's an area of about two and a half million people in Norman Appalachian,
or Appalachian, and I'm going to come out of it.
In an area of the United States, we call it a RUSBOT.
So if my accent comes out, it's okay to ask me to repeat this.
I focus a lot on developer experience, I believe that good tools, and
it's not muted.
Can you repeat this?
No, sorry.
Maybe I'll just hit tunes with a little bit of Pittsburgh ease, and then you'll know what's got on there.
All right, so again, my name is Colin Dean, case you missed that.
That's very important.
I focus a lot on developer experience.
I believe that good tools and good practices enable building good software.
I'm a huge fan of addressing problems through this problem diagnosis and
remedy approach, consistent with medical and other scientific approaches to problem solving.
If you search for Colin Dean problem diagnosis remedy on the internet, you will probably find things
that I've written about using this for general proposals, as well as architecture decision records.
At a high level, I'll be talking about some of the pains that I experienced and
observed over the last approximately four years as an experienced developer working in
Python for the first time since 2005.
I'll show you how I solve those barriers to productivity using a very old tool.
48 years old this year.
Make.
I will say that this solution is an active development.
It is an active maintenance.
It is evolving over time.
There's some pain points here and there.
But by and large, it works.
Some things may have changed since I created these slides.
Originally this presentation is from 2022 and I've updated it approximately
yearly, including last week.
This talk inspired the longest blog post ever on our corporate blog.
When I turned it in, it was originally 30 pages.
We squittled it down to about 14 by the time we published it.
Most blog posts hadn't been more than about three or four pages at the time.
It also earned a coveted open source release in my company.
Eightth largest retailer in the US.
So, of course, I have to thank my employer for allowing me to open source stuff.
And I wish that more employers would allow their people to open source even
the examples of the things that they're doing inside.
It's a great portfolio work to think.
Let's talk about some problems.
Working with my teams of engineers and data sciences, data scientists,
we encountered these problems.
What Python version should we use for development?
What should be used in production?
How do I install Python at a very specific point version?
How do I ensure that my project works on a first development and production installation
based, and of course, how do I install the dependencies?
Some things that too many of us in this room, these are kind of easy to answer,
but to many of the people, these are really hard questions to answer.
If you're getting that same feeling that all of this could be easily written down
and automated, you're kind of normal.
But this is some of the questions that I had in my first week of working in Python
in about 15 years of not working in Python.
A lot of this presentation boils down through this question.
My diagnosis of this situation.
My team uses macOS for development and deploy solinics.
Apple tends to keep its Python 3x installation within Python and of life dates.
But it's always a little out of date.
A few point releases behind.
Apple removed the system Python in macOS 12.3 as they weren't in the 10.15 release notes.
If anybody reads it, Apple's release notes.
It's still included in Xcode, though.
That's how whenever you run Python and you're getting like user bin Python on macOS,
it's coming through Xcode.
Apple says don't rely on this and they could remove it at any time from Xcode,
because it's an dependency internal to Xcode.
This policy is similar to Hombrous policy on Python,
which is like, yeah, you can use brewed Python,
but really the brewed Python is there for us that use apps
that use Python as a runtime environment.
But Hombrous Python won't disappear in a release.
There are a ton of different ways to install and use Python.
My team is consistently bringing their own Python,
doing setting up however they wanted to, or whatever they knew best.
And this was proving to be unreliable and frustrating
because people's environments were different.
Their paths were different.
They weren't consistently installing new dependencies,
when they did a get poll, some basic things
that we might consider to be a little bit basic.
So we had to think about which Python I wanted to use.
Using the one provided by macOS was unreliable
because it'll change and probably go away.
Using Hombrous Python was hitting a moving target,
especially at the time.
You could just brew and install Python 3,
and you couldn't install specific minor versions.
Now you can.
Using Python is about as the same as choosing anaconda,
but anaconda, of course, gives you some access to other dependencies as well.
But anaconda is not free.
Anaconda itself is open source and free.
Using its repositories, configured by default,
is not free.
Using manyaconda is not free.
And our company found that out the hard way.
This suddenly there are still people who are using mac ports,
and guess where to hit give a fuck in the next track.
So hello to the next users out there.
It's okay if you don't know where any of this is,
because the developer experience in the end
should abstract away most of these details.
You shouldn't have to worry about it.
And it should be done on a per repository basis,
so there are minute differences between a corny copia of different repositories.
You don't have to think about it.
You can just run one command and all this works.
Really, we just didn't have a clear single source of truth
for what version of Python we wanted to use,
or what Python, where we wanted to get it from.
While many of our projects are tied to a particular version,
because of a hard requirement on our cluster,
Python 371 installation from 2018,
we still had this unexercised freedom to choose
the Python we wanted to use in other contexts,
including containers and similar.
Since originally created this system in this talk,
we have moved to containerized deployments,
so we can do anything we want inside the containers within limits.
So it's very lovely no longer to have to run Python 371 in 2025.
PEP is a venerable tool and under the hood,
even poetry used PIP in the past doesn't anymore.
And of course PIP was built outside of the time of pie project,
so although that's changing soon.
PIP projects take some getting used to.
I think we're a little bit more used to it now in 2025,
and we were in 2022.
But even newer tools like Hatch, PDM, UV,
all have different ways of installing dependencies,
of showing dependencies, showing dependency tree,
of packaging, building publishing.
All these steps are ultimately automated.
If a repo isn't using poetry,
using something else, or needs to use PIP in some kind of intermediary step,
then it's best to script these actions instead of leaving them inside of a read me,
which is so many teams do.
Python uses have a tendency to expect things to just work.
You know, not have to think about compilation,
not have to think about building packages really.
Originally, it talked here about compiling
because the systems that we were using were too new for the packages.
But over time, I'm starting to see in the last about six months or so.
The exact opposite problem,
where legacy code bases that are using then current versions of things
are having to recompile again because those dependencies
are no longer running on the newer triplets.
We've got a newer operating system, a newer Python version.
We now have to go back to compiling because the wheels aren't available for those particular versions.
So, there's a time when I was giving this talk when it was switching between,
like, oh, you don't really have to compile anything anymore.
But just over the last six months, I'm sorry,
and I see things can pile more often.
As a safeguard, we have to plan for this.
We have to plan for the possibility that we might have to compile dependencies again
in the future sometime.
This is all scriptable, but, you know,
any sufficiently advanced collection of scripts,
just re-implements a direct, a DAG build tool,
directed a cyclic graph.
So why don't we just use one from the start?
The remedy.
We need a task runner for onboarding re-sync.
Something that is going to be easily executable and not just better documentation.
Because people don't read documentation.
So let's install it.
From something else, we're not bound to an off-benders whim.
We're all using the same versions, the same tooling with the possibility for,
as similar of the experience as we can get,
to build a common vocabulary, common procedures for developing and debugging our code.
Of course, as I explained, there's a ton of different ways to use and install Python.
And it's a ton of different package managers and now, of course, with UV installing its own Python.
Most of the time we were choosing either home brew or a condo on our team.
So I have to return to this question, again, which Python?
And the clear way forward is to explicitly say which and provide a completely mindless way to install it.
I think almost all of us have seen this same as comic from 2018.
From XKCD, fun fact people used to think in college when XKCD first came out, used to think that it was me.
Because this is my style drawing.
I did deny that.
Random one rose too good.
I chose Python at the time in 2022, and I probably make the same decision now, still.
Mostly because it seemed to be the lowest barrier to entry.
The most scripting friendly is random dash, so I knew it very well.
We could choose the exact version of Python.
We wanted down to the minor version and void changing it from underneath the developer,
whenever there were point releases as home brew would do sometimes at the time.
I allowed for the user to continue to use condo or home brew, managed by fonts,
just in case what I was doing was going to mess up their productivity.
It was incredibly important for developer experience for me to not mess up their workflow,
even though I was trying to push a new workflow upon them.
Home brew is a base requirement for all of our projects, all of our setups.
A little biased because I'm a home brew maintainer and it's project leadership committee.
It wasn't at the time, but I'm a little biased.
Advancements in build tools feature to install Python automatically.
Like UV does now, and that may significantly alter this pattern in the future.
I might talk about that a little bit more later if I have time.
It's settled on poetry as most new projects in 2021 and 22 were doing.
It was made sense that were a pip-end.
It's still a great choice in 2025, though UV has kind of got its rocket boots on towards the goal line.
There's a ton of ways to install poetry too.
We have this problem of which Python and how do I get Python, and now we have this problem of which poetry and how do I get poetry.
The poetry maintainer strongly encouraged using their installer or pip-ups.
Their installer is mostly just a wrapper around pip-ups.
While poetry can be installed through home brew or via pip, upstream doesn't really want you to do this.
They prefer that you're using their particular installation methods, and that's what I've always tried to do.
If you don't follow that, you can break through poetry installation, and I've done that plenty of times.
While running poetry, it'll create its own virtual environment, and then create a virtual environment for the code base.
So if you're just installing poetry for the first time, it creates its own virtual environment, and then creates one for each project that you have.
And then of course, the user can construct poetry, exactly which Python executable to use.
Newer versions of poetry will figure out based on pion or some other things, which version of Python is installed that meets your requirement specified in the poetry configuration.
I rejected a few commonly suggested solutions out of productivity concerns.
The big one being containers.
Containers are just too slow for what we do on our max.
We're running Kalima.
Containers on Linux on Mac, basically running it inside of the VM.
It's just too slow to do real world data science work.
Especially when we've cut these like $4,000 max.
It seems kind of silly.
I needed some glue to get all of this automated from some common tooling.
So of course, the greatest common denominator is make.
I've long used a minimal make setup in every ecosystem that I've touched.
I've lost shell scripts of course.
Java back in the day.
To provide a common onboarding experience.
I've wrapped it around Maven, Gradle, SPT, Rake, whole bunch of them.
Make is very widely available.
It's built into Mac OS.
It comes with Xcode.
It's never going to go away.
At least enough of a make installation.
There's always going to be some difference between a new make and BSD make.
Most of the time, we can build a make file, configuration for make.
And we can easily share it across Python projects.
You may get into some copy paste, but ultimately it is very portable.
It's going to fly through a introduction to make.
Very simple make file.
When you run make task, it will run command that does something.
It will run every time that you invoke it.
Sometimes you'll see dot phony above this to say, this task doesn't actually produce anything.
Other basic use case for make is to produce files.
Make was originally intended to compile see.
And still, of course, why the use for that purpose.
It also makes it really easy to add into Python projects where you have some non-python
Transformations need to happen, whether it's like templating markdown or something like that.
Full make tutorial is certainly out of scope for this presentation.
Longer versions of this presentation, I might delve into it a little more, but not right now.
It's 45 years old, well, 48 years old this year.
So you can find lots of stuff out there on it.
One such being this URL there where I've built a workshop that goes pretty heavily into make.
Make file tutorial is another great one, not mine.
So why not something else?
I've used most of these and almost always end up coming back to make to wrap behaviors about these that I can't remember.
Principle engineer Haroku many years ago said that every make file in the company every repo in the company
needed to have a make file that had at least those six tasks.
Make depth check test build help clean.
Depths installed dependencies check ran lints test ran test build built a package.
You can figure out what the other two mean.
One of the clarifier reason why I'm so adamant about developer experience and this particular way of realizing it.
It's important to have easy on ramps to code base is because it may not be somebody's new to your team.
It may be you who's lost a computer and it's coming back to work that you started yesterday.
It may be you six months or a year from now.
Recently I picked up a code base that I hadn't that I started and hadn't touched in four years and I could remember a thing about it.
But I ran make depth make rest of the command everything is worked.
I'm pretty limited on time so I'm going to skip through this a little bit.
We have lots of good online help built into our make files such that we have a description of each main task and what the task is.
Lots of things for installing dependencies.
We want to install things from home brew.
Peru is a great ecosystem independent package manager.
I use extensively for my slides and so on and so forth.
Easily run tests build a package and publish to our internal pipi system.
Then of course we have plenty of code quality checks the gamut.
Nowadays we mostly just run rough but back then when we created this we had to want more tools.
This some challenges in this in my last two minutes here.
As the fourth generation of max rollout, fourth generation of arm max rollout to developers.
We're still having to support Intel machines sometimes.
We're having to compile less far less often but we still have to compile sometimes.
As I said earlier I think we're going to probably maybe compile more in the future because we're probably not going to update our dependencies quite so much because that takes a person time to do that.
This is an example of what we had to do in order to enable compilation inside the make file.
Again if I had a little more time I would go over this in greater detail.
But ultimately boils down to either running with a whole bunch of flags or just poetry.
So when we would run this on Mac OS arm 64 you'd see this actually running on Linux or Intel Mac which is be poetry.
Hopefully this was enough to let your whistle and make you interested in providing a really easy way to get your projects both open source and inside your companies with a good developer on ramp.
It's something that anybody can approach and it just works on their machine.
Some improvements most of the teams that I was working on of this banded where it was originally working on this but we've all been these.
Seeds in a flower and we've now spread ourselves all over the field that's been interested to see how this changes over time.
If you're interested in learning more please talk to me.
I'd love to share the original version of this set up and you can also see my slides here.
Check the matrix room in the left link in there as well.
Thank you very much for the nice talk.
If you have any questions then please go directly and then ask questions all right in the chat.