So, our next speaker is David Wujis, I hope you're not such a wrecky.
Good morning.
He's going to talk about parts of the monorepers.
So, give him a warm welcome.
Okay, do you hear me?
Good.
Okay, welcome to this session about Python monorepers,
and the pololith developer experience.
So, my name is David Wujis.
I'm a developer, and I live and work in Stockholm, Sweden.
So, the developer at a consultancy called The Beolite.
So, I'm mostly in Python, like, today I'm, like, 100% Python,
but I've also been working with, like, functional programming, closure,
and a lot of Node.js, and I was many, for many years,
in the Microsoft space, like, dot net, and C sharp, and stuff like that.
But today's, like, all in Python.
So, I would like to, like, begin by try to, sort of,
like, the definitions of this terminology.
So, starting about monorepers, like, Python monorepers.
So, what's that, and how does that work?
And this thing called pololith, how does that come in,
and what problems does it solve?
So, let's say that we have, let's say that we have our
Python service or app.
Maybe it's a fast API service with a lot of endpoints,
and it's kind of solves every business problem in the place you work today.
So, a monoreper wouldn't make any sense there,
because you have just your one service.
But, at least, from my experience, the last couple of assignments,
the companies, sort of teams that I've joined,
we have, like, build larger apps, or maybe platforms.
At one place, we have developed an e-commerce platform.
So, a common approach to these bigger things is that you use,
like, a microservice approach.
Like, you separate the different parts of that platform or app
into separate deployable artifacts.
So, maybe, let's say that you have an e-commerce application,
or platform.
Maybe you have one place where you deal with your products,
and a different place where you deal with your,
with your, with the prices of that project, products,
or maybe stock and ordering and stuff.
So, they are, like, different parts.
So, the thing that I have noticed, at least,
the place I've been, is that when you have these microservice set up,
many different services, you often have them in different repositories.
Different git repositories.
So, you might have, like, you start small,
and maybe you develop new things.
So, it starts to grow.
You have more and more services that you deploy and run.
So, I think, a common approach for teams that has this set up,
is that you usually agree on that.
It's a good idea that your services,
let's say that all of them are Python,
run, sorry, are structured in, in this,
kind of same kind of way.
So, you use the same tooling, like, linkeding tools,
type checkers, and even the folder structure.
So, when developers are going to,
solve problems in different replos,
they will kind of know the structure and how to use the tooling.
Maybe you use make or just or something like that.
Oh, I didn't start this.
Okay, so, but after a while,
you know, after some time of the development,
maybe all teams doesn't work on all services all the time.
So, you might have started to diverge a bit.
You develop your new services and your,
you use maybe use new tools, maybe,
new versions of the same tools or different,
like, maybe you had Sonic as a web server in one service
and you have fast API in this service.
So, it's starting to diverge because, you know, things evolve
and tools like are developed.
And, also, maybe some, you get new ideas on how to structure
structure or your code.
You see, this third one has this source layout.
So, things might start to look a little bit different.
I think that's at least a risk or a probability.
And, but our risk of having these things is that,
let's say that this team works on this service
and they need this problem to be solved.
And they know that that problem was solved in this service.
So, we need a little bit of code from here.
So, the easiest approach is just a copy and paste it.
And, and I think that's quite common that you do that,
but the risk of that is that you have starting to have
duplicate duplications in your code base,
like you duplicate it code in your platform.
And, the worst scenario is that if that code part of the code
has a bug, then you need to go all of the,
or if you need to change something, you need to go to many places
and change that.
But, common way of solving this, these kinds of problems
is to extract the common code into libraries.
So, you can do that.
But, then you will have another repo to maintain.
And, you know, libraries really good,
but then you need to have an agreement on versioning,
maybe you semantic versioning.
And, and if you want to make sure that most of the services run
on the latest version, you need to keep all the services up to date.
So, there are a couple of trade-offs and like risks
with this kind of set of, especially as the platform grows,
and you have starting to have a lot of services and a lot of code.
Also, you're going to see something about libraries.
Yes.
You know, some code is like almost too tiny, too small
to start creating like an entire library of it.
Maybe it's like almost like code snippet.
You know, like maybe you have like this dictionary parsing thing
or how to set up a logger that you want to do in the same way
in all the places.
So, it's maybe it's like not worth making a library
and versioning and deploying things like that.
So, it's just easy to copy and paste.
So, one solution to that problem is to
would be to just grab all the code and put them in the same
git repo, same repo.
So, and that would be my definition of a monor repo.
So, you have all the code, but you deploy separate things from it independently.
So, a monolith, it's like a related to it,
but a monolith is just if you have your repo and you have one thing
that you deploy.
So, a monor repo is where you have several, all the code in one place,
but you deploy several different things.
But managing a monor repo can be really, really difficult.
I know I've been to join in teams that has, like a monor repo set up
and with, you know, custom scripts and simple links
and all this kind of magic to make everything work.
And when you like open your IDE, at one place,
you couldn't even navigate the code because it was like
the editor didn't really understand the references.
So, there are problems with the monor repos.
It's not like the perfect solution just because you have a monor repo.
But this is where this pololith thing comes in.
So, pololith is an architecture or, well, it's a way of structuring your monor repo, basically.
And it's really, really simple.
It's like almost like it's too simple.
When I was introduced to this, I was like, you know,
if it was this simple, why didn't anyone come up with this before, you know?
So, but the basic idea, the main thing is that you structure your monor repo
and you have three top folders, one called basis,
and now the components in a third one called projects.
And if we see, let's say that we have all these different services in different
recalls, what's common between these is that they have some sort of an
interpoint. If it's a fast API service, you have this like, you know,
where you define your interpoints, when you're with a decorator and things like that,
or if it's like a CLI, you probably have a main function.
So, they probably have some sort of an interpoint.
And those things you would put in the basis folder.
So, ideally, you basically have that interpoint,
but it doesn't do much logic.
Logic, it calls the other code in your interpoints repository.
Interpoints should be like thing.
And the rest of the Python code, you put in the components folder.
If you would just copy it, it wouldn't look like the ideal polylif way,
but eventually you will end up in having a list of namespace packages
that are ideal are very thin.
So, let's say that you have this folders structure,
but you identify the different logical parts of it,
and extract them into smaller namespace packages.
And that's what's called components.
So, the ideal is that they are flat and small.
And it can be like a tech component.
It can be something that does dictionary parsing or something really simple,
or it can be more feature like maybe like a Kafka consumer producer.
So, it doesn't have to be the same size.
It can be different sizes, but logically they are like separate.
So, you probably would have a long list of different components there.
And just by looking at that folder, you will actually get a quick overview
over what's in this monoray.
If you have a fairly good naming in your namespace packages,
you will understand what's in there.
And the third part is projects.
And this is where you would put all the Pi project homophiles.
So, you don't have any Pi fan code in there,
but you have like your thing that you want to deploy,
you have the infrastructure of it.
Like basically the Pi project homophil,
maybe Docker files and things like that.
The configuration of the actual project.
And the idea is that each project reference
to the code that lives in the components and bases.
If there are several projects using the same component,
they will use the same version of it.
And this thing is what the structure of this is what's called
the Ponylith workspace.
So, you have the bases, components, and projects.
But what we also have at the top level is a Pi project homophile.
And that will contain all the bases, all the components,
all your code, and all the third party libraries,
that you use across the monorepo.
So, that means, and this one is meant for development,
because we develop at the top level.
The project's folder is basically where you deploy things.
But when you're doing development, you are working at the top level.
And that means that you have one virtual environment,
and that virtual environment will have all the code in your monorepo.
And that also means that you have one set of linkeding rules,
one set of type checking rules, and all the good things
that we usually have in our projects.
You have one set of rules for the entire code base.
So, the development experience is basically like a monolithically.
So, you have this kind of monolithical developer experience from it.
And the developer experience is so important that you even have a separate folder for it,
because you might want depending on what kind of where you want to begin
if you want to start a new project.
Some people want to start very way defining an endpoint and work from there.
Other might want to experiment and try things out.
And if you want to do that, you have a place where you can put your,
if you use like pie charm, you have you know scratch files.
It's something like that, but you will version them.
You have your file, your Python module in the development folder.
So, you can keep it an experiment and when you're ready,
you can extract code into your component, maybe, or a base.
So, it's like a developer, like a playground.
You can also have like scripts, things that don't fit in,
like in a regular structure, maybe you have like one of scripts
that you run like once a multiple.
So, you can put them there too.
So, I mentioned the polarity thing is an architecture,
so looking at it from like a sort of a high level.
So, we have something called bricks.
And bricks is the common name for components and bases.
And bricks, polarity really likes Lego.
So, it's about, thank you.
It's about Lego bricks.
And that's why we have a folder called basis.
Because if you, if you have not developed Lego, but built Lego,
maybe you have used a Lego base plate where you put your bricks on top of it.
So, the base is the thin layer, the entry points to your app.
So, that's the kind of the metaphor from a polarity perspective.
And the Lego bricks that you build things from is the components.
Then we have something called libraries, which is basically in third-party libraries,
dependencies that we use.
So, fast API requests, pedantic, all that good stuff that we need in our project.
So, the combination of a selection of libraries and a selection of bricks becomes our project.
So, in Poland, we have one or several projects.
And each project just picks their bricks and the libraries that it needs to be deployed.
So, and for Python, we have, with this kind of setup, you probably won't want some tooling support.
So, this is the thing that I have developed.
It's called Python tools for the polarity architecture.
And it's just been picked up in the Python community.
And I'm really, really, really happy about it.
And it's like, if you want to, if you visit the page, you will find a discussion forum where people have like,
kind of active forums.
So, I'm really, really happy that it's at that state now.
So, the Python tools for the polarity architecture is, basically, the main idea is that it will visualize your monoraper.
So, you can see what's in there, what are the smaller parts, what is the big picture.
And there's also like helper commands.
If you want to create a component, if you want to create a project,
you will use that by templating, basically.
So, one of the commands, the most straightforward one is the polar infocoman.
So, it's type, polar, and then there's a command.
And the infocoman will show you an overview of your monoraper.
So, I around this command in my example repository, which is tiny.
It's like simplistic, and it's almost like it's two simplicity.
It's very little code, but in a real world monoraper, you will probably have a lot more code.
So, what this command will show you is all the bricks, all the basis and components.
That's why they have different color.
It will also show all the projects that you have in your monoraper.
And it will show you the connection between the two.
So, you can see where one brick is used in which projects.
So, you get this kind of overview of your repository.
There's also a Libs command, where you can find out what third party libraries are used in bricks.
So, you can see, okay, this brick uses PyDantic.
And you can also see where the libraries are used in which projects.
And then we have the Deps command, which is, I use it mostly for like refactoring purposes.
Because it shows you the relation between the basis and components.
So, if you remember, the components are like Lego bricks, but one component can be made out of other components.
So, it doesn't have to be like a one, one, two, one.
If you have a small tiny, like utility component, it might be used in several other components.
So, this command will show you like a matrix of how they relate to each other.
So, again, this example is like, so it will just output a tiny matrix of a very little code.
But in a real world scenario, you would have a lot bigger.
You can also zoom in to an individual brick to see, I think, something to a component,
also a brick called the message.
So, I can see where it's used, and I can see what it's using.
So, you can follow the kind of employee usage as well as that command.
And I think this command, Polydiff is probably, I think it's one of the most useful,
especially from a deployment perspective.
Let's say that you change something in one brick.
In this example, I'll change the Kafka brick, the Kafka component.
I'll change something.
And if I just, if I'm going to merge that into the main branch,
I don't want to deploy all the project all the time.
I don't want to deploy everything all the time.
But I want to deploy the projects that are dependent on this change,
or affected by this change.
So, this command will show you.
So, the Kafka brick is used in these two projects,
but it's not used in these other projects.
So, this is useful information for your CI,
so you can determine on how what to do when emerging code.
There's also like CI, friendly, shorter, so you can programmatically figure things out.
I think this was my last slide, actually.
And thank you for listening.
Any questions? Yeah, I think you were first.
Thank you.
I thank you for the wonderful talk.
If you can only have one virtual environment when you're developing,
how does that work with different projects,
need different Python versions?
Good question.
So, actually the basic idea with this pollinating thing is that you
ideally have the same version across your entire web,
because everything you have there is shared code.
So, that's the ideal situation,
but if you need something different for a while,
then you can create a new component that has that separate version
in your development environment.
You can switch on and off the different components,
but ideally, you should have the same version across the entire monorepo.
Thank you.
Could you make a brief comparison with other tools like poetry?
Please.
Yes, thank you.
Actually, the tooling that I have developed
is built on top of tools like poetry,
because I didn't want to reinvent how to import,
sorry, install dependency,
because poetry and all of these other tools
has really good support with that.
So, I built it on top of tools like that.
So, if you're a poetry user,
there's a poetry plugin for polylith.
So, if you use UV, you use a separate CLI.
So, it supports, I think, almost every like patch,
MPDM, a lot of these package management
and the dependency tool.
So, it's not a competitor to poetry,
but it's like an add-on, basically.
Hi.
Thank you for your talk.
I think we're interesting,
a very interesting problem here,
relevant to many of us.
So, what I can see happening is,
let's see if component doing AI training,
role training, and component doing an API.
I can see it happening that I have a brick
for my AI model, and that's then used by another team
another dev for the API,
which is not what it's meant for.
How do we avoid that?
Okay.
So, the other team use it
because they think they want something in there.
Yeah.
Okay.
I think maybe that's like a communication problem,
but if it's possible, maybe if you could extract the parts
that the team needs from that AI component
to separate components.
So, the AI component can use it,
and the other team can also use it.
So, my suggestion would be to try to split that part out
to find the shared need for that everyone needs.
Basically.
Okay.
That's communication really.
Yeah.
Okay.
Thank you.
Hi.
My question is related to testing.
How would you structure your test,
and within CI?
Is it possible to only run test for a specific project?
If you have components sharing,
not projects, and things like that.
How do you optimize that,
to not run your full test with every time something changes
for one component?
Yeah.
Good question.
So, Ponellith doesn't have its own test runner,
so it relies on, if you have like,
Pite, the Piteist, or any other tool,
but what it can deliver is a list of the changed bricks.
So, like, Piteist has this query,
or a marker feature.
So, you can use it.
If you have marked up your tests,
or named them in a special way,
that's kind of related to the brick that is changed,
then you get a nice query from the Ponellith,
from Ponellith command.
Right.
Thank you.
Thank you for all the presentation.
I have more of a bit of a hypothetical question.
Let's say I have a project where I predict
soon I will have multiple components,
but currently I have like multiple projects.
Do you, does it to expect you to have one single
or virtual environment,
or do you make a virtual environment project?
How does it handle it?
So, the recommendation is to have one virtual environment
at the top, so you have everything.
But I know that some Ponellith teams today,
because they want to develop isolation
for that specific project.
So, they can navigate down to the project
and start a virtual environment there.
But that's not what I would recommend,
but you can do that if you have these needs
and need to separate things.
But the ideal situation is that you have
a virtual environment at the top,
and one virtual or whatever you want to have it,
but it's just one virtual environment.
So, you have access to everything.
We just wouldn't it cause it like a bloated virtual environment
where you might get lost in the dependencies
that's what I'm afraid of.
Just lots of questions.
By having one virtual.
Yeah, like I have a completely like separate,
I'm a data engineer, so I have a completely separate data processing
versus some like DVD, which is really different package.
Okay.
And there's a completely separate.
Okay, I think is mean that the virtual environment
will contain everything, but the team only needs this part.
But I think that's a trade-off, but I would go for that anyway,
because then if you find out,
there's a connection between the two teams
that, hey, both of us needs this part.
Then you can start share code.
Even though they are completely different business domains.
So, that's what you will probably discover
when you have this monorepo and this polyester
that's probably more things in common
that you might think in the beginning.
And just by separating out the code in smaller parts.
Yeah.
Perfect.
Thank you.
So, thank you very much for all these questions.
And thank you to David for this excellent talk.
Thank you.