Okay, so we're going to continue with the next talk.
Yarek Potchuk is going to tell us a bit about his experience with using UV on Python
mono repos and he's working on the Apache F-Low project which has a huge mono
repo to maintain and I think he's one of the maintainers.
He's going to tell us about this.
Give him a big hand.
Thank you.
Hello, everyone.
I brought something today what you wanted badly.
A little bit of airflow.
Of course.
I'm an Apache airflow PMS member and cometer.
They actually look at the who here knows what airflow is.
How many?
Oh, yeah.
Oh, this is almost everyone.
So I don't have to explain anything.
So I'm on a number one contributor, cometer, in terms of number of comets.
I'm also a member of the Apache software foundation.
Here is our new fantastic logo that we have and sure you can visit our booth at the K Building.
If you want to ask about the foundation, I'm also a member of the ASF security committee.
And I'm going to talk about airflow.
Unfortunately, I will not break the real airflow sorry for that.
Airflow is a data orchestrator and I was thinking for 30 years in a choir.
So for me, the metaphor of explaining what airflow is and like this this decision was made by my wife.
So that you know, people ask me what airflow is.
So airflow is a kind of conductor.
The conductor that directs data processing in data pipelines.
It doesn't do any work like conductors don't do much work.
They just wave and then ball and that's it.
That's that there is an old joke about that.
So airflow is conducting the data processing in pipelines.
It's one of the most or actually it's the most popular orchestrator out there.
We are holding the first place and pretty much any big name that you know is you all those.
You probably know all the logos in there.
Everyone here uses airflow.
We've learned at the last Airflow summit in Seattle last year.
Open AI uses airflow for all the data pipelines training their models.
And that was like okay.
Okay.
If open AI can do it and anyone can do it and that was very good learning for us.
Very quickly since everyone knows airflow here.
The the our pipelines are just tough connected with with some predicates with some conditions.
And you build your pipelines in Python.
Hence the Python the room.
Of course.
And we have this really beautiful UI especially Airflow tree abroad a new UI written in React.
Which is optimized right now to actually manage those pipelines.
So the whole idea is like you have you can have huge pipelines across all the organization.
All the departments thousands of people developing the pipelines.
But then you have can have like one or two people just using the UI to manage those pipelines using the UI.
Because it actually helps them to understand what's going on.
So that's basically the value of Airflow.
So I've been always working as a CICD dev and person in Airflow since I remember.
And for years we had a very complex setup of our development environment.
Especially considering that we were for Airflow 2 which is like five years ago.
We decided to split Airflow into a back then maybe 20 30 separate distribution.
So separate PPI.
PPP install something you could install Airflow and 20 30 modules or distributions.
Right now it's around 100 distributions we have.
And we've always been struggling with how to make the development environment working for this because it was like,
Okay somebody developed just a core Airflow somebody developed just a particular providers as we called them.
So one of those 20 or 100 things right now.
And the whole setup we had didn't, wasn't very quick, wasn't very UX friendly.
Well I developed it so it couldn't be here.
I'm an engineer.
So we developed it initially to very in a very custom way.
Pretty much because there was no support from the packaging ecosystem to do it differently.
So we had to figure out our own way.
Very quick hold up.
So like this talk is all about is based on the blog on a series of blog posts actually four of them.
Oh fantastic almost nothing.
Yeah.
Oh now better.
So I wrote a series of four blog posts and this is the first one which is describing this and it's mostly about that.
Like you can find more more details there.
So first part of this blog post and these presentations of some page we had.
Explanation house standards are helping us now help us to overcome those those pains.
Then a few words about something that helped us even more which is crack.
If you haven't heard about crack I will you will hear of this presentation.
And then the last part is a bit of like innovation we are doing still in this space.
So we still are doing a little bit better than standards allow us and we develop something called shared leaps.
Which we hope maybe eventually we will become something that others can use as well.
So when you have this kind of big modular application you have the dilemma dilemma.
Either you can have like all separate repositories which in our case would mean 120 repositories to manage.
Or you have the giant monolith repository where you keep everything.
And I have always mean the the proponent of monolith and did everything to make it happen but it was like terribly terribly difficult.
We are like decayed old development or our flow more than 700 dependencies.
And we have to build everything in the way that it's easy to contribute to many, many contributors.
Our flow is the most popular ASF.
So a participatory foundation project in terms of number of contributors like we have 3,000 more than 3,600 contributors which is a lot.
So the idea is like how we can make it really frictionless for contributing to such a huge modular base.
And just to give you a little bit of overview of like what kind of difficulties we are dealing with.
So this is just one week.
163 merge pool request in this week.
70 out or 100 30 commits.
So like we have basically every day of every week we have more than 20 PR's merged and that's merged.
Not reviewed.
Like so that's a lot a lot of PR's.
You can see actually me here at the very beginning.
I usually I am usually at the beginning.
Second one is the bot which I wrote.
So like I this counts twice.
And the third one is the ends who actually people mistake with me because he's bald.
So similarly as I.
But he's actually almost 2 meter high.
So like it's very easy not not to mistake.
But the fourth one is also bot I wrote.
Okay.
We have a lot of people contributing I'm just joking.
So the cost of custom code was like we managed 100 distributions.
And that was a serious pain point because because it was completely custom.
Like you had to use the tooling that we had to build the dependencies to build the distributions.
We basically like copied the code, put it somewhere else.
So you generated dynamically by project almost set up by like it was like completely completeness.
And also there was no isolation between proper isolation between those those distributions.
Because we they all live in a single source 3 like single director structure which means that at any point of time anyone could add an import to any other part of the repository and it would work because it was the same.
The repository to even if it shouldn't.
So we had to add some artificial checking if we are using something from this to this director or that directory.
We used precomit for that.
I'll tell about that a little later.
And then the burden was was was was really high.
So fortunately Pi Pi team PIPA team like Python packaging authority.
Yes, that's right.
That's the right Hugo.
Yes, Python packaging authority together.
There is one of the core release manager of Python here last last release Hugo.
So there is a whole Python packaging authority team that was that were relentlessly working to improve the situation.
And I was fighting with them at the very beginning because I have other ideas and they they fought me and like that was a little bit difficult.
But then I understood why they are doing that.
And then finally they did something that we did a lot of things that actually made our life easier even though without it won.
So the PIPA I was focused on standard focused excellence of the ecosystem and I would say like five years ago and now it's like in comparable like.
Lots of standards that we based on our conversion of airflow to little the latest standards.
And also them which we I read there in detail we applied we made sure that we are following them meticulously because we knew that we have a lot of you know baggage and some things were implicit and not really following those standards.
So we just made sure we are following all the standards we can and that helps a lot.
However, there was one thing missing in the standards or not yet not yet implemented because nobody focused on it.
Yet something that we so so the first part it helped us to convert those distributions in we have in the standard like PIPA project.
Tom knows that apply any more all the dependence is nicely defined and all the stuff like it's it's a kind of super modern right now.
That was a more modern more than two years ago, but now we needed something to bring our the distributions together so that we can develop on them separately on each distribution but also we can.
develop on them all together because we want to be able to install all those distribution all hundred of them at the same time in the same virtual environment.
So there was a little bit like it take and have it to like either we want to have a separate distribution but we also want to be able to run them together with other distributions and for example be able to find out.
Whether we have a set of dependencies we have more than 700 of them which allow us to install it everything together without any conflicts which is kind of difficult as you probably know.
So.
Meet UV who didn't hear about UV.
Okay, I thought so like no, maybe you two hands here. So yeah, we adopted UV.
We struggled a little bit with that because UV is owned by by Astral. So it's it's an open source project that the governance is not like PIPA to like hat or or or people or whatever, but like we decided yeah it's okay for us as an opposite of the foundation for development to depend on it.
So we adopted it and we love it and actually the thing is like we will actually work with Charlie Mars and his team on in Astral because the workspace was largely modeled on the needs of airflow because airflow is one of the biggest problem project in open source very well known and one of those who had definitely the need for workspace.
So the initial implementation even had a provider in the documentation which was taken from airflow.
It's gone now like the different different set of documentation structure.
But generally what it is like it allows you to define a worst workspace where you define which of the folders basically in your in your monorepo which distributions in there which each of them is a separate kind of small distribution which of them are part of your workspace and it allows you to do various things within the workspace.
One of them is allows you to install the whole workspace together or run on full workspace together.
But also another very interesting thing is like you can allow distribution in the workspace to refer to another distribution in the workspace.
In a regular requires the pendency requires the standard Python pro.com.
But then it will not install this dependency from PIPA but it will use it from the workspace from the sources in the same monorepo which is like exactly what we need it.
So as you see our directory structure this is like a small fragment of our directory structure.
So we have for example airflow core and airflow CTL those are two distributions we have.
Each of them has its own SRC and test folder each of them has its own PIPA project home.
This is all fully standard.
If you enter to the airflow core directory is just see the subset of the tree and that's a fully flat regular Python project.
But this is all part of monorepo and you see all the blue ones are the same. So like we have lots and lots and lots of them.
So we have 122 or 3 PIPA project home files right now in our monorepo.
So how it looks like how it works. It's very simple.
Like this is like the top root of Apache Airflow repository.
And you are on UVC refresh all packages there are a number of options.
And as you see it looked and we solved all 913 packages we have there a lot.
And you might see me it's long like 10 second but I'll tell you the secret.
This was run on euro star train from London to Brussels with a very weak connection.
Like that's why it took so long normally it takes much much faster.
And the interesting thing that happens and this is something that is really the fantastic feature of that.
If you go to the sub directory which is this Airflow core this is our one of the distributions and we run UVC there.
It actually uses the same virtual environment which is at the top of the root of your workspace which is an interesting one.
And you see it's not 900.
It now uninstals 512 packages from those 900 installed before.
What happens here right now when you run UVC it tells you okay now in my virtual and I want only the things that my this particular distribution needs and nothing else.
So I don't have any other dependencies right now which I don't need as in this distribution because they came from other distributions which we have in our repository.
But but Airflow core doesn't depend on them so doesn't need them.
And this means that your tests your code completion you're all all of the things that you have.
They just see those packages that this particular distribution is so it's exactly as if you had a separate distribution in separate repository and you did just didn't know anything about other things.
Which is quite cool and it's super fast with UV because UV is you know catching a lot and able to switch switch virtual and immediately and recreate them.
And another thing for so this is one of the providers Amazon provider because like we have 90 of them right now.
So if you see what happens when you do you think in the provider it uninstall again many packages and you can see which ones it uninstals.
Because those are all the things that that are not needed by Amazon providers with only only get what that's what you need.
And that allows for example for one thing like when you hear you do in this directory you do UV run by test.
It will actually sink and run by test with all the dependencies that are only declared for this particular distribution and nothing else.
Which means that if you do it on the top level you have everything. If you do it on the directory level you have only this part that this particular provider needs.
And very interestingly for example if we have a dependency between Amazon and Google provider we do.
If we also install the dependencies that the other distribution in the same source three needs as well.
So it automatically resolve all the dependencies cross our distributions as well.
Which is like super cool because then we can run all the tests individually which we do in our CI only like you going to the directory and running UV run.
And this means that if somebody accidentally imports from a distribution from air flow import whatever which was not really cloud as dependency of this particular provider that we are testing.
It will just not run because it will not be there. It will not be installed. It simply and installs also all the source dependence source distributions that are locally available.
It has a very nice like it just surprisingly it just works.
The only thing that we need to do like it even works with IDE setup. So the idea is do not have yet like full support for this one or repo.
But like we develop a simple to script setup idea setup VS code. It just walks through all our distribution creates the right IML file or the VS code. I can't remember what it is.
And it will just get the project definition which has everything you need from all those repositories. So we can also very nicely work with IDE in this way.
But then if it's distribution is isolated.
You have to explicitly declare that one distribution depends on the other in order to use it in order to import this particular distribution.
So which is like super cool super critical for us because we are in the middle of splitting air flow even more isolating things and we just got it out of the box thanks to that.
And it's super flexible because you can very easily switch between distributions and sync things as you want.
So we would love to have a workspace as an industry model and now hat already.
So I work to the effect I even and co-out or of a path which is being long, long in a draft state.
I don't know, long in the not pronounced state yet. But I know of it and he even wanted in he did or the in hats which he owns.
The work space is implemented right now as well. So we can use hat work space implementation which is very very similar in small adult after the UV.
There is no standard yet for that. However, we had some one of one of the things we had a problem we had a precommit bottleneck who is a precommit here for a lot of people yeah.
So we also used school is nice but then single precommit at the top of the repository doesn't work well if you have 90 distribution because you want to have precommit in each of those sub distributions.
And there is this new kit in the blog Prec. I highly recommend is like one to one replacement in Rust. Many times faster uses UV to install everything and working with Joe is fantastic.
Comparing to working with the precommit author who didn't want to accept any of our proposals for years. Joe is very nice and implemented monorepo support for us and it works beautifully. We now have many precommit files precommit config files.
So Prec, Rust base, one to one replacement, it's monorepo aware and it has a number of very very cool features. For example, it supports inline script metadata.
So we can now specify inline script metadata in your scripts and you don't have to declare them as dependencies in your yam file and it will just run any style of your trial environment for each of them using UV which is like super cool we use it immediately.
Yeah, so there are monoreotex integration with UV all these things which are really cool. So we have now more than one precommit config in our report and we will have more because we haven't completed all the migration yet.
How it works, for example, you can go to the provider to one of the distributions pre-cran. It will only run those that are for this particular distribution as you would expect. That's that's beautiful.
And if you're running on top in RAM, it runs all of them of course.
So one thing that we didn't solve before early this year, try. So we have all those distributions.
For example, all our distributions now now we are splitting our flow into, I don't know maybe, 10 more distributions like separate for ui, separate for core, separate for scheduler, separate for figure error, separate for data access or for various reasons including security.
We want them separated, we want to have them really nicely isolated, but lots of them are using the same shared code that we want to reuse across of them.
And the traditional approach is like you build another distribution which is like common kind of which you import like which we depend on like shared logging whatever and you depend on it in both.
And then you have this shared code and that's a new install that dependence. But that has a trap because dry has one trap that not everyone is aware of is namely coupling.
If you do stuff which is dry, then you also couple things together which are using those dry, those dry code, which means that if you install for example one version of your logging library, all your users have to use the same version.
Airflow Python doesn't have like npm has the capability of using libraries different libraries in different versions of the same time. Python doesn't have that. So you only can install one library which have both good and bad sides.
And I think it's a good choice by the way, npm is not a good choice for those libraries not when you have to download half of the internet when you install when you run the same command.
So the problem is that yeah, I'm in the rather loads exactly one version of each library and then you have to like run testing for all the different version combinations.
It's very quickly gets out of the control if you want to use it this way.
So it doesn't really maintain when you for example one to use the same log here or config library in several distributions and you want to be able to freely change the versions of those distributions that you install at the same time.
This is this is like a problem we hand. So we have an inspiration from static libraries from C you probably know static libraries. So their code is embedded in the like once you link the library the code is embedded into this particular library. It's not installed from a shared library.
And Python has something similar like for years Python including people and including all the all the IP I managed things they use the vendoring to vendor either whole libraries or pieces of libraries in if you didn't want to have a library as a dependency just took the code change the module from which it was installed.
It made all the modifications so that imports are working properly even inside and you put it as part of your repository which we didn't want to do manually on staff. So we chose another solution and it's a very interesting one and I hope sometime in the future we will turn it into something reusable.
Our shared code is another this is another distribution which is inside our repository the same way as any other but then it's it symbol symbolically linked to a particular Python package in the package that is using it.
So we are not copying the stuff we are just symbolically linked it in the source directory which is very interesting because like we have to do some small modification in have linked to replace the links with the physical copy during packages but that was like built in feature that we have to configure.
We have to implement some track tools I'll get that to doting them on so track tracks to keep consistency how we are referring to those modules and it works only on certain environments because you know like windows doesn't have directly natively symbolic things so we have to use like you have to use the WSL to but that was not a limitation for our flow because that was always required to use WSL to.
And and it just works as we implemented it just works in the ideas without any modification which was a bit surprising for us.
So what it gives us is like oh you'll be seeing and track hooks they give us automatically the kind of guard lay guard raise that we can check if we have everything synchronized between like for example we have a shared library.
So our blogging library which uses track to log as a dependency this library is synlinked in a new folder inside the distribution that is using it but also dependencies of of this shared library as our synced to that user of the to that distribution that uses it.
I'll show that in the moment and we have some track tracks track hooks which are which are very fine if we are properly symbolically linked and refer to those this shared distributions.
So how it works so we have a configuration like air flow chart configuration.
So that's the definition of shared configuration directory in the shared folder or listener so that's so those are the three example shared libraries that we use in our code.
And how we use them those are just symbolic links that refer to those those configuration and they are in two places so those are two users of those libraries one of them is linked in air flow underscore shirt and another and the second is in air flow SDK shared so.
The same library linked in a different folder in different package which means that when those two distributions and are installed at the same time they're using library which was valid at the time of preparing of that package because that was used from the sources.
It was imported from one folder and the other can use the same library but maybe in a different version maybe in a version that was like week later or two weeks later but it is imported under into different package and they can be both used together.
Yeah so we have so we're using there are some configuration that is needed for.
Piper project on will without going into details all this is kind of automatically synced by our practice so we make sure that all the all the distributions are are used and we think to the future is modular so we found out that it really works for us very well.
And I hope in the future we can have a standard out of that thank you very much.
Thank you very much for the nice talk and fortunately we are out of time for questions so.
I'll be around.