Okay, so our next speaker is Johan Haaland and he's going to talk about faulty depths, undeclared and unused dependencies.
Give him a warm welcome.
Thank you very much. Hello, my name is Johan Haaland, and you can find me as Jay Herland on GitHub,
on Masad on LinkedIn, wherever.
I'm an origin, but I currently live in Delphs in the Netherlands, and I work as a developer
productivity engineer at Twink, which is a software consultancy within the larger mod is create platform.
At the Twink, we help our clients solve our problems with open source software, and we contribute back to the open source community
whenever possible. Today, I will talk to you about faulty depths. It's a dependency checker for your Python code.
But I want to start with some motivation for why this is such a big problem in the first place, and why we decided to build this tool.
So the following looks at the problem from one particular angle, but there are many different avenues that lead to why a Python dependency checker might be a good idea.
So one of the major problems in science and academia today is known as the replication crisis.
A lot of scientific results are being published, but then other scientists struggle to reproduce the same results.
And this is actually core to the concept of science itself, because if a result cannot be reproduced, then is it really science at all.
The field of data science has a particular responsibility here, because not only has it grown into a huge discipline on its own right,
but it's also become an important tool in many other scientific disciplines.
And Python, being such an important tool in data science, is also affected by this.
I want to briefly highlight two papers that have looked at reproducibility and Python specifically.
First, we have this paper from August 23 that looked at reproducibility of Jupiter notebooks in the life sciences.
These researchers collected a large amount of notebooks via papers available on PubMed Central.
And then they automated a process to copy these notebooks in attempt to reproduce the results.
I'm going to jump straight to the numbers they found.
A spoiler alert, they're not very good.
Here is from their abstract, out of more than 22,000 Python notebooks.
Almost 16,000 or 70% declared any dependencies at all.
Around 10,000 or 46% had dependencies that could be installed.
Around 1200 or 5.3% could be run without errors.
Only 879 or 3.9% could actually reproduce the results in the end.
So let's take a closer look at the notebooks that were run, but then failed with an error.
This corresponds to the yellow bar, but subtracting those that actually run without errors.
The most commonly raised error by far is modular found error.
And in position number three, we find the more general import error.
Both of these are closely related to missing dependencies.
And together, these two errors were raised by 29% of the total number of Python notebooks.
In other words, if we overlay these numbers and focus on the failures at each stage,
we find that 30% of notebooks fail to declare any dependencies at all.
For another 24% of them, the declared dependencies fail to install.
And even then, for another 29% there are errors while trying to import dependencies.
So when we look at everything together, it looks like more than 80% of notebooks failed to reproduce
because of missing or incorrect dependency declarations.
Let's quickly move on to the second study.
This paper came out a couple of months after the first, and it looks at the PI ecosystem,
which is a very central part of the overall Python community.
These authors start from the 10,000 most popular libraries on PI PI,
narrowing it down to 8,282 libraries that have more than two releases.
They then attempt to install each and every release of these libraries to find issues in the project configuration.
These issues could be related to dependency declarations,
problems that become apparent when you try to install and use the library from PI PI.
I'm going to shortcut to the most relevant numbers here.
They find that only 65% of PI PI libraries can be successfully installed and imported into your code.
Around 5% fail to install for various other reasons.
So around 30% can be installed, but may encounter errors as soon as you import them.
Depending on exactly which parts you import and what your runtime environment looks like.
Digging a bit into how they categorize the various failures.
We can see that the vast majority of failures are again due to missing or incorrect dependency declarations.
So both these papers show that missing or incorrect dependency declarations is a significant and important problem in the Python world.
But what do I really mean when I talk about missing or incorrect dependency declarations?
So let's quickly recap the basics of dependencies and dependency declarations in Python.
This should be familiar to anybody who has worked with Python for a while.
I import some third-party library like NumPy into my Python code.
When Python runs this code, it will look into my runtime environment for the NumPy library.
This is the dependency.
In order for my code to work, NumPy has to be installed.
But let's assume that I haven't installed NumPy.
So this is no longer a problem, right?
Well, what happens when you copy my code onto your machine and try to run it?
It fails because you haven't installed NumPy.
This is the works of my machine problem.
The code we run is the same, but the runtime environment is different.
And because of those differences, it works on my machine, but not on yours.
The real problem here is that I fail to declare my dependency on NumPy.
Now in Python, there are many different ways to declare dependencies.
I could write the Pyprojectomol or set up the Py to configure my project.
Setup.cfg is another popular alternative.
And we can even discuss whether the dependencies should be declared loosely.
Like here, I'm just depending on NumPy.
Orpinto-specific version like that.
However, this talk is not about those details.
So instead, I'll just go for the simplest possible declaration here.
This requirements, the Xt file, is the bare minimum that any tool would need to be able to discover our dependency
and help our users to resolve it.
By writing NumPy into this file, we declare that we expect to have it available when we run this project
and it is a declaration of our NumPy dependency.
So let's import another module.
Pandas into our code.
This is a new dependency, but we haven't yet declared it.
So, unsurprisingly, we call this an undeclared dependency.
And we can have the opposite problem as well.
So let's say that we experimented with TensorFlow at some point in our code.
And we did the right thing by declaring it in our requirements, the Xt.
But then later, we ended up not using it after all,
and we forgot to remove it from our dependency declaration.
This is now an unused dependency.
Unused dependencies do not directly affect the reproducibility of your project,
and therefore not as critical as undeclared dependencies.
But we still want to avoid them.
TensorFlow, for example, is several hundred megabytes,
and this can end up wasting a lot of space and time for our users.
So to summarize.
On the left-hand side, we have our real dependencies,
the set of packages that we actually depend on in our Python code.
And on the right, we have our declared dependencies,
the set of packages that we have listed as required,
either in our project configuration or in something like requirements Xt.
We can make a venn diagram out of these,
our undeclared dependencies end up on the left,
these make it hard or impossible to use,
and we produce our project,
and conversely, our unused dependencies end up on the right.
These add unnecessary blood to our project.
Obviously, we want for all our dependencies to end up in the center.
We want to properly declare the dependencies we actually use and only those.
As I said, at the start,
at the twig, we help clients across many industries to solve our problems,
and we often encounter projects with or without Python,
that struggle with reproducibility in one way or another.
A couple of years ago, a colleague of mine,
Nora, who then worked in the date engineering group at Twig,
found herself repeatedly in conferring projects that suffered
from undercare dependencies.
And she wanted to automate the kind of dependency analysis
that we just did in the previous slides.
Maria, also then in our date engineering group,
join her and a little later I also joined.
I'm not part of the date engineering group.
I'm part of the scalable build group at Twig.
So my usual day-to-day consists more of fixing build systems,
and less about date designs and date engineering.
But even if I usually don't work on the same kinds of projects and clients as Nora and Maria,
I still often see the same kinds of issues with respect to lack of reproducibility
and the works of my machine problem.
Along the way, we've had some more tweak colleagues,
jump in and out of the project,
and today we have Jihan and Richard with us as well.
Together, we have created faulted apps,
a tool for finding undercare and unused dependencies in Python projects.
Faulted apps is mostly a passion project that we work on
in between client engagements.
And yes, the name faulted apps is based on the
Montipython adjacent sitcom faulted towers.
The point of faulted apps is to do exactly the kind of analysis that we have just walked through.
When you run faulted apps on your project,
Faulted apps will find the set of packages that you import in your code
and compare that to your declared dependencies.
It then produces a report showing you any undercare or unused dependencies.
So you can consider faulted apps as a linker,
not a regular linker for your code,
but rather a linker for your dependencies.
Here, we see faulted apps being run without any options in our little toy project.
The output first lists the undercare dependencies
that faulted app found in this case, pandas.
And below, we see the unused dependencies in this case TensorFlow.
And if you run faulted apps in a large project,
you can ask for a more detailed report as well.
And here, we get the file and line numbers
where each undercare and unused dependencies is located.
So where can you use faulted apps?
We support Python code, whether it's in a regular
the file script or inside a Jupyter notebook.
We support any version of Python that is not ancient.
We run on post-exystems like Linux and Mac OS
and we also add a support for Windows.
For dependency declarations,
we support all the most common formats used in Python ecosystem today.
Requirements TXT is probably the most commonly used format.
If you're starting a new Python project today,
though, you should obviously use Python and Tomo.
And preferably in the way that is recently
standardized in PEP61 and related PEPs.
That said, we also support the custom fields
that poetry uses in Python.
A lot of Python packages use a setup.py file
to the care dependencies.
And supporting this is more complicated
because a setup topy can really contain
arbitrarily complex Python code.
And we don't want to execute this.
That said, for simple setup.py files,
we are currently able to parse out the declarative dependencies.
We also recently started supporting content projects
and also the Pixie Package Manager,
which tries to bridge the gap between
the user and the user and the user.
So, we can get the same way that you typically consume
your Python dependencies.
A PEP install will work, but of course,
a careless PEP install is often the first step towards adding
an undercare dependency in the first case.
So, you should rather think about whether you want to add
it to your project as a development dependency
or as a standalone tool that is generally available
in your user environment.
If you use a modern tool like UV or PEPX or Po3 or PDM
or anything similar, there is typically a command available.
That will do the right thing.
We also specifically support running faulted apps as a pre-commit
hook or as a GitHub action in your CI pipeline.
More about this soon.
I want to emphasize the configurability of faulted apps.
You can configure everything on the command line,
but also via the environment or the tool.faulted
app section in PEP.
And finally, we have also a command line option
to quickly create this tool.faulted apps configuration
based on the current command line options.
Let's look at some more output options.
We want to make faulted apps as transparent and helpful as possible
so that you can understand how and why something ends up
in the final report.
You can run faulted apps with list sources
to see which files faulted apps reads while looking
for your actual and declared dependencies.
This is often the first step when debugging faulted apps.
You're risking asking the question, is it finding the actual code
and the dependencies that I have declared?
Next, you can run it with list imports to see which
imports is finds in your code.
And you can run it with list apps to see which dependencies
it finds to have been declared.
Finally, if you want to write some script to interact with the analysis
that faulted apps does, we have a JSON flag,
and you can get machine readable output with all the glory details.
So let's move on to deploying faulted apps into your project workflow.
Running faulted apps on your project once in a while is obviously a good idea,
but there is extra value if you can automatically run it
whenever you make changes.
The pre-commit tool is one way to do this,
and I'll show you how to set up faulted apps to run on every committee
in your repository.
This is mostly useful in the beginning of your project,
while it's still relatively small and dependencies
come and go all the time.
If we are not already using the pre-commit tool,
we need to install that first.
The pre-commit homepage lists a few different ways.
Here I'll just do a simple paper and so on.
Next we need to tell pre-commit how to run faulted apps.
So we create this pre-commit config Yamufile,
and we have these lines to it.
These lines tell pre-commit where to find faulted apps,
which version to use, and which parts of faulted apps to run.
We want to find both undercred and unused dependencies
in the code that we're trying to commit, so we put that there.
And next, we activate the pre-commit hook
by running the pre-commit install.
Pre-commit is now armed and ready to run on every commit that we do.
So let's test it out.
So we add an import statement to our code,
and when we try to commit this,
pre-commit will fail and prevent those from committing
and undercred dependency.
You can see that we also get the faulted apps up with,
pointing to which dependency and where it was imported.
So let's fix this by declaring the pandas dependency properly.
So we added to our requirements to XT,
and then we retry the commit,
and now pre-commit succeeds and our commit is done.
The same interaction happens when we try to add an unused dependency.
So let's add a TensorFlow to the requirements to XT,
but without actually using it anywhere in our code.
And now when we try to commit,
pre-commit fails again and faulted apps tells us what's up.
Another way to make sure faulted apps run regularly
and quickly catches problems with your dependency is to run it
and check in your CI system or continuous integration system.
With CI checks, you typically have a little more patience
than in a pre-commit hook.
So this is usually the better option in larger projects
where analyzing the code can take a little longer.
Here I show a GitHub action setup for faulted apps.
If you use a different CI system,
the setup should typically follow similar steps,
although this syntax will probably be different.
I'm also using the faulted apps action helper project here,
which we have created to make it very easy to run faulted apps in your CI.
But whether you use that or not,
the end result should look pretty familiar to anyone
who has already set up GitHub actions for a Python project.
First, we set up a GitHub action by writing a file
like this into our repository.
We spend a couple of lines declaring what platforms
we're running on and checking out our Python project
and then comes the faulted apps part.
We refer to the faulted apps action helper with a pin version
and then we also pass an extra option to indicate
that we want to run faulted apps with the detailed option.
If you have configured faulted apps in your Python repository's
title file, that configuration will of course
also be picked up and taken into account by faulted apps.
Once this is committed to your project,
GitHub will automatically run faulted apps on new commits
and pull requests.
So let's see what this looks like when we create
the pull request that adds an undeclared dependency.
Here we have a pull request that adds our old friend pandas
as an undeclared dependency.
Once the GitHub action has run,
we get an indication on the pull request or review
that the faulted apps check is failing.
And if we go to the details of the failed check
and then through summary, we can see the output of
faulted apps pointing us to the problematic pandas import.
The Toy example we use so far has helped show the basic concepts.
But things are always more complicated in practice.
First we will look at a part of the dependency
analysis that we've glossed over so far
but that is really crucial in order to produce correct results.
And this is one of the cases where we believe that faulted apps
goes the extra mile, especially compared to doing dependency
analysis by hand.
So let's draw the rest of the demo.
And I'm going to talk a little bit about
machine import names and package names in Python.
It's more tricky than you would think.
So far we have imported NumPy in our Python code
and we've declared the NumPy package as our dependency.
NumPy happens to use the same name for the package and the import.
But this is not actually a strict requirement in Python.
The package name you declare as a dependency
does not always have to match the name that you need to import in your code.
The package cycle to learn is a popular example of this.
The package name that you people install or the
clear as a dependency is the cycle to learn.
But the name that you actually import in your code is called
Escaleran.
There are also cases where single package provides multiple
import names.
So the set of tools package for example has long provided both
the set up tools import name as well as the PKG resources import name.
In order to import either of those modules you need to have set up tools
installed.
There are even more complex cases as well which I won't go into here.
But it should already be clear enough that we need some kind of mapping
between import names and package names.
And without a good mapping, we won't be able to correctly
match import names to declare dependencies and will end up
reporting on the clear and unused dependencies incorrectly.
Fortunately, we can always build a good mapping
when we have access to your project's Python environment.
Now, we don't really know how or where you install your project dependencies.
If it could be a virtual M somewhere or you could depend on a
system-wide installation or something else.
However, let's assume that your project dependencies have been installed
somewhere.
And when you run your project, let's see here.
There we are.
Yeah.
And when you run your project, Python must be able to find those dependencies
otherwise your project wouldn't work at all.
So as long as faulted apps can find the same environment where the dependencies
are installed, then it can build a correct mapping.
So naturally, we put a lot of effort into finding the right Python
environment so that the analysis will be as correct as possible.
First, we look for environments that are associated directly with the project.
Without any configuration, we will search on the current directory for any
environments located inside your project directory to put the virtual M.
And as long as we can find those, everything is fine.
However, you can also pass one or more paths directly to faulted apps.
This will limit where we look for code dependency declarations and also for
Python environments.
This is often useful in monorepo settings where your repository contains a lot more
a lot more or less independent Python projects.
So you can then point to the Python project that you want to analyze.
You can also use the PIN option to directly point faulted apps at your project
environment.
In addition to looking inside the project, we will also always consider the Python
environment in which faulted apps happens to be installed.
Thus, if you install faulted apps into the same environment where your project
is installed, you will always get good results.
Together, these two approaches give good results as long as the dependencies are
installed.
But there might be situations where the dependencies are not installed.
So how do we deal with those?
First, for really advanced users, we have a way to provide a custom mapping.
Where you can tell faulted apps directly have to package name, how package name
and import names correspond.
But it's usually not something that you would want to configure.
But when it's given, it takes precedence.
Otherwise, we have a really cool option called install apps.
This sets up a temporary virtual end and install missing dependencies from
Python, just so that faulted apps can get the correct mapping.
And then this is obviously quite expensive, so it's not enabled by default.
Our final resort, when all of the things fail, is something that we call the
identity mapping.
And this is a simple assumption that package names and import names actually correspond.
So if we have no other solution, then that is sort of the final resort that we
go to.
And it works out correctly in many cases, but it's also sometimes incorrect.
And that's why we only use it as the final resort.
So let's quickly go over this again with an example like scikit-learn.
So remember, when we declare a dependency on scikit-learn in requirements 60,
what we actually import in Python is called escalern.
Faulted apps needs to find that these two names are related.
Otherwise, it would report escalern as on the cleared and scikit-learn as unused.
So here's how we figured this out today.
We take the dependency name, scikit-learn, and we pass it to the first stage in our
funnel, the custom mapping, which takes precedence over anything else.
But as I said, it's usually not provided.
If you're basically a power user and you want very tight control over this.
So next, we look at the Python environments associated with the project.
If we can find an environment where scikit-learn is installed,
then we'll see that escalern is indeed provided by this package,
and we have a match.
Otherwise, we move on.
And look at the environment where Faulted apps is installed,
same procedure here, if we find scikit-learn, we're done.
Otherwise, we're starting to run out of options.
So now, we're checking here.
Did we use the installed apps option in that case,
we can auto install from PIPI, and find the correspondence between scikit-learn and
escalern that way.
And then the very final resort, if installed apps is not used,
we need to use the identity mapping.
And in this case, that actually gives us a wrong answer.
Because through the identity mapping, we would then assume that if you install scikit-learn,
then it would provide an import called scikit-learn, which is incorrect in this option.
So that's why we never want to get to that point.
But as you can see, we tried very hard to find the correct mapping before it comes to that.
So yeah, I'll quickly go over some more complex use cases.
We've added support for.
We have an option to exclude parts of your project where Faulted apps is analysis.
It's used as getting north-style patterns, and you can even pass getting north-style files to that option.
Also, another popular or problem that often shows up is some dependencies that
are real dependencies of your code based, but that you never install and never meant to be installed.
The most popular ones are tools like the talks black, flake out, and even Faulted apps itself is part of this.
You can add it as a development dependency, but you never intend to actually import it.
We carry a default list that takes care of most of these, but you can also use the ignore unused option to specify these separately.
And then, also, sometimes there are imports in your code that are either conditional or alternatives.
Think of like you're trying to import something, and if that fails, you're either importing something else, or you're doing some other fold-back solution.
Those don't always correspond to something that you want to declare as a proper dependency.
So in those cases, we have an ignore-underclared option that you can use.
And then there are some things that we're still working on that are hard to solve.
When you do dynamic imports, you can do, like Python is a real dynamic language.
You can do stuff like this, import something that depends on some arbitrarily complex expression.
Those, we don't handle yet, and it's hard to figure out exactly how we can handle that.
We are actually executing your code.
We, let's see, and then also, Python allows for dependencies to be categorized and made optional in various different ways.
Depending on your context, some of these might be considered optional, and should be skipped by faulted apps.
We have plans to add support for telling faulted apps to ignore some or all such optional groups.
But for now, faulted apps will create all of these declared dependencies as something that should be imported in your code.
So to summarize, reproducibility matters, in science generally, but in Python specifically to.
The biggest hurdle to reproducibility in Python is missing or incorrect dependency declarations.
And faulted apps helps you fix this by highlighting undercare and unused dependencies in your project.
Faulted apps works with all these kinds of various Python projects and Python versions, as I've shown.
And it should work out of the box on most projects.
And you can automate it, that's one of the most important points here.
Finally, that's it for me really.
Thanks for listening and go check out faulted apps.
We're on GitHub and Pipei.
We're on the 3-Bug and there can also read about all the other things that we're so active.
Yeah.
If there is any time remaining, I'm happy to take questions.
Thank you very much for the nice talk.
Unfortunately, we don't have time for questions.
So please ask any questions that you might have in the Matrix chat or just approach your hand directly.
Thank you.