All right, I'm Vlad, and I'm going to be talking about binary dependencies.
Just doing it.
All right.
So we want to be able to identify the dependency graphs of all packages generally ideally, but specifically
Keystone packages.
What are Keystone packages?
Nadia and her book in her report has this great definition where it comes from sort of this
Barrelage definition.
And the Keystone package we want to say is a package that has a widespread effect on its ecosystem
or something that is disproportionately useful or dependent upon by other things or whatever.
So especially the most important packages we really want to know there's dependency graphs.
There's two reasons for this that I'm going to talk about.
The first reason is open source financial sustainability.
So obviously open source maintenance don't get paid enough or at all,
burnout is a big issue when you have burnout.
Packages don't get maintained, you know the deal.
And open source supply chain security.
So if we don't know any particular packages dependency graph, we can't know what security issues
it might be vulnerable to.
So the first thing, open source is not financially sustainable.
Companies charge money for their products.
They're products incorporate open source software.
The money goes to the companies that doesn't flow to the open source product.
The open source packages.
And so maintainers can sustain them.
They're packages.
This is a problem.
Technically, this is according to the rules.
Yes, open source licenses allow this, but also this can still be bad even if the rules say that it's fine.
Both things can be true.
I'm not going to go into a lot of detail about that.
The heat is doing the great talk tomorrow.
The community track about burnout and open source.
It's really powerful points about that.
And then you can also reach out what occurs.
Block both the open source sustainability crisis.
So here's an example from my work.
On the open source pledge.
This is an initiative where we try to get companies to pay the maintainers who's worked
They depend on.
We ask companies to pay $2,000 per full-time equivalent developer to the
employee per year to any open source maintainers really.
But particularly to the maintainers of the software.
They depend on the money goes directly to maintenance.
We don't handle any funds.
We had our one-year anniversary last year.
And so far, our members have raised $6.1 million for maintainers.
We're very happy about it.
But.
Thank you.
But where should the money go?
So obviously companies don't understand 100% of their dependency tree.
And so don't know necessarily what the best way is to allocate all of that money
that they're paying.
And like I said, we struggle to identify, especially the most important packages that
keep our global infrastructure running.
So that's one problem.
Another problem is open source supply chain security.
So let's say you have a thing and it depends on fully.
It depends on some other live and some other thing.
And there's a security problem in that last one.
You might want to know because that means that there might be a security problem in your project.
But if you don't know about that package, you can't do anything about it.
So that's bad and we want to solve that.
Yeah, so we want to be able to identify packages dependency graphs.
Now you'd think look in the manifest, right?
Because it says in package.json what things things depend on.
The problem is some things are not in the manifest.
The dreaded phantom dependencies, which are sort of described in this really cool post.
Yeah, so if the dependency is not in your manifest, you're not going to know you depend on it.
And then you can't fund the people you depend on if you wanted to do that.
And also you can't spot security issues from the stuff that you depend on.
So that's a problem.
Now I'm going to talk about these different kinds of phantom dependencies.
I'm going to talk about one kind, which is binary dependency.
So this is when you have something and then it calls into this dynamic library,
which is usually written in C.
And this kind of dependency is usually not recorded.
So for example, you might have some Python code and your Python code depends on numpy.
But numpy depends on open blasts.
But it doesn't say that anywhere.
So we don't know that.
So that's bad.
It's not in Pyproject.tomals.
So you can't support the developers of open blasts or any of the other sort of binary dependencies that you rely on.
And you can't track security issues in those packages.
So that's not good.
Now how binary dependencies work?
I wrote a post.
It's on Vladau website.
You can read it.
I'm not going to get into it.
But I will get into this specific thing.
So how you usually call into C code in a lot of different ecosystem.
So this is true for Python and Ruby and JavaScript.
The details are a bit different.
But basically, you want to be able to have some kind of code that you write that calls into the C library.
But you also want to be able to manage type conversions in between the C types and you know,
your Python data structures in a way that is, you know, maximally flexible.
And so what a lot of ecosystems do is you have this thing called an extension module,
which is a bunch of C codes.
You write a little bit.
You write a little bit of C code.
And then that C code includes some headers as like Python.htr or whatever.
And those headers tell it about the types in Python.
And so that means that in your extension module,
you can convert things, you know, back and forth whenever is best for you.
And you know, it's a performance decision.
But anyway, the point is your Python code and your extension module are going to communicate in whatever way is defined by your ecosystem.
But then your extension module itself gets compiled to dynamic library.
This is the important sort of point, right?
And your dynamic library is going to get linked dynamically to the C thing you're calling.
And so that means that your compiled extension module,
if we can get a hold of that, is going to say in it what libraries it needs, right?
It's cool because then we can figure out what binary dependencies you depend on.
So, Python wheels, right, include vendor binary.
So, you know, if you depend on open blast is going to include the dynamic library,
compiled version of open blast in your wheel.
So that's good.
And we can detect that.
So let's investigate.
I looked at the 15,000 most downloaded Python packages.
I could only download 13,074 of them for various reasons.
And of those wheels, 1531 wheels contain dynamic libraries.
So that's already interesting.
And I found a total of 12,137 SO files.
Now, those SO files, some of them are bundled binary dependencies.
So stuff like open blasts.
But some of them are just those extension modules that we wrote, you know,
that we just looked at.
So, you know, it's either the extension modules or the phantom dependencies.
So we want to find the phantom dependencies.
Okay, how do we find it?
In your extension module, that's on Linux, at least it's an L file.
Your L file has a section, the dynamic section.
And in there, it has a bunch of information.
But some of that information has a tag.
And the tag is detected.
So it describes a library that your extension module needs.
And in the name, it's going to tell you the name of that dynamic library.
And it's that name that the runtime linker uses to find that thing on the computer.
And so in theory, the name is sufficient for us to find it.
Right?
So we can look inside the dynamic.
We can look inside those dynamic libraries and find those names.
And so I decided to figure out what the most common dynamic libraries are in our 13,000 odd wheels.
Some of them are obvious.
I think, you know, lib m whatever.
So that's fine.
But some of them, I don't, I mean, maybe you all know all of these things.
In which case, that's super cool.
I don't.
I mean, I know like a few of them, but also wasn't expecting them to be this common.
Other ones, I just don't know what they are at all.
So anyway, I don't sort of have a concrete breakdown.
But it's interesting because there's a bunch of stuff that's really dependent upon that we don't know.
I don't know what it is.
And so I wanted to come up with a number of how common binary dependencies are in the Python ecosystem.
Generally, it's going to be a very inaccurate number.
But let's give it a try.
So we have 13,000 odd packages.
We have 1,531 wheels that contain the SO files.
We have 1,300 free other wheels that depend on those first wheels, right?
So that gives us 2,834 wheels.
Now, when we generalize this, we're going to keep in mind that this is obviously a small sample,
even though it is the most popular packages.
And also, this is only one way of doing binary dependencies, right?
There's other ways.
But, you know, taking that into account if we were to average that out.
We can say that around 20% of Python packages have phantom binary dependencies.
Which means that for those packages, which is a lot of packages, we don't have the full picture of the dependencies,
especially since frequently.
You know, you're not going to have a binary dependency for something that's like a left-pad type situation, I hope.
So it's going to be a thing that you really rely on, right?
So we want to know about those things.
And this affects pretty much everyone.
It affects users of the open source software.
And it also harms maintainers because the maintainers can't get supported.
Yeah, so I'd like to continue working on this.
This was just for a Python.
We can do this in more detail for Python.
We can do it for all of these other ecosystems.
We could look at the symbols inside the dynamic libraries.
In case we have some trouble identifying what library a file belongs to.
We can look at the symbols and build a database of, you know, which symbols are provided by what package.
I would like to publish some kind of tool that makes it easy to explore these binary dependencies for any given package.
And integrating into services like ecosystems would be great.
I'd hope to get some funding to work on this if you know of anyone that might be willing to fund this.
Let me know.
And that's it.
Thank you very much.
There's more details on my website.
All right, let's do some questions.
I'll fund you.
Amazing.
Easy.
Thank you very much.
It's not a question, but I welcome it.
I'm kind of a question.
Sorry.
The answer is yes to your question.
Let's do a different question.
We have a question over there.
Do you think the future culture should just debate these kind of binary dependence?
I don't.
Yes.
So do I think that package manager or sort of package managers or package managers?
Yeah.
Registry should not allow these kinds of binary dependencies.
No.
I don't think so.
I think if anything, we need to get better handling them because there's, you know, a lot of people
use them for very good reason.
If anything, we don't have the infrastructure right now to do that in a reliable way.
Yeah.
Just one word.
There's a button on this thing.
We'll close it.
Yeah.
Two.
Four.
Girls.
And we have the offer right there.
Raise your hand.
Hello.
The truck.
Bineries.
That's our vendor.
There's another thing.
Not my son, please.
Raise your hand.
So main things of package.
Test truck.
S-S-T-R-E.
Which instruments.
G-C-C and L-D-A-B-E-L.
To inject all of the dependencies in an L-F-section.
So there are people who believe that you said there's two.
So there's pebb.
Haven't you haven't happened then?
You know that the dependencies because they were bound up at the up time.
So.
We isn't as bound.
For the time we spread.
Yes.
So just to repeat this.
Yeah.
You said there's pebb.
Seven.
Twenty-five.
Okay.
So that's I will definitely look into that.
Hello.
And then the other thing was called.
S-T-R-E.
Okay.
Cool.
So I'll look at those things.
Thank you very much.
You can go to getup.com slash sunny.
Yeah.
Yes.
Amazing.
Thank you.
Any.
Raise your hand.
I may sound.
You can go to this guy there.
Amazing.
We have one question over there.
If I still have time for one question.
Yes.
I'll get a curse.
It's not so much a question.
It occurs to me that if you said this is called phrase or run.
You will discover every day.
Yeah.
I've built a blue.
Exactly.
For that.
He's a trace.
It's painful.
Because.
Yeah.
Interesting.
And doesn't know.
Because you don't find exactly this.
Okay.
Then you don't know.
All right.
Thank you very much.
Good job.
Thank you.
Good job.