Yeah, I think it will be great.
Yes, thumbs up.
Okay.
Let's go again.
Yeah.
Are you hearing at the end of the room?
Yeah.
Okay.
Great.
Okay.
Next talk.
Next talk will be, give the blame for your dependencies.
Please welcome Andrew and there's a bit with his talk.
Well, before hand, can I get a share of hands for anyone who went to the package manager
Devroom yesterday?
Woo!
Thank you very much for coming.
That was great fun.
Yes.
So, get packages.
This is a fairly new project, but it is based on about 10 years of work that I've done into
mining dependencies from get repositories and package managers.
It is the combination of a lot of different ideas or kind of dragged into one project that
I have just been hacking on over Christmas because I had a crazy Christmas that didn't involve
much leaving the house.
So, I was going to live demo this and then I realized how much of a dangerous game that
is and the morning woke up and was like, I'm going to rewrite my whole talk and do this
as static things instead so that it's a little bit more reliable.
So, I'm basically going to drive you through how this project works.
The idea of the project is, oh, I was going to hit that button.
Sorry.
We'll just imagine that there's a minute missing.
The idea is this project basically wants to make get more aware of how your
algorithms, how your dependencies are actually integrated into get.
Generally, get is completely oblivious of how you're using your dependencies.
Maybe if you're using sub modules, but also if you're using sub modules, you probably
hate yourself because you chose to use sub modules.
So, this is instead primarily looking at manifest and lock files that are in produce to
configure your packages and also record the full transfer dependencies.
Tree that was produced once you successfully installed your packages.
Can we make get more aware of the kind of the semantics of what's going on there?
And then what kind of comes out of that once you can do that?
So, it's really easy to install with home brew or it's written in go.
So, you can just do go install straight from the repository.
And I tried to make it act as much as like get would look.
So, it installs it just kind of plugs itself into the get-possilin.
And it has all these commands.
And we're going to go through some of them.
We're not going to go through every single one because some of them are really uninteresting.
But also it supports loads and loads of different software ecosystems and different package managers.
Many different kinds of file formats.
The JavaScript one I just cut it off because otherwise it would be going over into the next room.
Because of how many different kinds of file formats they have and how many different lock files.
But there are many of them.
And yes, they are generally supported.
These are primarily language package managers that are supported because those are the ones that actually record
kind of what your transitive dependency tree was for your application.
And thinking about this as you'll get repository for your application rather than your system dependencies.
Although it does have little bits and pieces in there which we will see.
Start to show up.
Straight up the back.
The way that this works is you have to run an in it command for most of this to really work very effectively.
But when you run that it actually becomes very quick.
If you were to not do this it would be every command will be slow and you would just be like I hate this project.
I don't want to use it anymore.
So it will basically spider the whole history of your repository pulling out every time that you changed dependency file through the history of the project and make a small sequel like database of all of those dependency changes.
Which then we basically just do sequel queries across that database to be able to kind of inspect and see what's going on.
And you can see when you run that in it you see the size of the database you get out.
This is for a project of mine Rails app which has a good number of dependencies and has been around for kind of 13 years or something.
Let's add a lot of dependency up there.
So Penderbot just loves pinging me every single day on this thing.
And then it actually can start to show you some interesting stats.
How many dependencies do you currently have?
Which ecosystems are you using?
How many times have you changed dependencies?
4,700 times I have updated dependencies on this repository.
That's mental.
Also who changed dependencies the most?
It turns out dependable has done twice as much activity on this project as I have.
Which is kind of like depressing but also the old version of dependentbot is up there as well.
And Mark come to comes in a fourth place there.
This is very high level stats which isn't necessarily particularly useful.
But is cool to start kind of go digging around is also quite fun if you just find some random project and you're like,
Oh, let me see what actually is like going on under the hood of this project because it might be a thin layer under some interesting dependencies.
You can also then say like okay well list me all the dependencies and this then looks across all the different files.
You could then break that down say actually only list me the gem dependencies.
I'm not bothered about the JavaScript stuff because there's way too much JavaScript stuff for me to ever really like comprehend because it's insane.
You can also then say like oh can we break this out and visualize this as a tree.
There are also some I'm not going to show them here but ways of saying can you give this as Jason or CSV.
If you want to pipe it into other kinds of tools as well breaking it down by where these dependencies are used which files they're defined in different ways of kind of slicing and dicing that stuff.
You can also do searches and this is literally a like query across the database to be like oh what kind of rails related gems so I have and when were they added and when were they last updated so you're kind of getting a very rich picture of what's going on.
But also because it's get we can just use regular things like oh can you show me this like what happened in this commit and this commit changed a number of dependencies.
But I can just start poking around in the history of this thing.
I can start to kind of get a real picture of rather than just what is the massive text if Ruby gems not so bad but if you have got a 10,000 line Jason file from MPM.
You're just not going to bother and get how literally hides those files when you should try to review it in a poor request is like don't look at this and it's like yes that's good advice but I might actually want to look at this at some point so this gives you a rich.
A semantic understanding of what's happened rather than just.
Which you know they're all good projects basically give you that same kind of functionality.
You can also say can you tell me what happened over the life of the project for this particular command.
I'm going to try and go quick because I got like 25 slides but we're touching on the load of different things.
You can also just say okay well what's the log of like all dependency activity and this works for every different branch you can basically just kind of.
Hop around exactly as you would do with get and I tried to basically go what commands are available and get and can we do the same thing.
But for just packages I mean eventually it would be really nice if we just lose the word packages there and be able to say like oh like here's a semantic interesting thing that happened as I was going through the log rather than just get a big.
You can also say why is this package here which shows you the commit that was added and also the commit message.
To go and kind of go like okay give me more context around this.
Where is this package which then finds each different manifest file that this thing is described it.
And then also browse now this is cool because this actually shows out to the package manager under the hood to say what path has this package been installed in locally.
So you can open it with your editor and just start diving straight into the code even if that package hasn't been blended into your git repo.
Which is where we start to get into some really cool things.
So you can do a blame you can say like who put this in turns out Andrew is responsible for almost all of the dependencies being added to Andrew's project which is you know there's a bus factor problem there that we won't go into that's a different talk.
You can also say like okay well how many of these projects are outdated and you think how does Andrew know that and like how is this project know that or this is where we get into.
We've moved beyond the intrinsic data in your gem file or in your manifest in your log files and we start to enrich this with.
Extrinsic data so we're going out to the registries.
And but we only do this kind of on demand but then we cash that back into the SQLite database so this is really quick.
We start to be able to build up a history of what is going on but also what future things are available what extra stuff can we collect.
The really easy one is to say okay well how many of these packages are.
Have new versions available that I could update to.
What are the licenses of these packages and this kind of output gets us get overwhelming again especially a few more and more packages but there are different ways of slicing and dicing this.
Grouping by license as well or being able to pass a flag to say like what about only copi left just show me all the packages and the dependencies that I have that are copi left license or have no license as far as we can tell.
Then vulnerabilities now I'm quite happy with this because when I ran this I was like I don't have any vulnerabilities like I'm in a good place.
But you can also then go backwards in time to say like when were these vulnerabilities.
Here in this project which is basically saying like there was a vulnerability between this version of this version what time.
My project depending on those versions and how long did it take me to resolve and upgrade to a version that is no longer affected by that particular CVE.
Again depending on the driver could project that you run this on that could give you an overwhelming amount of things and maybe a heart attack.
So you know you can also say can you make me an S bomb because some people really really like S bombs.
It's basically exactly the same data as I had everywhere else.
It's just in a particular format and there's different flies because of course there's two different standards for S bomb.
So you can say I want an SPDX S bomb or I want a S bomb or I want an XML S bomb for some reason.
You can also buy set. So this is cool.
You can then say like I know that a dependency is screwed me at some point but I don't know exactly when.
So can we do a semantic thing to just step through and try and find when a dependency changed.
Here's a bad commit that I know this is bad and here's a commit that is good.
Step through to a binary thing until we find where it is where it was introduced and what version did it change.
Five minutes okay cool.
Now we get into the the kind of weird and wild world of how far down the rabbit hole can and you go.
So intrinsic data, extrinsic data.
What do we actually just start driving the package managers themselves?
So I then made a set of commands that will basically map over all the different package managers.
Come on lines to say when you say add a dependency it means this in mpm.
It means this in root gms.
It means this in UV.
For 35 I think different package managers to be able to say can you add a thing?
Can you update a thing?
Can you update everything?
Or can you remove a thing?
What this can actually enable you to do is say goodbye dependable.
I don't care about you anymore.
I'm just going to do get a package update on a cron every so often.
And all of that stuff just happens by querying the registry.
Getting the updated command and then pumping it back in and running that against each individual dependency.
Which then gets committed back in.
And then it's all, you know, happy.
And we totally papered over the individual differences between each package manager claim.
Which means you don't need to then worry about it.
But also you can have one CI command that just says,
I'll do this for whatever.
I actually don't care about what the underlying pieces are.
And we start to be able to treat individual package managers in the same way.
And build tools that think about things at higher level rather than just kind of getting stuck here.
You can also say, can you give me a schema for the database so that I could write my own secret request.
Because I really like doing that.
Very simple one is just to be like which of these dependencies have been changed the most.
But you know, the word is your oyster with that schema you can then write secret queries with it.
Or you could just give it to Claude and say, Claude, go ham on this and tell me, you know, what is the weirdest and wonderful things you can find in this database.
The cool thing about this is it is made up of a number of small components.
It's not one big honking mess of things.
So there's libraries for passing the manifest files.
There's libraries for driving package managers.
There's library for fetching the registry metadata from upstream.
There's libraries for passing the licensing data and also for dealing with the version range resolution.
Because each one of those package managers has a slightly different syntax for how you describe version ranges.
Which becomes very useful when you're trying to work out.
Does a CVE affect me because I'm using this particular kind of package.
Maybe in the future we could also even plug in vex statements to be able to say, yes, we have a CVE but actually this vex statement doesn't mean that we don't actually affect this because we have the source code of the application there as well as the dependencies.
And we use package URLs everywhere to basically normalize and treat it as an individual dependency.
So, very cool.
I encourage you to go check it out.
We have a website and the Git repo.
It's all MIT licensed.
But there's one cool thing I wanted to add in at the end, which I started hacking on very recently and you think,
but this is cool.
It's all in my command line.
It runs locally, but wouldn't it be great if it was integrated into forjo and code book?
So, this is the same libraries, but it's plugged into my experimental forjo.
The same rich diff, the semantic understanding of what happened in this commit.
It's really small.
I apologize.
Basically, green means something was added and orange means something was changed.
You can see what version through went to and from.
And this would be the same UI from whichever package manager you're using.
That's just a diff driver.
There is a diff driver built into Git packages, which can plug straight into forjo and just work.
But then you can go farther than that because why not?
Mad science.
And I've got one minute.
We can build a whole bloody dependency graph feature into forjo.
So, this means we can then have all the features that GitHub dependency graph has,
but it's entirely open source.
The effect is data from the registries.
It's agnostic to how that happens because it's all using those low level libraries and those pieces.
The work that I've done was in building those tools as components.
As soon as you have that, then you're like, okay, well actually putting it all together.
It's actually pretty easy.
And we can then have a fully open source entirely like self hosted set of dependency information and kind of like intelligence tools.
There can be used by anyone without needing to be in a centralized platform or to have kind of a big security company selling you those services.
So, that's what I got.
And I got five seconds left.
So, I haven't got time for questions.
Let's thank you very much.
Thank you.