You
Green
Green is good
So rusty
Thank you
That's a completely unnecessary but we'll take it
So yeah, I was just saying we're sorry about the just face stuff. We're a little bit rusty. We haven't spoken since about 2018 when we did the package management
Devroom which I think you've been smashed into doing next year
Back in the day, everybody's coming back to foster
I just dumped you in that you can punch me later
So thank you all for coming
I appreciate it. It's a really long event and this is the last session of a very long event
And that I'm standing between you and most likely a beer
But yeah, thank you for listening
And thank you all so those of you who don't speak English as your first language
I have a tendency to talk too quickly
This is the last session
It's likely that we'll overrun and that they'll be Q&A at the end for which we have plenty of time
So if I'm talking too quickly, just put your hand up and I'll start talking a little less quickly
But yeah, thank you for coming and seeing us
And the other
Oh god
Really
Okay, so
I did write a bit in this but it's a bit silly
It says I kind of feel like Elton John closing down glass library
Which is basically just a way of introducing who we are and where we're from
So Andrew
Lifts about ten minutes away from glass library the world's biggest festival
And I live ten minutes away from nowhere
Because I'm in the middle of nowhere in structure
We've been working on open source for about ten years
All started with heart bleed
And Meredith Whitaker and Ben Laurie who pulled me into some conversations about where the next heart bleed comes
And I met Andrew, I think I was talking a thing and you were building a system called
The time was it called libraries? I think it was, wasn't it?
Yeah
That was about being able to point people at good open source projects to kind of contribute to
Years later, you know, we've worked on very many projects
Most recently working on ecosystems, which was a little bit about
But it's not a sales picture or anything
We kind of have to in order to get the talk done
But yeah, we've been working for ten years together
And we got bugs
So on the left is Leo and on the right is Mabel
I've got one dog, you don't, you have, is that Luna or Felix?
That's Felix, that's blossom, the wookie
That's Luna, the latest to do
So I just have the one dog Andrew's got four
So our talk is labelled
I'm not afraid of a bit of clickbait, I'm very sorry
But I will defend it if necessary
But the honest fact is that we're treating this genuinely as a kind of research question
So after saying, you know, and working in this space for ten years
Are we in a closer saying that we are supporting our digital infrastructure?
And we're going to treat this as kind of two hypotheses
The first part is going to look at our opinion is
Which is that usage is the best determinant of software criticality
Which we hope to convince you of, we might not
But genuinely, you know, we're here to try to convince ourselves
We've already kind of convinced ourselves, but that's that
And then part two, if you assume that that is true
We believe that most funding that we see
And there's a very specific word in there
Is being misdirected
It is not being, it's not used to fund critical infrastructure
It's going elsewhere
So what methods do we have at the moment?
This is looking at the first part
So is usage the best determinant of critical kind of open source project
What methods do we currently use for identifying critical infrastructure
Both within an organisation and within the border landscape
And we've come up with basically two methods that we see in the world
Which we're characterising thusly
The first is the defining rod method in which one person
An organisation company or maybe an individual
Basically decides, decides what's critical
Decides what's, you know, important in the world
Just kind of looks using the cross sticks and says that that's that
All we do the same thing that we have a group of people in the organisation
Which we call the wage board method
In which you all place a hand on the wage board
And one of us is pushing a little bit more than the other
And we end up, yes, towards your own projects
I'm not going to say the name that you want to say
But yeah, I believe that we can do better
And I believe the evidence based methods
Because, you know, ultimately we've come from a mathematical background in robotics and computer security
The evidence based methods are going to be the way in which we can do this
I'm going to make a point about that
But what evidence based methods can we access currently and what do we see in the world?
Well, we see things like popcorn, the popularity context
Contest for devion packages that basically just takes a log
Of who is downloading and using packages in the devion ecosystem
We've got something similar in home brew
Where they show you all the downloads that they've got
An Andrew once scared a room full of rubyists
Is that what they call them?
In 2018 by doing the same in the NMPM package or was it a ruby package?
I put good analytics in my JavaScript packages
So that I could get real time install information
And then just presented the real time analytics page
During your conference
Which scared the life out of everyone in there
Because they didn't realize how close that you could watch all of the activity
Because person's school scripts of
JavaScript packages that you do whatever you like
And they were all in the one room
They were all in the room
Which was good fun
So we've got a bit of a close up of the results there
Thank you very much
This is incredibly impractical
There are other things that we can do together data
So we have something that is effectively doing that
But on a larger scale scarf
Which is basically a gateway in front of your own kind of download your else
So that you gain access to the information
Taking ownership a little bit more
Or we can do the Linux Foundation's approach
Which is aggregating data from faster, sneak, etc
That is representing proprietary usage
In an aggregate manner
And they can continue to do this
The Linux Foundation in 2022
Published their mobilization strategy that said they want to create
A data lake that companies can basically dump data into
And that was one of their kind of main
Screens of their report that was published just after their second
I think it was meeting in the White House
So you know who will like those
My argument is basically regardless of how much information you
Think you can collect on proprietary usage of open source software
You are never going to be able to say that that's representative of the whole
You are never going to borrow that ocean
I don't believe that is the right approach to take
I think we can basically save ourselves some time
By instead trying to gather all of the data that we can see
The available data
Be able to demonstrate a correlation to what is
Representative of overall usage
And then say that that correlation indicates
A strong enough indication for us to say that what we can see in the real world
Leads us to believe what you know is holds for all of you
Sage of open source which at this point sure that Andrew can say
A lot better than me with some data
Yes, now we agreed before not to make me the quant of open source
But what we've ended up with is now I'm going to do the quant section of
Of the talk
Yes
With your plan all along, isn't it?
Okay, so we to be able to use all the data that we have about open source
To try and get a good picture of how people use it within both within open source
Projects and within closed source projects to be able to try and say
Here is a global representation of usage of open source
Especially from kind of like a relative perspective so you can say these projects are used more than these projects
Which will help us define which projects are critical
What I have been working on is collecting a normalizing data from every different software ecosystem possible
And then mining the dependency data of every open source project that is available
So far I have a database table in Postgres with 20 billion rows in it
And it's a little bit of a headache but the value that you can get out of that
And that we will explain is that we can use data from this open source usage to imply data about proprietary usage as well
By looking at correlations with other measures that include proprietary usage of open source software
And hopefully that will give us a number of
Formulas and kind of statistics that give us confidence enough that we can draw those parallels
Without having to collect all open source usage within proprietary software across the whole world
And instead just use the data that is available and then kind of work just from that
We actually are able to reproduce this data and share it and other people can do the same thing
And where we don't have the data for where we don't have the data for downloads and so on
We can infer it using the information that we do have because we've shown strong correlation in other ecosystems
So yes we believe hypothesis number one we can work out what this usage figure is to be able to give us a picture of what are the most critical pieces of software
Let's have a look at some examples of kind of the approach that we've taken to suggest the open source usage is a good parallel for
Close source usage or at least good enough that we can work with that to then move forward
This is a graph of download of this one is
Packages pH P packages some of the most used pH P packages as you see you can imagine this long tail of dots that ends up in zero zero is pretty massive
But
It's simplified just so that I could actually render this in a browser because five million dots is too many dots for chart tears
Yeah, well
So the other access that we have here is the number of open source repositories that depend on each one of these pH P packages
Looking at the correlation between downloads which includes everyone who is using pH P packages in their proprietary software and
Their open source software against just how many times they're declared as a dependency in an open source project
You can see this actually has quite a nice trend and we can look and see other ecosystems
This is the rust ecosystem and actually has an even stronger correlation
It's still a little bit hard to see here, but you can definitely see the trend locally. We have maps that can actually quantify that in a very useful way
Pearson correlation coefficient of comparing the number of
It's been a long for stem the number of
Repose open source repose that depend on these packages versus the number of downloads that those packages have received over all time
If the correlation is above 0.8 it's very very strong. We can be confident that one will infer the other if one is large the other will be large
Going down the scare, right? Let's have a look at some examples of what we measured in open source usage for a number of very large software ecosystem
So rust and packages are incredibly correlated. Basically if you know how many people in open source depend upon a pH P package or a rust package you can guess this is going to be a similar level
Relatively like it's not going to give you an absolute number, but relatively it's going to be able to go I can infer this. This is really helpful because as you notice there isn't a go
Row here you can't get go download figures they're just not available, but we can measure dependent
Repo usage in open source of go packages if the correlation is strong we can imply that this we can kind of infer what the downloads that's would be if we had
And people much smarter than us with a statistics background could do a lot more I'm not really a con I'm a Ruby developer
We have all of this data that is available on public for people to build more interesting statistical models than I am able to do for a conference talk
To help us do really interesting insights into different ecosystems and across all of open source potentially
To be able to like really build out strong models that imply what we believe the correlation looks very strong
To give us a picture of what proprietary usage of open source is even if we don't have that data
And we actually had a funding proposal that we put out it turned down unfortunately, I was too sure you want to throw that shade
But if anyone isn't interested in funding that research then do come and say hello
You can see
So having established that we believe
You can tell we don't do this often but we do work together a lot
Right, okay, so having proven we believe that you know, there's a very strong correlation between observable usage once saying and kind of all usage of open source software
The next thing is that if you agree with that can we demonstrate that funding for critical infrastructure is being misdirected
At which point I have to obviously say no, don't I because the data that we do have being predominantly from GitHub sponsors and open collective is not specifically directed for projects on the basis of their criticality
So straight off the bat I have to kind of say no, but we can run with it and we can see where we get to and we can see the funding that we do have available that we can see
We can take a look at that see where it's directed see how well it matches up and then we can talk a little bit about you know lack of data effectively
There are plenty of other sources I'm going to talk about them a little bit in a second but I think now basically I have to sub you back in sorry
Okay, so let's have a look at some more graphs and I kind of want to give an example of two different data sets and then see what happens when we smash them together
And it's a little bit of a car crash but it should be fun
So this is a graph of all of the accounts that are on GitHub sponsors by the number of people and other accounts that have sponsored them
We don't have data for how much they gave
GitHub we'd love that data that would be great
But for now we can use this as a proxy
It's a very nice graph you might have seen other graphs like this the
The kind of long tail of anything ends up looking very much like this where there's one account at the end
There's like more sponsors than absolutely everyone else trading off to these accounts all have like one or two sponsors ever
Again, we don't know exactly how much money ZIG tools could have received like absolute huge amounts of sponsorship
But we have to go with what we've got
We know that what was it 68 million last year was given in sponsors
Yeah, so
I think it was
Yeah, I have these Abby's not going to answer that question right now legal told her to stay quiet
We also have very nice data from open collective there's maybe
2,500 collectives in open source collective and again very similar graph
I like this because you know we got two different data sources and the graph comes out looking kind of similar that's encouraging the area under this
This line is approximately 50 million dollars
Which is cool, but also like you can't really see the line
I've actually cut this graph off because it goes over into like the next building. It's very very long tail
What's interesting here is then like we have these funding stats for how much projects have received in
total amount of dollars and let's use open collective because we have dollars not just number of sponsors
Let's look at all of the rust packages that are all open collective and how much they have raised versus how much uses they have
I wouldn't pay too much attention to the access here because the numbers in dollars are not 40 million
Dollars that some of these rust packages have raised but I had to squish them to get them to show up otherwise all the funding looks like nothing compared to the number of downloads
So it's relative right the
We have some interesting things the some packages in rust that are highly used and you know are on open collective are actually receiving a good amount of money
But there's also this kind of like the most downloaded doesn't necessarily have the most dollars collect there
If we step through some other package managers and other ecosystems you start to see everything goes a little bit wobbly and we start to lose our trends a little bit
The Python one has a numpy on here, but course numpy is actually under num focus foundation and doesn't have very good data here at all
But num focus is has as a foundation has a lot of money to be able to support that project
But we haven't mixed that in because it's really hard to discern exactly how much funding from foundations goes into individual projects
JavaScript is actually really popular on open collective. It's like one of the first mentees to adopt that kind of open funding
It's absolutely all over the place
You notice a real correlation with some of these bars as well right where it's
These are all Babel's projects and they're broken up into small modules
How are you supposed to say like okay, well the Babel receives lots of money through open collective
Which one goes to which package how can you then kind of a line that is actually quite difficult to do and how to how to
Square that is really painful and then if you go to the Ruby ecosystem
You're like oh no, what's happened here?
We have almost no correlation between this funding data and the amount of usage of these projects. This is this is disastrous
I have no confidence in this graph whatsoever, but it's actually based on real world data, right?
We just don't have good data to be able to have ten minutes left
So the answer is inconclusive
We don't have enough data to be able to have any confidence in these graphs at all
We're like well, that doesn't make any sense, right?
But how do you know we don't know is kind of the problem
Thank you, it is tiny, unlike me
So as we said at the start of this section look you know the data that we're looking at is not targeted at critical projects
We accept that but even if you were to say that it was the results are basically, you know, budget and not great
So you know, going back to our original research question
After a decade of saying that we need to support our digital infrastructure, we only close to saying that we do
That's basically as my presentation, but unfortunately we've got a few more minutes and you have to listen to me
So what do we need? How are we going to address this?
Martin did such a good job on this.
I know, such good
So there are a few things we need and you know, I'm not saying I come up with all the best ideas
But what I am going to do is just amplify good ideas that other people have had at this point
Over a year ago just about I think it was a year and two weeks ago
Frank Nagle and a few others wrote this report on the value of open source software
Got some interesting kind of figures thrown around about the demand side
And supply side value of open source. I suggest that you read it. It's free
But I would like to share them out because Frank said exactly what I'm going to say
Which is we need better data for financial support both direct and indirect
So that's direct financial support and indirect financial support
Which might be we're employing someone on 50% of their time at whatever rate that they're employing on
To work on that particular project so that we can map what resources are available to whichever project
Now there are some people who are trying to do this and I think Emmy did not manage to make their talk this morning
Because of train but shout out to IOS and invest in open infrastructure
They have been trying to map 400 million dollars worth of funding both state and philanthropic institutional level
To a bunch of open infrastructure projects which a lot of are open source projects
I'm not saying the same thing please don't shoot me Caitlin
But yes, I would just like to give them a shout out and shout out to Emmy who
Yeah, I wish I could have heard your talk
And before you're alert on that
It's really hard and it's very very manual to do right now
Yeah, yeah, they've manually done that and there are very many issues including how to map actual money that is going to the project
This is being used in administration the argument about whether that would actually count a support for a project anyway
It is super super complex and basically what we need is to agree on a way to describe these resources for this project
That can be led from the project side or from the funder side
There are initiatives that have been spoken about
I think IOI may be released a report earlier this week or maybe it will be coming next week
That says hey the 360 data giving data standard was proposed a while back
This is the method that we think we should use for describing how funding is being used and moving to these projects
I actually worked on a open contracting data standard in 2012 when I worked in civic tech with the World Bank
That was deployed in 2014 to describe I think it was most of New York state spending
It's a similar kind of thing and I'm not proposing another standard. I know the SKCD comic no worries
I'm just saying that there are ways in which we can do this and basically we need better information
If we had that information what could we do with it?
Do I take this bit or drink this bit?
Yeah, this is no maths in this bit
So let's imagine a world where my graphs weren't terrible
But also we had enough data that you could be like okay, they're really weird
But at least we can rely on them because like it was representative of actually where the money was flowing
And where others have kinds of support were flowing as well
Some measure of that all kind of fluff together as well
That would potentially be able to give you a picture of where critical projects are under-resourced or over-resourced
And then when you come to make decisions about where should funding go to a project or where should resources go
You'd be able to make a data driven and let's call decision rather than using the OuG Board or using the defining rods
To kind of just wave yourself around and land on whichever project happens to have a really easy way to donate to them
You'd actually be able to go this is a critical project to my ecosystem that doesn't have enough support
And I can confidently give them support until they get to the point where they are no longer kind of under-produced that term of it's a make-up end-up end-up
I had this concept of under-production which basically is like these projects are really heavily used
But they're absolutely crushed under the weight of the pressure of open source
We can recognize that and we can give them more support so that they can stand up
The project to make it much more stronger and sustainable which then has an occupant effect of improving everything in its ecosystem
And kind of being able to scale it up to support for the whole world of open source because the whole world runs on open source now
Thank you, that was very smooth
Yeah, so in addition to that I just want to say you know, we start with the research question
Like can we say that we're supporting and sustaining all of open source?
I'm not saying that finances the be all and end all of open source
I'm saying that it's a constituent component we can probably represent some of the other implicit implied and you know kind of hidden
Kind of support for projects
But ultimately what I want to do is be able to prove for ourselves that the thing that we've been working on in our case for 10 years is actually working
That would be lovely if we could do that. Thank you very much
It's not having to go anyone it's not you know, throwing any shade anything but having spent the time that we've spent in this space
I would really really like to be able to know that I had some sort of impact. Thank you
There's plenty more to come we're going to continue working on this. I've managed somehow to intertwine a couple of organisations that I work at and director of
So that we can do that
There's very many opportunities for us to work together with many of you out there
All of the data that we've just spoken about is freely available as is all of the data on ecosystems
This is not a sales pitch. It is just an open call to say that we are more than happy to work with you on any of these subjects because we genuinely care about the space
So I think that is about it and now we've got a bit of time for questions if anyone has
That's remember to request you
Yes, I have a question and I hope this fits on the bracket of more to come as opposed to reach forward
If we accept the premise that criticality is this kind of a measure or sorry usage is a better measure of criticality
Can you extend that with sort of metrics or indicators that are on the project level
And by that I mean sort of indicators about sort of community or project health
So we have opinions I will repeat the question so if we give if we take it as a given maybe the usage is a determinant of criticality
How might we include other scores and be able to maybe complement that data so I will say from the first thing to get out of the way is that some measures appear to actively punish projects
Looking at them for not having maybe a correctly formatted license file or yes, I know that's another issue but let's get us ignore that for a second
Or you know not not having great documentation so on to which point I think that is probably wrong
If you accept that your project is one of the most used in a particular ecosystem and likely used in industry as a result of that which we've shown
Why would you want to demote some sort of criticality school because they don't have certain documentation
It's still going to have an impact if it blows up but yes, I think you know we're aware that the work of chaos and the work of open SSF
And so on in drilling down into how you might direct your efforts based on measures that you might see in how well the project is managing the demands that are placed upon it
Right so things like you know how well is
You