All right, welcome and one, and I hope you still have some energy left because I have about
60 slides for 30 minutes, and you're done, okay I'm done.
As I love the energy, thank you so much for still being here at the end of Phasm, I was told
to stand over here.
We just heard that sometimes we need to make the decision to fork, fix or forgo, and how
do we do this at scale, and I want to talk with you about that.
One of the thesis that I have, so this is the punchline, is that tracking open source
project health is proactive risk management.
When I look at the health of open source projects at the potential to continue developing
quality software and risk management to look for reasons why that might not be the case.
And I think project health is a place we need to look because what is happening inside
of an open source community today will determine what the software looks like down the road,
whether it's still maintained or not.
And so I'm gear link, I'm in this space, I found that the chaos project I've been
around for some time, and really want to help make open source across the healthy and sustainable.
Here is what I have in store for you across the next 60 slides, being perplexed by open
source in the software supply chain as well as we'll start.
When we look at open source across the board, we know that about 10% of everything that
gets published gets used, only about 10% of everything that that's published is actually
maintained.
So there's a lot of noise in open source, and there are bad people who have found this
to be very attractive attack vector with more and more malicious packages being published
and discovered.
This is costing the industry 60 billion as of today, approximately.
And any company that thinks, it's not me, about 45% of companies I expect to have been
hit by a software supply chain attack, according to Godner.
Well, what's the impact of that?
From the Sullivan example, we know it was about 11% of annual revenue to mitigate this software
supply chain attack.
So this is something that is bugging us.
So now we're in the Espanthew Room.
We can start to articulate what it is that we have to worry about.
And I want to use a metaphor of software, a just more like milk than wine.
Why is this relevant?
When we look at milk products or dairy products, we know exactly where that is being
used.
The industry has already established that we need to track the source.
And now with the CRA as the latest example of why we need to do this, we need to do our
due diligence and actually check what it is that we have in our dependencies.
So as an industry, there has been a move towards uncovering what are we actually using and
documenting this and with 70% to 95% of software containing open source, we really need
to focus on the open source part.
And we also need to look at the transitive dependencies because if we don't, then we have
things in our software that we don't know about and so we have unknown risk there.
So this is where the software bill of materials come in.
Because our inventory of open source software that we are relying on and other components.
There was a really brief overview of why we are here talking about S-bombs and why I want
to talk with you about project health.
Now that we have S-bombs across the industry and I know from talking to many companies
that there is a lot of work in this space.
We can do new things, we can do new kinds of analyses and give you another metaphor.
Think of driving a car, driving down the road and the fuel gauge comes on.
You know what to do, you pull over to the gas station, you pull up, it's good to go.
The engine light comes on.
You go to a mechanic, you get it repaired, no problem.
So long as you are using a modern car, once there are 20 years old or so you start going
into the junkyard for replacement parts and you have to look for a specialist.
And when we transfer this to software, we have a similar situation where we can look at what
we have today, scan for licenses, scan for vulnerable security, and we can fix that.
But when we don't pay attention to whether there is a community around, maintaining the software,
then we might run into that problem of having un-maintained unreliable software.
And so I am here to propose that we have these three pillars that we need to be able
to trust our software dependencies.
And with licenses, we have license scanners with security, we have software composition analysis,
we have vulnerability databases, for project health, we have the community, chaos community
to talk about how we should look at project health.
But I don't really see a lot of uptake on that yet across the industry.
I think that third pillar of project health is still overlooked in the open source strategy
by many.
And so this is where I want to think about how can we anticipate the risk by looking at project
health and how do we do this?
Remember the beginning when I said I had a thesis that looking at project risk, a project
health is a way to proactively manage the risk.
Well, now we are back to this.
How do we do this?
How do we operationalize this?
I want to propose that we look at whether a project is maintained with some metrics that
we can look at at scale.
Look at the community handle the demand or the work they might have work that they have.
Do they have the capacity to quickly respond to issues and pull requests and resolve
them?
And do they have the talent, the people in the community, whether it is through new people
coming in, people staying long time or that we have at least more than one person responsible
for maintaining it.
So there are several things that we can automate from all the metrics that we could look
at just as a starting point.
I'm not saying this is the only set of metrics that matter, but as a starting point
to start looking at project health.
And then we can start scaling this.
And this is where really S-bombs come in because when we look at project health, we've
done this community by community so far, but now that we have inventories of tens of thousands
of components that we care about, we want to do this at scale.
And what if we take our list and calculate the health for each component?
But if we assign each one a single score, take the seven metrics, normalize them, put them
together with whatever algorithm we want to use, what else can we do with that?
But once we have that, we can look across all of the components we use, we can filter
by the different packages that we are building, the different solutions that we are building,
and we can calculate the score, at least start to filter where do we need to pay attention.
And so I want to show you a prototype that we've built.
We've tried this out with the use case, and the data I'm going to show you is from
Kubernetes.
We've tried this also inside of a company already, but here's what it looks like.
We built the dashboard, so this is for the use case of a controller or someone who wants
to see, okay, I want to analyze.
There's of course another use case of embedding it into the development workflow and
CICD pipeline, but this is more of the background, I want to see the whole picture.
So once I filter down to the project that I really care about, I can see how many of the
dependencies fall in the buckets of low, medium, high and very high risk.
I could color these in the future so that we have a bright, red category, and this helps
us focus our attention on, okay, these are the dependencies that I want to spend some time
investigating.
And I still have the original data on what each of those metrics returned.
We'll see this in a moment when we look at a particular package or component.
When we drill down, we can see the overall score, but then we can also see why we got this
score.
And on the right, you see all the metrics, the raw data that we got from analyzing the
open source community, and then our assessment on whether that is low medium high.
So there are thresholds that we can parametrize, we can build our own thresholds here.
And so for this one, we can say, okay, medium lead time to pull requests and issues
is where the problem seems to be in this particular component.
So now we have the data to make the decision, we started with, do we fix fork or fork?
So if we want to help with the median lead time of pull requests and issues, well maybe
we go and we help with buck triaging and make sure the right people get attention on the
issues and pull requests and maybe that is the way we can help.
So we can start to have that conversation knowing more about the health of the community and
why we might consider this risky.
I also have an overview of the tools that we have used, these are the Grimola tool from the
Chaos community, which has arrived out of university research, more than 20 years of history
already, it's been used by many different foundations and companies, it's been used in
the Mozilla, Rebel Alliance Report, and it's the foundation for different project health
platforms that you might have encountered already.
The process really is where the key lies, that we have the espom on one side, but then
to understand the community, we go out and actually look at the activity in the community
and collect that.
That data needs to be enriched, we need to do the calculations, before we can analyze
the health and built the score.
So that's the process, one of the insights that Grimola has that we haven't used in this
model yet is looking at identities, because there's a lot of conversation around knowing
your contributors and knowing who is contributing to open source projects.
So we have a component called sorting hat built in to consolidate identities and also
track organizational affiliation and other things, we haven't used it in this model yet,
but it's an interesting avenue to explore further.
Also when we're thinking about data privacy, GDPR, how do we manage that so that we can
work with the data?
If you're interested, I have this chart about the architecture, the slides are all online
in the schedule, so if you want to go there, you can explore that further.
Let me talk about the roadmap for Grimola, we are working on scaling right now with the
use case that the having S-bombs and we're looking at tens of thousands of components
at a time, rather than one community at a time.
There's some advancement we need to do in the tools to scale, and that's what we're working
on right now.
Here's the way to get started, you can work with the open source project, or you can
come talk to me, I'm happy to see about commercial support for that as well, and I want
to leave you with culture action, because one of the challenges we have run into with
trying to analyze the components listed in S-bombs is we can't always find where the original
source came from.
So if anyone building S-bombs, if you could include the origin, that would be very helpful
to take the next step of looking at project health, because having only the list, looking
at the components and adding that project health really helps us understand better what
we are relying on.
And that's what I'm leaving you with, we've done almost 60 slides in 15 minutes, so thank
you for your attention.
The questions together?
Yes.
The question is whether we can reuse the metrics from the S-bombs, health, product,
project, in Prometheus, but we can use them in Prometheus.
So what's the S-bombs health project, I'm not familiar with that project?
So, okay, this data right here, yeah, yeah, the data is stored in a elastic database,
or an open search database, we switched away from elastic, or the license, the documents,
and went to with open search fork.
So yeah, the data is in there, I'm not familiar with how to get into Prometheus, but
I don't see why not.
Yeah, thank you.
Yes.
It's not a question, but let's do it.
Yeah.
10,000 sub-projects.
And now you realize these compliance guys, so I see the component that they use in Prometheus,
and I'm talking to the developers.
And now we have a large number of developers teams who could tell me by heart, which components
they are using, and they know the main case, so all of this could be fine.
And then they are the others, the JavaScript project, the 16 levels of dependencies, how
to do it there, because our people would tell me, we are not interested in this, and
I can tell them, you should know your components.
You have to know that because of cyber-resilience, but it still, you only need the combined
stuff.
So I guess most of the people sitting here in the room, they are very relevant, but what
you are telling, but how do we get this more to the developer?
Yeah, thank you.
So for the livestream, the scenario was as the compliance person with tens of thousands
of dependencies and developers, the developers are working in their day-to-day work.
They know their direct dependencies, they trust the source that they are using, but then
there are other situations like in Java, where we have JavaScript with lots of dependencies
that the developers don't care about, and like who are you as the compliance person to
make me want to care about these?
I don't.
How do we have that conversation to raise awareness to get more compliance that we are required
to have in the future, or we should be aiming toward?
So that is a very good pointer, and I see lots of hands going out.
I think you were first.
Yeah, you're in the green.
So how do you see this enabling better automation of open SSF best practices rather than
if being periodically, or maybe never again, completely non-quested?
So the question is, how this can help automate like the open SSF best practices rather
than being a periodic check?
The open SSF best practice batch, which is fantastic, and I'm the person behind the
German translation of those, so I know them very well.
I love that they are a way for projects to take us to back and think, are we doing all
the right things?
What I'm proposing here is not adding burden on to the maintainers, it's something that
companies can do at scale to look across their dependencies and start conversations.
The ideal situation that I wish we would arrive at in the future is that focusing on project
health leads to healthy conversations and relationships with our communities that we depend
on.
So it's a different focus.
The next hand I saw back there.
The question is, is there a lot of people that don't have to solve the voice in your
experience problem, and we can do two things like the surface of this information to
identify the errors, so whether we're going to do something, we show that this is bad.
I don't know if she is great, can you be it?
So it was a comment that we can do two things.
One is, raise the awareness during the development and built phase of the software development,
raise it to the developer, maybe through the CI-CD pipeline, and the second thing is to break
builds.
So that was the proposal.
So correctly, if I want to scan some projects and get this data, I have to set up the
instance, and then scan the projects and everything, so is there any effort to provide
any open data on scan and like a bunch of critical projects like, for example, as a tool
maintainer, it's a bit easier to integrate committee and metrics when some data is obviously,
for example, the scorecard provides sometimes like data on some critical projects, so it's
a bit easier to get the data through if you're going to run the scorecard to ourselves.
So is there any like plan to send any open data?
Yes, so the question was whether someone wants to use this approach has to stand up their
own tooling, or if there is already a service where you can just ask, hey, what is the
health score?
And whether there's movement in that direction.
And the answer is, yes, we are currently working on that to build out, I don't know if
it'll be an open data set, it is something that we're working on building out an API,
and I just had a conversation with Michael who had the talk right before here to see if
that is building out the bigger database with more stakeholders is something we can do.
Yeah, but it is something we're thinking about.
Thank you.
Point.
Yes.
We've got scorecards, we've got critical projects, 80% of the open source projects are small,
one-two people, outpicks.
There's a danger that what you're going to do is end up highlighting the bigger projects
with me, the health, and the real small innovation was from the hobbyists that actually
we rely on.
Yeah.
Get ignored and don't get the love and attention they want.
Right.
Well, I can see what you're doing, this is great to then surprise is how to be sure they don't
just focus on the big projects, all the volume well funded and well resourced and we don't
get ignore the little projects.
Because I see that as a single maintainer, I see people doing metrics and I get a
poor score because they don't need it.
Right.
So the question was, we already have metrics and one of the experiences that it tends
to draw attention to the big projects that already have a lot of attention or funding and
the small projects with just one maintainer are not getting as much attention.
And what I'm thinking is, for one, this approach where we can change the parameters, the goal
is to find a place where we can pay attention to because we only have limited attention,
we need to filter out what it is that we pay attention to.
And so looking in the long tail, trying to find where do we actually want to spend our
focus on?
And I think looking at the health, the smaller projects and the metrics that I've been
looking, or that I was proposing here, are highlighting the smaller projects more
so than the big projects, because they already have a lot of people, they already have
other things going on.
And so it's a matter of waiting, yes, and from our experiment, we have seen that different
departments prefer different waitings.
And so that is something I think as the built tools, we need to allow for that flexibility.
Thank you very much.
Thank you, Gary.