Okay, I'm to be on this phone this time.
Good morning.
How's it doing today?
Great, ready?
Excited.
Okay, so my name is Michael Windsor.
I am the co-founder of a project called Alpha Omega.
We started in 2022 with the goal of improving open source security.
We have over the past several years donated around $20 million
towards improving security and open source.
It's a huge problem, obviously.
And, you know, we started with the organization.
Even if we have a lot of money, we can't do all the things.
And so our focus has always been around creating change,
catalyzing and improvements that then take off and do things for themselves.
And, pretty excited about the opportunity.
When we think about how to do things,
the Alpha set of represents our ability to leverage to change.
Things where we can work in with a specific individual and entire ecosystem.
So, when my favorite example is in the Python ecosystem,
it was nobody's job there to worry about security.
Or it was everybody's job.
Same thing.
And we were able to fund a security engineering residence,
named Seth Larson, who has continuously just improved the culture of security,
the tooling of security, the standards of security across the entire Python ecosystem.
And, in fact, his work has spread to other language ecosystems as well.
The scale problem is actually really hard.
And, this is one of the things we're talking about today,
which is we want to get solutions can just be applied to the hundreds of thousands of projects
that are not directly tied to either a language ecosystem or a package repository
or some sort of point of leverage.
How do we solve for the rest, everything?
And, it's not easy.
And, I want to talk about just the trends that we're facing.
So, as a maintainer of an open-source project,
you know, you might build out an espawn of your stuff
and you have a bunch of stuff upstream.
You're consuming a large amount of open-source packages.
And, in fact, I would say until sort of the exeutils event,
people just assumed that everything I get from upstream is great.
It came down in the back of the unicorn with rainbows and everything
and I'll have to worry about that problem, right?
Which is ironically, just as bad as all the corporations we complained about
consuming our open-source without giving a shit about how it get made.
And, these numbers are not good, right?
They only get worse.
They get bigger and bigger and bigger.
We get log jams as well where people can't upgrade
so they can't patch the vulnerability and now you have a problem as well.
And, so the force that this represents creates a lot of work, right?
As a maintainer, you now have to look at every CVE in your upstream
or as a consumer's same problem.
And, you have to decide, do I upgrade or not?
Is this a risk to me or not?
Right?
And, the default answer is upgrade all the things.
Which, if you look at how it multiplies across your work,
it's one thing.
But now if you imagine a deep hierarchy of projects,
there's a geometric explosion of work.
Shared across lots of people, great, okay?
It's shared, but it's still a ton of work and it will not all get done.
And so that cascade of work here is really what creates the risk
that we're worried about.
So, is that landed my slides?
Think so?
No, okay.
Thank you.
So, how can we reduce the geometric pain here?
Right?
Well, you can't change the geometry of it, but you can change the constants
and some of the multipliers here.
Right?
And so, if the span of vulnerabilities, you could look at it and say,
well, not all of these vulnerabilities are actually going to affect my code.
And everybody can do that downstream.
Then we take this very big win, find out,
and we can hopefully narrow the volume down and make it smaller,
and make it manageable at scale.
Right? And that's really what we're here to talk about today.
So, with that, I'll tee off to Kevin.
Thanks, very much.
All right.
Hi, my name is Gavskarnas.
I'm a maintainer of Apache Lachfordian commons,
and there's security member of the ASF.
So, this is the ASB of whom you certainly know what an ASB of them is.
How many of you have already had vetses?
Okay.
It's a big number, but for those that never heard about this,
Alex is a machine-readable format to express the fact
that the vulnerabilities that is contained in your application
is actually exportable.
The abbreviation stands for vulnerability expertise exchange.
Open SSF produces a very nice document,
a couple of, we can go, I think,
where they analyze where these are used.
So, we know that big companies like Microsoft Redhead open SSF.
Cisco service now.
These only use vetses.
Some of those companies are part of vetses as a requirement
for all their contractors.
And yeah, why vetses are important in these days?
Well, this was already in Antony Slides.
The CRL says, we found no exploitable vulnerabilities
except for us as stress, exploitable.
Yeah.
And yeah, I knew about vetses.
I learned about vetses in 2023 from the source,
from Steve Springett.
And he warns me, yeah, it's very, very expensive.
Of course, I had to verify that.
And so that's what happens if you do it.
By hand, every day in engineer gets there are around 100 CVs
that are published.
You need to check if the CVE actually applies to your application.
Nowadays, we have S-bomb, so that's easy.
Fortunately, then you have to understand what's written
in the CVE.
Some CVEs are very worried, some are very, very bad.
Then you have to dive in all the calls,
the trace from your application to the library.
If this is the library, which is seven dependency levels deep,
you have to look at a lot of code, assess if this is
an exploitable and repeat for each version,
because each version changes the code path.
OK.
And so, well, this is very expensive.
Last year, I met Munawar guests in the room,
so thank you, Alexis, for organizing the dev room.
And we decided to experiment if we can use dev access
for all open-source projects.
So we know that Vexes for an open-source project
will certainly improve the confidence that commercial vendors
or downstream projects will have project, but unfortunately,
at the time, we're very, very expensive.
OK.
So let's talk about the cost that Peter and Michael were mentioned.
So if you are a large organization, that's why,
so you encountered about 700,000 update decisions every year.
And if each one of them, when you're generating the Vexes documents,
it takes on average 10 hours, that takes about 7 million hours,
which is about 3,365% years.
Even if you get cheap labor, that's about 400 million dollars per year.
That's why organizations find that it is very expensive.
It's actually even more than that.
This is, like, assuming the lower side of things,
we have heard from experts in this area,
that for those organizations generating one Vexes documents
requires about half a million dollars.
So this is a very expensive problem right now.
And if organizations find it expensive,
it's just impossible for maintainers.
So think of Apache Solar, with whom we're working,
and they have about 460 dependencies.
About 300 CVs are found on average on these dependencies every year.
Again, with 10 hours per Vexes, that's about 3,000 hours,
about 1.5% years of dedicated work in order to generate that.
That's also impossible for maintainers.
So that's why, even though this is needed,
the adoption right now is not there,
because we are still trying to figure out how to do it at scale.
If we, however, is able to do that, there's benefits of that.
There was a study from the same group,
when they did post-analysis of their SCA results,
and they found out that 97% of the bugs that they find,
even they do these SCA, they are actually false positives,
as in those are not reachable.
So if Vexes documents or somehow you find out,
whether a vulnerability is reaching you or not,
and you can eliminate the 97% of the toil,
then all of those compounding numbers suddenly become tenable.
So that's the idea where we come in.
So how do we generate Vexes? Here's a simple idea.
So let's say you have an upstream package A,
which is used by a package B, which is used by a package C,
which is used by a package D,
and there is a vulnerability that was reported in package A.
The question is, is that vulnerability reachable from package D or not?
So how do we define that reachability?
The way we do that is we look into,
we create call graphs of A, B, C, and D,
then try to find a path from D,
which is the downstream package of interest,
all the way to the place where the vulnerable code resides in A.
So the first problem that we are solving here
is to find out where which part of A is vulnerable.
So that's what we call root cause analysis.
So we have a component which is a root,
we have written an AI agent that is,
that we call the root cause analysis component of this puzzle.
And what that does is given a particular CVE and a particular package,
it identifies a set of methods or functions
that when executed that vulnerability would be manifested.
Then we also have to calculate the call graphs for each one of those,
each one of those intermediary chains,
and find those call graphs.
And that's going to give you a layer of the terrain,
how are the methods one calling each other.
And once that is there, once those two components are there,
then you can write a simple stitcher that's just going to hop from one method
or another and eventually you are trying to find out
whether you are reaching to that particular vulnerability or not.
Now remember, this is,
we are using the term reachability here.
There is a more stronger term exploitability,
which is even though you are calling something it may or may not be external.
That's an even harder problem.
We are not even going there.
We are just solving or trying to solve the reachability problem.
So as I explained the three components,
we have released open source with the work that was funded by Alfa Omega.
These three components that find the root cause,
generate the call graphs,
and then also do that stitching in order to,
like, create these vex documents.
And the link over there that tells the,
that shows the repo org,
that's in GitHub that stores these things,
these components.
Right now this is supporting only Java ecosystem.
So the call graph is the one where it is language dependent.
The root cause analysis,
it looks at different kinds of,
it's an AI agent,
so it looks at different kind of CVs,
and attempts to find the root cause in that particular language.
But the call graph service is limited to Java.
So that's why this whole thing that we are demonstrating
is specific for Java language or Java packages.
So how does this help?
So let me tell you an anecdotal story that happened last year.
So this was April 1, 2015,
when there was a vulnerability that was found in Apache Parket Avru.
This was CV-2025, 3065.
And around April 15,
Peter, who has access to this internal conversations,
inside Apache ecosystem,
got this door attention like,
okay, there's a conversation that's going on
that should we fix that vulnerability or not.
So let's try to see if we can,
if we can fix that.
If that is reachable or not.
So the chain was very simple there.
It was just like Apache Hadoub is calling Parket.
However, the Parket Avru is shaded.
So if you are looking for binaries,
it's not able to find that.
However, we are working at source code level.
So we are able to find that dependency chain.
We started working on April 16.
And at that time, the technology was not fully developed.
So it was partially managed with two to some time.
But by April 18, we generated the vex documents.
The result was that it was not reachable.
So we explained, like, created the vex document,
created an explanation of why we are finding it,
that we shared it with that team.
So we got the response that they were going to do the update
anyway for other reasons for compatibility reasons,
but they appreciate the fact.
And this is like,
eventually, like, whether this can be done in a CICD pipeline
or some other way by those maintainers.
So there's actually, if the tooling is available,
there's active interest in people trying to adopt that.
So that's where we, and there's also another side part of the story
that there were two other additional vulnerabilities
that they were exploring at that particular point.
Those two were, however,
exploitable, both of them were reachable.
And they did the fix.
But one after 12 months, the other one after 18 months.
So with the absence of vex like help,
it's hard for people to prioritize the vulnerabilities,
like, which one to fix, which one to prioritize,
because there's so many of them coming at you,
and which one you do have to fix.
So we created those vex documents,
both of them were exportable and they were eventually fixed.
So that's a specific case study that happened,
that also motivated us to create these vex documents
and basically see how we can integrate that with the CICD pipeline
of the appetizer of the foundation problems.
So I just want to recap a little bit here.
We have a lot of information flowing through the S-bomb
driven CVE pipeline, the crates toil for everybody downstream.
From individual maintainers along the way,
to the end-use your applications,
whether it's airflow or solar or whatever,
the toil is spectacular.
When I already talked about how this is for Java today,
this is actually based on work that was done
for project called caps lock,
that does call graph analysis,
initially was done for go,
and now there are versions of this for Java and the rest as well.
It is a strong belief of mine that call graphs
needs to become normally available,
because they provide all kinds of interesting opportunities.
This is just one example of things that we can do
when you have call graphs available to do static analysis
and richer analysis on your code to understand
where things are happening, what the risks are,
even to generate more effective unit tests
to validate your high-ramps law contracts.
So this is a beginning of an experiment across
a whole bunch of things about reducing toil for maintainers
at every stage of a secure software development pipeline.
So I think we're sort of approaching time here
and we don't have some time for questions as well.
I am particularly glad to thank Kiotter
and Munowar for their work on this project
to get us to this stage here.
The most important thing you can do is engage.
If this is interesting to you,
when you want to be able to do this kind of things to your world,
show up,
this links start to asking for help,
see if you can use it,
tell us what we're doing wrong, et cetera.
This is a classic early stage open source project
where all of your PRs are very welcome.
So again, thank you both for the work you're doing.
Thank you for the time today.
We'll take some questions now.
We have plenty of time because they've been warning me
about time left and we like questions more than we like talking.
So, thank you.
Thank you.
We're going to repeat the questions,
but you're close, so I'll bring the microphone to you.
So I would guess a valid ingredient
of basically ingredient of all your,
what you suggest would be that the vulnerability report
comes with a call graph or describing under which condition
it could be used.
I mean,
I mean, this should be like the CICD pipeline that we create.
Oh, we're good.
So CICD pipeline that we created,
this runs with as a GitHub actions,
and this finds all of these info.
You do have to provide some stuff, but basically right now,
there's this metadata GitHub repository.
That's where we are storing the results that we're calculating for future.
Like other people can use it.
Once we calculate a call graph,
there's no reason for somebody else to calculate that call graph again.
So right now we're doing the easiest way,
which is storing this in GitHub,
but we are open to, I mean,
that's actually one of the areas that we mentioned in our future work,
that we should really figure out how to make it reusable
and for others.
Yes, so we can also consider an integrated into the project.
So we just saw that because
Monkeys' dependencies there are many projects that I work from,
also it's easy to convince those projects to publish it in the GitHub repo.
Follow me to the back.
Yes.
If you have a call graph,
if you have originally,
can you update the challenges to eliminate symbols,
functions, libraries that are unused,
such that you don't actually have those things present,
if they're never called.
Okay, I repeat the question.
If you have the call graph,
and you have the reachability,
can you update the libraries to remove the symbols?
That are not used.
Okay.
So this is a problem I've been thinking about for a while.
Sorry, I'm off camera.
There you go, I was hiding.
And there's a lot of ways to think about this problem, right?
So this particular project is about helping you understand the reachability
and exploitability ultimately,
or aiding your evaluation of that,
in a set of projects we are not modifying your dependencies to suit your needs.
But it is obviously very interesting to think about what you can do.
I understand this here.
What you can do to effectively fork and maintain a subset of your dependencies,
so you're reducing your attack surface in a variety of ways,
based on your usage of it.
Similarly, you can also generate unit tests that validate your contract,
or shim layers, that sanitize your calls,
and to give you exactly the kind of controller that layer.
So this is why I'm excited about making call graphs more normal,
because this kind of operation used to be a lot of work to think about,
or a heavy maintenance burden now becomes a normal thing you could imagine.
Spongebob thing it says, go look at this dependency,
evaluate the calling patterns,
generate unit tests for me that validate my contract with this.
I call it Hiram's tests,
and it worked really well.
It was great.
So we'll take the next question now.
On the way back.
So right now, the question is,
if we have multiple languages,
can we use this,
like basically use it to go through call graphs across different language boundaries, right?
Yes.
The answer is, right now we don't do that,
but these are created in a modular way,
as in each of the call graphs are in separate files,
and so if you're doing stitching,
once you identify the functions,
it's just a matter of just stitching one across another.
So theoretically it can be done.
Back in the academic year,
like 10 years ago, I did,
like my students did the original work
on doing multiple languages analysis,
static analysis of stuff.
It excites me,
it's just not done yet,
but theoretically it can be done.
Thank you.
Yes.
I just want to say,
we just finished getting this work in the first time
when you're already asking for cross-platform course language support.
Okay.
So I have a statement to you.
PR's welcome, sir.
Yes.
I'll be back to you.
Yes.
I should get to a good class for the resources
or a class that you can show that you've had to do then
to be part of the voice system.
Could you have a question?
Can you have a question?
If you just look for Google GoCapSlock,
you'll find the original project that was built.
Generally, it's a call graph,
but what's also really doing is doing analysis
of the underlying capabilities that you're
call graph enables.
System calls down to network environment,
file system, whatever access is standard libraries
providing.
So you can understand what capabilities your dependency graph
is imposing on your process,
and essentially enabling.
But that was a go-specific thing.
Alpha-mega-funded ports or implementations of caps lock
and call graph analysis to both rust and Java.
Those are still ongoing efforts,
but that's sort of the state of affairs right now.
The question is, why cannot the new caps lock
be stopped?
The question is, why can't we use caps lock?
So caps lock is basically finding,
like creating these call graphs to find out
whether capabilities reach you or not.
We're talking about whether vulnerabilities reach you or not.
It's essentially the same problem.
In our case, two, the call graphs that we're generated,
they're generated in caps lock format.
So they're usable.
And that's why like this can also go across multiple boundaries.
Once we create these across other,
there's a standard format that was originally created by Google.
That is now can be usable.
And there's a working group if you want to join that too.
Okay, here.
So the question was, I'm an open source maintainer.
Sometimes, I think, our ability is my own code.
So the question was, I'm an open source maintainer.
And I occasionally, sometimes,
fix vulnerabilities in my own code.
I mean, that seems nice.
And so you're asking, why do I care about this?
I think that, as a maintainer,
fixing vulnerabilities seems like a good idea.
That, as a maintainer who cares about how people downstream
are having to deal with the changes you make.
So when you make a patch to a vulnerability,
you're saying, I fix this problem, right?
Great.
And if everybody could pick up your update immediately
all through the cascading effects of that transitive graph,
you know, you fix all the vulnerabilities.
Everybody just picks up updates, no problem.
But in practice, the cost for everybody to pick up,
you're updated version, you know, it's just to have it.
People will stick, they'll use to bend apart to pick up
the latest version and rate up until the build breaks,
and then they fit in it and don't change it until somebody forces them.
And so your act of doing vulnerability,
reachability analysis on your own vulnerabilities,
creates a statement that tells people and helps people
understand whether or not they need to take up your fix right now,
or to that gentleman up there's point of view.
We don't use that part of the code or we sanitize our inputs,
so we're okay.
We're not going to pick up his patch this week.
We'll wait until our patch choose day next week and do something differently.
Do I just do that?
Yeah.
Well, it's not, well, the device is something
that is produced not by the project that has the vulnerability
by downstream projects.
So I started being interested in CVEs.
When Kafka wanted to integrate log4j,
but they said, oh wait, log4j depends on many parsers,
and now parsers generate 70 CVEs per minute,
which will make Kafka much,
generate a lot of work to Kafka.
So my answer to that is, no,
if we generate a Vx file, you know,
project, and we say that all those parsers are only used
on trusted code, so all the Vxs will be not affected,
then there is still less work upstream.
Down through.
All right.
We are out of time.
We'll continue to take questions outside in the fresh air,
where we can all breathe fresh air.
Thank you very much for your time today.
Thank you.