Hello everyone, welcome to our next speaker, I'm here, who will be talking about
out-of-think open-source.
All right, thank you, thank you so much.
Hello, good afternoon, everyone.
Thank you for lasting this long and being here in this afternoon talk.
I'm really excited to speak with you all today and also kind of live my former
life stream of being a college professor, being at a university, so this is a lot of fun.
So yeah, thank you so much for being here.
I'm a mere Montezary, I'm the Managing Director of OSTIF, that's the open-source technology
improvement fund, and I'm going to talk about security auditing as a practice and the success
stories that we've seen by applying the best practice of security audits, but in a open-source
centric way that is focused solely on, or it's primarily focused on empowering maintainers
and helping maintainers with the issues that they face and fulfilling their security needs.
So security audits are typically, no, are defined as code reviews where experts review code,
an analyzed code looking for vulnerabilities, testing code for logic bugs and other issues.
And according to a lot of research, it is a best practice, both in the software development
space, but really just a best practice in general, to have a independent expert come and review,
validate, and help address issues, because as we have seen, a lot of open-source maintainers
are not security experts themselves, and typically security is not always at the forefront
of development or as at the forefront of the focus of development.
So there's a lot of technical debt and help that especially open-source projects can use,
and by applying the best practice of a independent security audit, we're able to do that
best practice for projects.
So a typical audit process or a more traditional audit process involves kind of a top-down approach
where you have a boss or C-suite or some kind of force working with a audit team, which
lots of times can be big consulting firms, which tend to be more generalists and lack some
of the specific expertise that some projects would need, and working with an internal
software development team.
So essentially the auditee and the funders are part of the same organization, they kind
of have the same objectives, there's a little bit of the stick, so to speak, when it's
your boss, for your company, and this is typically how these processes are.
That being said in the open-source way of what an open-source audit can and should be is
having an open-source centric audit team, which typically works with under-resourced either
solo maintainers or small group of maintainers and then a funding body.
What's really important here is having a champion or somebody to manage this process from
start to finish because unlike the first example, maintainers can work for different companies,
they can have kind of different objectives, lots of times they are very much overworked
and have enough on their plate already, and working with open-source centric audit teams,
so teams that understand the needs that maintainers face is really, really important.
And again, having somebody to manage or champion as I like to say the process from start
to finish is really, really important because when we do that as part of our audit process,
we are solely responsible for carrying the audit from start to finish and being that advocate
for moving it along because as we've seen in other examples, when there is that missing
piece of someone to essentially own the process from start to finish, it can lead to less
than ideal outcomes in audits and this way, having someone, again, solely focused on
completing this work, has been very successful.
So similar to a director of a film, so director of a film isn't necessarily an expert
in costume design or lighting or special effects, but they kind of are in charge of the vision
of moving a film through to completion and kind of play that role as the facilitator
and the kind of, again, champion of the process to create the outcome that is either
the film or in this example, a successful security audit.
So to share a little bit about our experience here, we started about 10 years ago, just celebrated
our 10 year anniversary mid last year, but in the early years, it was very much proving
the idea and proving the concept, so we were able to work on some early stage projects
such as the Veracript Project and OpenVPN to kind of validate this idea that this best
practice of independent security audits can, in fact, apply to open source projects and
the kind of nuances that come in open source.
After a growth period, it was a period of slow but steady growth, we were able to get more
experience, more case studies and examples of applying a security audit to open source projects
and so much success over those years, but it was a great opportunity to really hone in on a
process and understand what the typical needs of maintainers are and how to fulfill those needs.
All while building a network of researchers, advocates, audit teams that are open source
centric, that could fulfill the needs of maintainers and in the last two years, we've seen
significant growth in auditing many more projects, projects like Git, Curl, PHP, Drupal,
to name a few and many more currently in flight that are expected for 2026 and a really interesting
actually, a really interesting thing that we've been doing in more recent years as well is
more holistic security improvements that apply to whole ecosystems and have far reaching impacts
beyond just for example helping one individual open source project with a security audit.
Those results will be coming out this year, so I don't have a lot to talk about on that quite yet,
but it's been a really nice journey over the last 10 years of starting with this idea that we can
apply this best practice to open source, learn a lot of examples of efforts that try to do similar
things to varying degrees of success and learn from those efforts and observed very closely
those efforts and take all of that into really hone in on a process to be a
neutral independent advocate for improving the security posture of projects.
As the title suggests, it's time to audit open source so why now, it's always been of course extremely
important, it's been a problem since the beginning of software, but especially now with
sophisticated and escalating bad actors who are out there to cause trouble,
increased geopolitical tensions, pretty much all I've heard this weekend is the concept of
sovereignty and EU sovereignty and very understandably so as well as the global digital
interdependence that software is developed in. So taking all that into account and the fact that we
have a method, a process and a dedicated organization that is solely focused on doing this
process and doing it well in a way that is effective, that is transparent, it makes now an ideal
time to really scale this work up and do it for multiples of projects than we are currently doing.
As I mentioned earlier, we've been doing this for 10 years now and have marked over 800
security vulnerabilities found and fixed because going back to working with open source
centric teams and interestingly enough, I learned this weekend about some of the first efforts
and doing this kind of work of doing security audits for open source projects and seeing what
those pain points were and how those outcomes were less than ideal for what they originally intended.
One of those kind of consistent factors was again kind of applying that commercial approach
to open source where as working with audit teams that are fixed oriented that will work with
maintainers on proofs of concepts and providing fixes made it so that this process was
much more effective for the project. So because of that, we have almost well over 90
percent fixed rate, all of the high medium and higher risk vulnerabilities we have closer,
much closer to 100 percent fixed rate. So we're not just out there to dump kind of bug reports on
projects, there's already enough groups out there doing that and thanks to AI, that's just
exploding and overwhelming maintainers even more than they already are. So by having this very
maintainer focused approach, by working directly with maintainers and communities on
holistic improvements to software, we've been able to be very effective at applying the
best practice to open source and that's also been marked with over 20,000 hours of independent
expert security review and close to 200 projects directly improved and thankfully we have been
scaling up so it is of course it's not something that can be kind of fully scaled like a lot of
other technical solutions but there is the talent out there and the process out there to really ramp up
this work and do instead of 200 total do close to 200 per year. So I'm feeling very optimistic about
the future especially given sometimes a little bit of struggle or a little bit of,
yeah I guess struggle is a good way to put it can really spur action and incentivize
people to really think about the security of the open source projects that they depend on,
that they use, that their organizations use and groups like us like many other organizations that
are working towards solving this problem have the means to do so. So I am actually wrapping up with
a material that I had I was not able to look at my notes while presenting which is why it was a bit
sparse so we can turn the rest of the session into Q&A for anyone that has questions or things
that I might not have been able to address during the talk. So yeah let's hear I see a lot of hands
up so this is great and definitely open to doing some discussion so oh thank you very much.
Thank you for a great presentation I really like the concept the motivation and it's very much
needed my question to you is as you talked about scaling this is there a role of AI in doing
security analysis and will that help being adding a middle layer on top of your layer you been on
the top and some degree of AI in here for doing security auditing just to ease out the process
make it more committed and making it more scalable. That's a very good question and I'm not
surprised it was the first one. I think the answer is that it depends I mean AI is still very new
in terms of its development and from what I have seen so far it looks like AI is currently doing
more harm than good in that it's a great concept but at the end of the day it's in the implementation
and execution so when the incentives are just to create AI that can just find bugs that kind of
reward function you know there's a mismatch there and that's why we're seeing very public instances
of projects just being absolutely overwhelmed with AI bug reports that from what I have seen
are have a very low rate of being actual issues so I think there is potential and I think
over time AI could be used to augment the audit process but the where AI falls short and where
what makes this process so meaningful and powerful is the context so having auditors who really
understand what they're reviewing because they'll be able to actually find issues find classes of bugs
and I think over time that could be augmented but I don't think we're there yet.
How does the prioritization work for you go about where's the funding for for selecting
a project which project to audit could you say something about that? Absolutely it's a great
question as well and it's a question that comes up a lot actually where sometimes we end up spending
more time talking and more time and resources talking about which projects to audit when we could
just be auditing you know we could agree at least on some projects and actually start doing the work
generally in our experience it has been very demand driven so the funding the bodies that
fund the work typically dictate which projects to review however as we've grown we've gotten a
little bit more flexibility with project selection so we will do undergo our own analysis
and come up with projects that are especially showing signs of benefiting from this work but
from our experience I mean we've audited projects that are you know your stereotypical kind of
single maintainer project you know we always think of the XKCD graphic you know the little block so
we've audited those and we've audited projects as big as git and Kubernetes and others and
really from our experience all of them have benefited from this work and I think it's just a
testament to the under-resourced nature of open source and the fact that there isn't enough
funding and enough diligence really going into open source projects I do hope that over time as we grow
we can get to a place where we have essentially our own discretionary fund to help a lot of those
edge case projects because so much focus is going on critical infrastructure and understandably so
but there's a lot of edge cases and those projects that you don't necessarily seem like they would
be important like no one I like who knew what XZ was before 2023 okay more than I thought but
but I think that's a find example where there's a lot of edge cases there's a lot of small projects
that just simply don't get enough attention and we have a way to improve those projects and find
those vulnerabilities that sometimes make the news so we should be going out and fixing them
next question hi you mentioned the importance of heavy and champion and can you be more specific for
who selects or volunteers to be the champion yeah how does it work absolutely yes I'll go back
to the slide so what I mean by championing is let's say I'm a maintainer of a very popular open source
project and I am under a foundation and I want to get a security audit done or get some security
help just the process of finding professionals or finding audit teams let alone reaching out to
them let alone working with them on getting proposals or getting calls for proposals and analyzing
all of those proposals vetting different vetting the vendors and let alone all of the contracting
and administration that goes into the process it's it's staggering it's it's very high and it's
why this process is not more widespread because it's it's hard essentially it's a lot of what we
call cat herding you know even with projects where we have the funding in place we we have the
maintainers on board the amount of administration and project management to make this this work
so valuable is a very high and typically without having that dedicated resource the process just
does not go as well as intended so there are some organizations that have the capacity they have the
people who can undergo these processes reach out to vendors build relationships with them have a
contracting process have a invoicing and accounting process and they have those in place and
they have been able to procure security audits successfully but those examples are very rare compared
to the norm which is we don't even know where to start we don't know who to talk to let alone
something we've heard a lot today a lot of open source projects are just individuals they don't
have a bank account or a company associated with them or really any way to procure this kind of work
and that's exactly why we created the the the nonprofit that we did because it does all of that
essentially primarily on behalf of the maintainers with the mission of improving the software
and doing it in a transparent way so again it's it's extremely important to have somebody
dedicated to this because when it's somebody's fourth fifth job to to go and do this it's just
not going to happen effectively so I think we have time for one one or two more I see a hand up oh sorry
yes so you mentioned hiring experts to do the review for the security audits but in fact they're
also using tools that you can see tools fuzzing etc so in your experience with all these projects
are the tools that's you would recommend that's a broad category of tools that's
I've been profitable in these hundreds for that yeah that's an excellent question
I don't necessarily have opinions on which tools are better than others I can say that I know
a lot of our audits have involved a significant amount of static analysis tooling I know some
graph has been used in a number of our audits by the audit teams for for fuzzing as well that has
grown a lot in in impact over the last few years and so we work with teams that are very well
versed in OSS fuz and and our when applicable can build fuzzing sweets and actually build the tooling
for the projects to help them not only be more secure in that moment but from every moment on
words after the audit so tooling is a huge part of our audit process so I'm glad you brought that up
and I know for sure we've worked with a lot of different ones but we don't we're not really in the
business of proprietary tooling so anything we would use or recommend would be open source
so yeah so I'm with Dr for a little bit quite interested in this on this type of auditing
we have code we have things that look more like configuration we have processes around the code
and then we've got infrastructure so just wonder sort of what is the scope of what you can address
with these or this is it just the code or is it that kind of extraneous stuff around as well
that's the next one question as well I think in a perfect world there would be enough funding to
really dig deep into essentially all aspects of the project even down to things like
mitigating social engineering risks or any kind of indicators of maintainer burnout or
really any risks to the project overall in reality funding is very sparse so typically audits have to
be scopes down to the most high-risk areas the most critical functionality and be audited to that
but again in a perfect world and as we gain more influence on scoping and and funding these audits
then we can dig much deeper but for now it's it's more focused on again most high-risk
areas areas and is entirely risk based so I see my times up so thank you all again and have a great
pause