All right, so we have very little time, I'm told we have two hours and we have to compress it down into 15 minutes, so we'll do it we can.
My name is Michael Winser, I am the co-founder of our project named Alpha Omega, and this is Pierre Pochorelli, the secret engineer was the Fabius de Foundation.
We have one mic between the two of us, so we will do that.
Every time we talk about software supply, security, we have to put this slide up here.
There's always new answers to understanding this, because what most people understand is that they're actually at the top of this thing,
and they have no relationship to the things at the bottom, and so they can't fix them if their lives depend upon them.
So open source, supply chain security, security is hard, and we'll talk about some of the ways it's hard.
That depth of the pentagram, it's not just deep, it's entangled, it's hard to move things around, it's hard to cause change.
There are a lot of humans, humans are hard, they're complicated, they have motivations, they have priorities.
And over the past several years, we've seen an increased sophistication of attacks that are taking advantage of supply chain attacks, but why?
Because it's the easiest path in now, which should be scary.
So Alpha Omega is a project that was created with the mission of catalyzing, sustainable security improvements to open source across multiple ecosystems.
We've been doing this now for about three years.
When we think about the problem, Alpha represents points of leverage, where we can go in and make changes that will have lasting impact across entire ecosystems.
But there are also hundreds of thousands, if not millions of projects that are not directly reachable or even influential, and still represent dependencies in your supply chain.
And remember this, every piece of supply chain, every dependence you have, has total access to your build environment and total access to your runtime environment.
So it doesn't matter how small or critical or how big or not important it is, it has access, and if it gets attacked, you get attacked.
So with the past three years, Alpha Omega's raised a ton of money, we've spent a lot of money, we've made engagements with teams where we're doing audits, we hire people to have security engineering roles.
Our work breaks down into four categories.
Probably the most significant effort we do is making it someone's job to worry about security.
When it's everybody's job to worry about security, it's a tragedy the comments doesn't get done.
When you make it someone's job, everybody follows that person, there's a plan, there's a decision it gets done, and we've seen this time of time again.
So we frequently help staff security engineers and residents at organizations where they can have that leverage impact that we care about.
Category B is about package managers because they are the app stores of software development.
They are, again, critical points of leverage and policy.
Developers make trust decisions every day when they install a package from some package manager.
They assume that everything about that package that they believe to be true is true, even when it might not be.
Category C is actually where we start most of our engagements through audits.
And it's again, amazing how if you start with an organization, you say, let's do a security audit, even how they respond to that prompt.
Tells you a lot about the security posture of the organization, then what you find?
That's interesting, but how they respond to what you find is what's really telling about where we're going, where we need to get to and the changes that we can make.
And then finally, we have to acknowledge that we have no idea what we're doing.
This is a very new space. There's a lot of technical debt and a lot of new tooling still being developed.
And so we try to invest in the occasional experiments and innovation that help move us forward.
These are just some of the organizations we've partnered with over the past three years.
I have the unique pleasure of being sort of at the hub of this and talking to people in all these organizations,
which affords me the luxury of learning about how great some of these people are and these projects are,
and also how terrifying the situation is across the space and how many things we have to do.
So I'm trying to go fast to give Pierre all the time to talk.
I'm engaged with free bistie, actually just started somewhat organic.
I don't even remember who introduced, but Ed Mast and I got together and started talking.
And thought about how can we start working together and how can we make a change and how can we improve free bistie.
And like I said earlier on, audits are our friend.
We like starting with audits and Ed thought it would be a great idea as well.
It had been a while since anybody done a meaningful audit across some of the free bistie code bases.
So we started with that and the crux of our talk today is to talk about how that audit has already changed free bistie security culture.
It's momentum, it's direction, it's intent and it's going to have a lasting impact on what was already a very secure space,
but it's even more.
And so with that I'm going to hand over the Pierre, it does all the work.
Thank you Michael.
I'd like to hold it.
So we're in the bitty forum, you must have heard about open bistie free bistie,
you know that open bistie focuses on security.
But if you check the free bistie website, you will see that security only comes second to networking.
And free bistie is built on bistie unique anyway, which means robustness, reliability.
So we already have integrity,
jails, bring confidentiality, one of the innovative security features that free bistie has been pushing over the years,
because we've got sickum for a standboxing for isolation again.
Then we have additional mitigations which we should use by some sister projects like how done bsd trusted bsd and so on.
So over the history of free bistie has been a few remote bugs, but overall it's been a really solid record.
So there is a security culture in free bistie.
And it only made sense to have a good of it to like keep pushing the status.
And here we chose to audit the b-hive virtualization layer.
On this picture you have like a summary of the architecture, roughly drawn for you.
Where b-hive is actually the hypervisor.
So if you have a virtual machine, it talks both to the kernel and to b-hive.
And then b-hive is to the operating system with process like everything else.
It runs on AMD 64, which is what we used for the live testing.
And R64, the ARM 64 bits platform is also impacted.
So in total we had 20% days of audit on b-hive.
And then also 40 additional days on capsicum because b-hive is a sandbox process.
So it also made sense to capture this attack surface.
And I see people taking pictures of the slides.
You're welcome to do so, but we have a QR code at the end just for you to get the full deck.
All right. Thank you, Michael.
So we kicked off the project early in June about two months of total time of on the coded it itself.
We had weekly updates with the penetration testing team, which was a synactive French company.
So we had the pleasure to work with them and continuously validate the vendor bitu reported.
Then coordinating this with the internal free base details.
It also actually went all the way to net b-h-d because some of the upstream of the free base decode was from net b-h-d.
So we then gathered fixes prepared binaries so that we could have a single day for releasing everything,
coordinating the information.
So we had security visories released from September on in two or three big batches.
And then finally the report was published in its entirety in the middle of November by the foundation.
We can see that on the website of the foundation.
So in the report you will find the list of the vulnerability classes, which were important to track, that way we could group the different issues we found.
Some were specific to behalf, like the MMIOIPI could probably be improved to avoid making more mistakes in the future.
And otherwise, even if it's listing time of check time of views, basically risk conditions.
Sometimes incorrect reference counting missing check for errors and so on.
In practice it's all led to the same issue, which was memory corruption in most cases.
So classic before the flows, but they were triggered by some other condition before.
So we also noticed that it was really invaluable to have validators.
So the auditors provided us sometimes with proof of concept to prove the issue was there.
And we could use that very efficiently to check the patches to see that they were actually working.
Many issues looked trivial to fix, but actually were a lot more complex.
It came down to architecture specific aspects, sometimes even compare specific aspects.
It was very complex in some cases.
It's easy to underestimate the complexity, which is also the cause of the security bugs in the first place.
We found fabricated to be great for the reviews.
This is the tool that's already used by FreebaseD to check every committee into the system to verify that it's correct.
With two pairs of eyes at least, but we also found it is not ideal for operational security.
Because even though we were restricting the visibility of the issues to some people, then it was sending out emails in the clear.
So each time for each comment, so that's not ideal.
But that's also why it's important to find not just the audit, but also the fixes, the infrastructure of the project to withstand the security challenges of modern days.
So with this said, we identified a list of mitigations that we can apply to prevent these issues from happening in the future.
We found that we should apply maybe more source code annotations, so that the compiler can catch up and be more proactively attending the development.
Directively attending the developer directly what has to be done.
One way we can do that is to force APIs with returning error codes to be checked by the caller.
So the compiler can do that if you undertake the code.
This is either said done done because in order to implement this, we need to go through the entire source code again everywhere.
Understand everything, apply the annotations, check that you don't break the bills on every supported platform.
So this is not trivial, but generally you want to automate.
So in free busy, maybe we're like in behind on tests, like some parts of the system are very well covered, but some others are not covered as much, and we should probably look at that for the.
Yeah.
I think that the point about testing is it's we always talk about everybody knows, but what both people have said is that it gives such incredible velocity to an organization that is then in response to a crisis.
Because what's going to happen in the next year is it's going to be some nasty thing that happens and everyone's going to run around with their hair on fire trying to fix it.
And having this kind of depth of testing makes that all that work goes smoother more reliably and you actually get the fix out the door faster because you have confidence in the testing framework that you have.
So that automation basis is so valuable and I wanted to I hadn't caught this when we first practice this talk my practice, I mean we've got to get a 20 minutes ago.
The the ops process of an organization, right is actually a key aspect of its security maturity and posture in terms of how well they have operational playbooks and processes and the tooling to support those so the fact that the.
Fabricator tool wasn't set up for the limited disclosure that you need for vulnerabilities creates additional load on everybody or risk around it and the timeline for this particular audit was longer than anybody expected because it's sort of opened up all those doors and all those questions.
Which makes it all the more valuable because now the organization is aware of them and working against them. I just didn't want to do that point because I thought it's important.
That's great. So also to be fair, we had the set date for the next release so it made sense to wait a bit so that everything is ready.
And that we don't have more work pushed on the release engineering and also generally due to mission is fantastic when you work in a team, especially when you don't sit in the same building or in the same company, you have different.
Time zone, different equipment, different setups and so if you have a testing framework that was the same way for everyone then you can really validate that the bug is fixed and then it's going to work for everybody systematically and no matter how many pairs of eyes are going to have to to go straight.
Yeah.
So this also means additional documentation or maybe improving the quality because you don't want to also to overwhelm everybody with hundreds of pages every time.
So in other direction we could go in this direction would be to create a training maybe go back on the BSD certification program which I think last traction in the past few years.
We also suggested that the committee could be formed which would perform continuous security testing but also a lot of people to just ask questions maybe about specific patches is the reciprocity impact did I miss something without necessarily you know like asking the security team who usually handles vulnerabilities.
To look into that because their focus should be on actual bugs and the tracking the fixes.
So even if you have a general question about security to have a place to ask an essential place to prepare trainings documentation and so on.
So we want to reinvigorate the security culture in Fribes D. It's super important. I don't have to convince you of that.
It's a hostile world out there. There is a lot of work to be done. We need community engagement. Fribes D is a community project deal.
So we only need to pay attention for our common good and if you want to get involved we suggest to write to the Fribes D security main list.
I hope it's the most appropriate place. Maybe some Fribes D developers can confirm that.
But otherwise I would say we can.
Sure.
So this is actually the test.
So I put on my alpha mega hat here. How many people here are or have been a contributor to Fribes D in some capacity?
So easily a third.
Literally how many of you show up into that mailing conversation have become part of this transformation is one of the measures of success of this process.
Because if an audit is something we have to keep doing again and the fall began and it gets harder each time, it doesn't work as well.
And from my perspective, although we have reasonable resources to fund work, we don't have anywhere near the infinite trillions of dollars.
The resources need to change and fix every culture and every organization.
And so how an organization and a community like yours takes this and says, I care.
I can get involved. I will help.
Align priorities to this has a tremendous impact because your energies will create a wake behind you that other people will follow into as well.
You know, Pierre has put a ton of time and effort.
Ed and company have done a lot of work around this.
That has created this initial push.
It's on this community now to follow up on that push and to build on it.
So it becomes a normal part of Fribes D which ultimately then drives its success.
Because make no mistake.
The next 10 years are going to be characterized by how secure is your project.
So again.
Yeah, of course. And it's the perfect time to say thank you to the Fribes D Foundation.
But most of all in the Afro-Miga Foundation.
To an initiative. I don't know how you want to call it.
Whatever. Yeah.
We gave money.
They did work.
Sorry.
And actually I have probably the best job in the world.
I get with me.
No, no.
It's really me.
Because I have all this money that I have.
I don't control directly.
I have stakeholders to make decisions.
But I get to go around meeting people like,
working with organizations like this and starting these changes.
And people are very happy when I shop with some money because it's such a hard problem.
How do we get started on this?
And again, everybody's current workload defines their priorities.
And it doesn't include the Delta to start the security effort.
But we can't permanently fund security work.
So we try to find ways to start it and make it happen.
And I also thanks also to, in fact, the Fribes D team,
care, the community of how they handled this.
You know, again, it took longer than anybody thought.
But the results were 10 times more interesting and compelling than what we had hoped to get.
We thought maybe some bugs, fixes and see like that.
And it's there we thought there was an entire culture and process shift that was very encouraging.
So we wanted to make sure we had plenty of time for questions.
I did also promise some links.
So this is a QR code to the deck.
The deck also has some additional links at the end of the slides that you can go and click to.
And I am told we have a whopping 15 minutes left.
So we can either make stuff up and talk about things.
But we'd much rather take questions.
And here would have to say.
Yes.
No, the question at all.
But just saying thank you.
The Illumus community also uses the hypervisor.
So any bug who fades, if it's for us to.
So this was a, I mean, I'm going to repeat the non question.
And what was the community?
Illumus.
The Illumus community uses the Fribes D base.
So anything we do for them.
Does use the open scenarios base.
Open scenarios base as well.
Right.
And uses B hive and that makes everything better.
I have a, at your random question, how many people here represent businesses that are effectively monetizing
Fribes D in some way where they're taking Fribes D and using it to build products that they are then selling?
I would like to talk to you after the talk.
One of my goals is to change the economics of security and open source.
Because I guarantee each one of your businesses has product managers.
Like me who have business goals, growth, success, various things.
And then customers say I need to feature X or whatever.
How many of those customers are talking about security now and how are you dealing with a CRA expectation of security and so forth?
You know, but your organizations at your robot present sustainable lifeblood to a culture and a community like this.
And so it matters.
Next question, please.
Stand you in the science. Yes.
So going back to the XKCD.
Yes.
So what do you, resin, upper omega think about the long tail of the project down at the bottom?
As we're seeing the person outside.
Yep.
How do you fix that?
So the question was how do we add alpha omega think about the long tail of single maintainer or smaller projects?
How are we going to address them and fix them?
I would invite you to follow one of the links at the end of this deck here to read our annual report that will give a longer answer.
What I can tell you is, and again, follow the QR code to the deck.
Get the links. Click away.
Don't bother taking pictures of text.
It's a really hard problem.
We have a try to a few things and we're still trying out different approaches.
One of the things that we've been we had tried is essentially using automated techniques to scan literally thousands of projects.
Find vulnerabilities automatically, maybe not all the ones, but looking for the, oh my god, you know, stupid ones we can fix.
Generating fixes automatically and then sending them on to the maintainers.
And it did not work out the way anybody thought it would, right?
Some cases, they got fixed.
The maintainers are already overwhelmed with other bots and other systems that may or may better or not as good as ours whenever.
And the really important thing here, even tried to like talk to some people about having them go and do a basic security workshop.
And we had GitHub is sponsoring a security workshop sort of one day effort.
Just to tighten up your repo, tighten up your build process, basic hygiene of your systems.
But when we talk to people, especially in this sort of broader community people, they're already essentially maxed out.
They're doing nights and weekends, you can't pay them to have more nights and weekends because only thing to do is make them more stressed.
So we are rethinking our approach, very open to ideas and suggestions, which doesn't mean I don't have any.
And one of the things that I am looking at to do in 2025 is to normalize, commoditize and make widely available public data about the metadata across the entire ecosystem.
Another thing I'm trying to do is metaphorically when we try to do these automated approaches, you'd go out, pay an organization, do some work, they come back.
And they said we've scanned 3,000 or 9,000 projects, we found 10% vulnerabilities, we fixed 50% of those and 25% of those were actually accepted.
I didn't know how to talk about that usefully.
It was like literally sending some people out in a rowboat to the Pacific garbage patch, collecting a pile of plastic, coming back and say, look plastic.
The world's better now, right?
And so we're shifting our approach to thinking about it as sections of beach.
And imagine a community beach or a local beach that has become overrun with pollution or dangerous things and, you know, tin cans and whatever.
And there's no longer being used effective as beach or if it is, it's a high risk beach.
And imagine if we can kick start a process to clean that beach, not of everything, but of a key set of common risks and vulnerabilities.
And then if the community then says, wow, our beach is usable again, we care about our beach.
We're going to stay in that beach and keep it clean, right?
That's an effort that we've done now successfully with air flow.
I'll be speaking tomorrow with one of the air flow security committee about exactly that process.
And in many ways, what I'm hoping to achieve here is that the free bst, which is already a great beach and very healthy in a lot of ways.
But we have to acknowledge there were things that were not as good as they should have been.
Can we keep this beach even cleaner over time and make it a great place for people to hang out and keep maintaining it?
So that approach allows us for to make progress and tangible progress, which, by the way, makes it easy for me to get more money to fund these efforts.
Because when I go tell my stakeholders, hey, we did zero point zero zero one percent of the problem last week.
Please give me millions of dollars.
It turns out it doesn't really work that way, right?
All right. Thank you. Is that a good answer?
Yep.
I think we have time for one or two more questions.
What we've stunned you in the silence.
Yes.
You're hitting all of my hot points. So the question was, do you have automated ways to discover the dependencies?
Of your code? And it's, it's axiomatic that even if you think you know you're probably wrong.
And if your build environment is normal, it's pulling in random shit at build time that you didn't know happened.
So hint, turn off network access after provisioning your build environment so that it can't do that.
Break the build, find out what did it should in the head, kill it with fire, start again, right?
So that from part of my goals around this space is also to create this what I'm thinking of as a public data set of open source project metadata, including hopefully a canonical view into the dependency space.
I will admit that I'm less interested in telling you in your content exactly what dependencies you have.
And more interested in getting a statistical thing saying uses a food 1.2.3, 70% of them bring in bar 2.3.4, 6% bring in bar 2.4.5 or whatever.
And I can get a sense of where the space lies and where the risk is.
And one of my intuitive ideas is that there are centers of gravity that have become stuck.
Where people are using some version of something, the maintainers of that have moved on to a future version and are doing a lot of the fixed work there and their energies and intentions are there.
But there's a bunch of dependence downstream that can't get off that version.
And we don't know about it because each one of them knows they're stuck there, but then I don't have the global picture of that.
So this, what's one of my major initiatives in 2025 is to have a sort of statistically useful and authentically as an like a tested data set from open source tooling that produces it in a very ways that we can start to reason across this space.
I will point out that the world of CNC++ is a wasteland of dependency management and it's very hard to do it well.
I'm actually probably going to look at what can we do there and more in terms of recognizing patterns of code rather than trying to sort of package for bar.
The other thing I'm thinking about doing and it's another answer to your question is what do we do when the next exe situation happens to some soul maintainer who is already, you know, this high, right.
I'm going to be working with people like Pierre and started to create or even just a reason about what I think of as a security engineering core of engineers an open source core of security engineers who will start to form policies and guidelines and playbooks.
And to act as a collective, you know, Pierre can actually take vacations, but when something happens on somebody else from somewhere else can say, I've got this one.
And so that when the next exe situation happens, first of all, there's already a human connection.
There is a reasonable reputational trust that when you show up to help because right now if somebody showed up at the next exe saying, I'm from the government, I'm here to help you.
What's the reaction going to be like bullshit, right.
Get away from me, you're trying to hack me even worse or whatever, and so we need to establish those relationships first.
Again, these are all early efforts that we're trying to figure out, but it's part of what we're trying to do.
So, yep, please do.
Yeah, so maybe I can add something to your question.
So free based the, I mean, the open source spectrum is very diverse and free based is actually a bit different from what you explained.
Yeah.
You cannot just cut network access and expect the real to break because everything is self-contained into the business system.
Yeah.
But on the other hand, we cannot automate this process.
So right now, don't know if you heard about SBOM, the software a bit of material.
We have an initiative as part of the free based project and sponsored by the foundation.
To look at this and document the provenance of every component.
So we, this is actually ongoing right now.
And I think it's also going to happen with the ports.
I think there's also a third in this direction to make it work.
But in the port, it's probably a bit easier because we already keep track of like the home pages,
where the software comes from, where it was obtained from this is very systematic into the BSD world.
And it's also strengths of BSD.
Yeah.
Excellent.
That's exactly the right thing.
And SBOMs are very powerful across organizational and visibility trust boundaries.
When you're producing binaries and somebody else to consume it,
they don't have access to your inner workings.
SBOMs let them reason about your software even when you're not there.
Okay.
I think we're almost done with time.
And so in the interest of sanity,
unless there's any more pressing questions.
Going once.
Going twice.
One more.
So the question was, do packaging repositories effectively solve the what version are people using?
I can't answer that question without wondering into quantum mechanics.
I feel like it is actually not true.
And the resolution of a package depends upon the package ecosystem.
How it resolves pin versions or not pin versions.
How diamond dependencies get resolved at the local client in the Python ecosystem.
It actually get resolved by any one of six or seven different clients.
Each of which will interpret the resolution differently.
And so there is no correct answer, which makes me cry.
But this is part of the problem.
By the way, if you are contemplating designing a new package manager and or a new programming language,
and you are not including the two things together and making the language aware of the package and vice versa,
you're making a big problem for the future.
All right.
We have two minutes.
So.
I just want to say one thing for your question.
The problem with the package is.
I'm going to record your world.
We kind of record.
The problem is some built systems try to be helpful.
And so they say, oh, you provided me with the path to lip curl, but I have my own built in lip curl.
I'm going to use that one instead.
And it's silently does it.
And you don't even notice.
And then this thing uses the vulnerable lip curl, even though you thought you had it all up there.
And it's things like that, like,
built systems are tried to be helpful.
Most of the time are not.
All right.
With that positive and exciting note of the future.
I want to thank again, Pierre.
My colleague in this was fantastic.
And thank you all for taking my close today.
Your heartbeat's invaluable.
We appreciate it.