Let's start, okay. Hi everyone. Welcome to a talk you probably have never understood what
it would be like before. We didn't know either. Mine is Michael Windsor. I run a project
called Alpha Omega, and this is my friend Mirko. Hello. If you haven't figured out, I'm
allowed and talking to one. Mirko gets things quietly done behind the scenes. So we're
going to start by just talking about our respective projects and how we do things. And so
with that, I'm going to head off to Mirko. I was thinking you did a little bit more of an
intro, but I didn't do that. No, no, I didn't. So yeah, so the idea for this talk was basically
to come together to talk about how to combine the efforts like this life before actually,
and to talk about how, like industry and, you know, private, you know, initiatives can,
you know, combine the effort basically that thing, right? And Michael, me, we have been talking
for some time now, and we thought we do this talk together to actually, you know, also invite
you into our thinking here. And that's really about also the session to be able to do a little bit of
instruction of the organizations and what we thought, but then we'd be really like to make this
a collaborative thing so that you can also share your thoughts around things either from the
private sector or from the public sector or whatever background is or whatever suggestions you
might have. So I start with just quickly saying some sentences about southern tech agencies.
So my name is Mirko. I'm the head of the southern tech fund, which is now one of the programs
within the southern tech agency. Still, yeah, the major thing that the thing that you might also
know, we taking application, it's apply dot southern tech, southern tech, so we have to
adjust to the new website. And, but we also have other programs like the southern tech resilience,
which my colleague Tara is leading, who's also thankfully organizing a lot of around this
funding there from here, thanks to that as well. And then also southern tech fellowship,
I talked a little bit briefly about the thing earlier, so that's also a very interesting
program where we invest into open source maintainers directly. So there's like six people right
now, we started right first of January, so I'm very, very happy that we could make this happen.
There's lots of effort as you can imagine, because we are public procurement laws as we
have to stick to them and then I'm very happy that we can make it. So it's like six people
working with us from very different backgrounds from very different technologies and areas
with this one person taking care of cycling the eggs, also in the room I believe. And then
there's other folks from the Python ecosystem, we have contributed to curl in there.
We have people working on open street maps, so there's very different maintainers.
And we want to pilot this only six people right now, but we want to make sure to learn
from this experience now for both of us and for the maintainers, of course. And then to
scale this in the future, also depending a little bit of our own funding
situation, so we are an organization that's what I'm supposed to say, that is funded
or like, you know, financed by the German Ministry of Economic Affairs and Climate
Action, and we are very hopeful that this will also continue after the new, so there
will be elections in Germany soon, and we hope that people also get our budget, you know,
confirm again. And yeah, on the next slide, I just wanted to do a little bit of, yeah,
that's our nation statement, so we are maintainers fund, so it's really not about, you
know, innovation or AI or something like that, it's more like really the boring stuff,
like maintenance of digital core infrastructure, like we used to talk about this one, like
think of the look for days and all that, that is basically only visible when it's getting
to become a problem, right, those kind of things. And on the next slide, I just want to say,
we are growing, we are hiring, we have a jobs page out, so if you want to go there,
you get paid to work on this problem. Yeah, you get up with me, and not all colleagues
of course, and yeah, I just have a look, I just want to say, this is still running until
next Thursday, like on February 6th, we are closing the application for this cycle, so if you
want to have a job and open source, please apply, and then I'm handing over to Michael
for my family. Okay. Excellent.
All right, so I'm Michael Windsor. I run this project called Alpha Omega. I co-founded it
in 2021, 2022, when I worked at Google with a colleague of mine named Michael Skaveta who
isn't Microsoft, and if you read their mission, you look at our mission, it's an awful
lot of overlap. We are very focused on security. Security outcomes in open source, and essentially
helping solve the decades-long, like 60 years of technical debt in open source security,
because people have been focused on other things, and now we have to fix all the problems.
And it's a hard problem. The way we think about Alpha Omega, when we first started, we were thinking
about Alpha representing the top 100 most critical projects, and then Omega was the long
tail. It turns out there is no interesting top 100. There's an awful lot of very important
projects, and Alpha has come to represent points of leverage, where we can go and work on areas.
By the way, don't take pictures. I have a QR code at the end with the entire deck available
for you to save your time. You're running out of pixels, otherwise. Film pixels isn't there?
So we now focus on points of leverage, where we can with relatively tactical, small investments
or not small, cause entire ecosystems to change their priorities, to start doing work, to improve
doing and benefit hundreds or thousands or even millions of developers. There are people in this room
who are doing work on our behest, and by that I mean, be there, do something, and that's
really one of the key things. The scale part is still as hard. We have tried a variety of
efforts on how do we solve improving security on the hundreds of thousands of millions of
projects, some of which are no longer maintained, and yet still actively used.
How do we fix that? And we've tried and learned a lot of things.
We, I should have mentioned before, we are funded today by Microsoft Amazon and Google.
I know in this in climate, this is a different set of audiences like that, but they are very committed
to putting this money into the space. The total funding so far has been eight or nine million
dollars with another few more coming in this year, and we have a four-prong strategy.
So, a significant amount of our investment goes towards making it someone's job to solve
the security problem in a particular ecosystem. Mike, raise your hand. This is Mike.
Mike is working on securing PI PI among other things.
The package managers are the app stores of software development. They represent points of policy,
points of trust, points of choke points where we can introduce new behaviors, and Mike has been
doing a standing work towards that, and he has a colleague named Seth Larson, who has been operating
across the entire Python ecosystem and influencing beyond that. And so we've already seen
tremendous success with that, and we're going to keep doing that kind of work. It's expensive.
These people want to get paid. They want to have a life. They want to travel to the expense of events.
It's crazy. The package repos remain very interesting points of leverage for the
interest in points of leverage for us, and so we also invest in direct,
it's a contract work or project to go off and make tactical improvements. One of my favorite examples is a
project called home brew. Everybody here use it. Well, I think it just went live.
It's been in beta for about six months.
Home brew now has end-to-end attestations with six stores, so that the bits coming from the source
repo all the way to the binary on your workstation or more importantly on your GitHub
actions build environment are fully checked and signed the whole way. So no tampering can happen with those bits along the way.
That's huge. It means everybody using home brew now has a more secure set on their tool chain,
which means we can start to make assertions about the validity of something coming out of that tool chain.
If you haven't read Ken Thompson's articles on trusting trust, please go do so, and then you'll stop sleeping for a long time.
I kind of really see this is actually where we start most of the time.
It is amazing how much everybody learns when you start by doing an audit.
When a group comes to us and says, we need improvements in our security. Give us money.
You'd be like, what was the last time you did not it?
We don't need not it. Great. Yes you do.
And how they're ready to take on an audit is very compelling and interesting.
What happens when they do the audit? How they respond to the audit?
How that problem changes? And do they become more secure focused?
Do they stop having the same problems or in fact, is it just allow the rinse repeat?
See you in 18 months with a bunch of the same bugs showing up.
All it's our cost effective, have longitudinal effects, and have significant impact on a particular ecosystem that is being audited.
And then finally, because none of us know a damn thing about what we're doing in this space.
It is all new. People are still making things up every day.
We have a certain chunk of our readers is going towards just things we don't know if they're going to work or not.
Experiments, innovation, just stuff that seems like a worthwhile try.
So with that, we're here today to talk about what we've been doing together.
The lessons and learnings that we have had and the conversations we have and how we're looking at working together.
We come from very different sort of funding entities.
Our operational model is quite different. You know, you're looking now at the entire funding sort of fund application process for AlphaMega.
You talk to me.
I work with you. I collaborate on something. I tell you straight up if it's bullshit or I tell you straight up if I want to make this work.
And sometimes I come talk to you and say, I want to fund your work and they're like, who will help you?
It's different.
But at the end of the day, we talk a lot now about the things that work, the things that don't work, what we want to do or whatever.
And so we thought we'd take some time right now just to share some of the things that have been very interest to us.
I talked about audits.
That's a big deal.
One of the key things for me and I'll just to kick this off.
There are no small dependencies.
You know, the old joke that I know small parts in theater, only small actors.
There are no small dependencies.
Every dependency in your entire graph, the full transit of space has total access to your build environment and total access to your runtime environment.
So the least interesting, least effective, small little string parser thing somewhere down there, right, is now a really juicy point of attack into some applications of a structure.
And we have a very poor handling of that.
That sort of one of the lessons from the XE process.
We did an audit with the Airflow project where we didn't audit their code.
We audited all 719 dependencies.
And we didn't audit their code.
We audited their security posture.
It was awesome.
We learned a lot.
We didn't finish the auditing well as too many.
We're developing tools to get there.
But it's changed our approach to scale.
Our original approach to scale, we're trying to automate techniques and automatically filing bugs and automatically filing mediation.
We've all seen those bots.
It's not a great story.
And the teams that came back, we like say we went out and did all this stuff.
Look at what the work we did.
It was like going out to the Pacific garbage patch and coming back with a robot for the plastic saying, look, and nobody gives a shit.
But now we can talk about Airflow as a beach that has been cleaned by its community.
And by virtue of spending a relatively small amount of money, daunting that supply chain,
I've actually caused the entire Airflow engineering culture and community.
60 engineers, probably, to prioritize security work and more importantly, to prioritize the care and feeding of their supply chain.
And so here's the conclusion of that lesson that I'm going to give Mirko a chance to talk.
Open source projects.
Love to complain about how the corporations above them are consuming the source good for free and not giving them anything exchange.
It's true.
A lot of it happens.
Open source projects are just as bad as their corporate complainers in that they consume their upstream supply without even talking to them.
People like in wire guard and open SSH are using XEU tools and they didn't have a single point of contact.
If you work in a company and you have a vendor and you don't know who the vendor is, it's a bad story and yet we all do the same thing.
And so there's a simple model.
You have to apply one of the three F's to your supply chain. You either fix it, you fork it, or you forget about it.
Thank you.
Thanks, Michael, on point.
I think what we've learned, I talked a little bit about the Fellowship Program last year that was a big learning for me obviously because we had so many like insights with the survey that we did.
And that's really also really helping Adriana, so there should speak louder.
Yeah, that was really a big learning class because that is kind of the window into the community as well when you really start working with Fellowship.
Or like with the community, especially when they started the collaboration for me, I mean it started only like in January, but I have so many insights now from working with those people directly that is really something that I can feel is very
you know beneficial for the whole ecosystem that we really start working with open source maintainers directly, not just funding them, but getting into the collaboration and asking the interesting questions.
Like what is your current, you know, what you're thinking about, what's top of mind for you currently and just get into that conversation and learn about the different needs that people might have and that's really interesting for me because I said it earlier as well.
I think there is no one silver baller that I think the ecosystem is very diverse and very different, so money might help here and there and that's also what we are doing for Omega is also doing, but there might be also other instruments that we need to take care of and that's only when you have the conversation with those open source maintainers directly.
So like what we also learn every day or like every month is we are financing projects but then in the end we request the final report and that then report for us has like five very easy questions and it's like something like yeah what does it change for you.
But what has changed for you throughout the investment bias of an agency for instance and there's lots of things where people are just sharing their learnings and that's also learning for us to them right so when we have one portrait sitting here on the room who said like you know what if you wouldn't have you know invested in us then we probably would be in the different other situation right now because we you work kind of a life saver for us right and that's that's interesting because at some point.
You want to improve our projects or when I have projects improve for already on a good stage we're going to be going to make sure that they really maintain their their maturity but at some point we also want to help people to to get out of the trouble that they're currently in so that's.
So that's a lot of like learnings that we have throughout our work but then again what's also interesting for me because I have this industry background as well so I'm not another source maintainer and I'm I'm just a consumer.
And consumer all my life right and then for the industry I also know that there also be consumers and I want to get into the conversation to really leverage the potential that we have there in order to get more people into really contributing to the funding of open source but also developing together new instruments and that's why we here we want to talk about that.
And we're like collaborations and about like what can we do together but not only to just you know at to each other but maybe also use a little bit of a scaling thing there right where we maybe identify things where we can really make a difference together by identifying I don't know gaps that we wouldn't have seen each of each other but maybe also like.
So there were so many things you said they were great one of the which is like asking people to write what they did.
Amazing effect when we ask like the PSF to do anybody who does a grant with us we ask them to put a monthly report together and get hub.
We want to blog post the beginning and a blog post at the end and then we do an annual report and there'll be a link to our annual report and it's it's amazing reading.
The PSF started doing this monthly report to us and they're like well this is pretty good stuff we're writing down and they started sharing it internally in the Python software foundation organization and then brought it to the Python ecosystem.
And of course it had a great effect it created this week that people were following.
And one of the reasons that we are now spending more more time talking is that.
We see these fascinating sort of like I don't know how to describe it knock on effects or cascading effects of our investments and our work together and independently.
So when we fund staffing we really hope that the organizations that we've funded we find a way to make those roles permanent and staffed as part of their budget.
In practice that has been very hard.
But what has happened is a lot of energy has flowed behind those people.
And so as long as that thin edge of the wedge is driving more investment around that it's really powerful.
And you know one of the things that we're going to try to do this year is do some audits or some things put us together and we're going to ask for.
Other organizations corporations to come along even with a little bit of money or a little bit of engineering resources.
And that sort of brings me to like you know people often talk about funding.
All right.
If you look at the most healthy open source projects today, the ones that are just vibrant and growing, they are resource not funded.
And the more you can connect your project to an entity that has resources which means they have some path to money or they have some.
Resourcing staffing of some kind.
The more you can connect that work to their needs in their agenda, the easier it is for you to go to market and succeed.
And you know airflow is an open source project widely used but also widely monetized.
And so the engineering companies that are operating as a hosted service have engineers very close to revenue.
It's very easy for those engineers to be put on to airflow to accept the governance of the project.
It is a well-governed project.
It's patchy.
No one company has control so it has all the good things.
And it has a lot of engineering resources that are now trickling down.
And that pattern is really interesting to me.
And I think that it sort of reflects sort of going back to the exit details of this problem.
And I'd love your thoughts here.
There's an inequality of resourcing across our ecosystem.
We have projects that are very well funded.
Corporations that are well funded.
And we have lots of smaller projects that represent tremendous risk that are not well funded.
And we need to find a way to percolate those resources and to reduce that inequality.
I just want to go over to you then again.
But one thing that might be interesting to point out again.
But it just realized today is that also our black booking model,
Michael, it's a little bit different.
You are very hands-on on the things you are doing.
You are there.
You talk about yourself or the better end in the industry.
You have those 40 years of experience and being very technical as well.
Being hands-on on the projects.
Whereas we are a little bit more hands-off than that we got.
We are putting a lot of trust into the communities and technologies we are working with.
So that's a little bit different.
But still there's overlap on the things we are doing.
So one example that we might just briefly talk about is a clips where we,
from our perspective, that's just a collaboration by surprise.
It's just like we are doing the same investing in the same organization.
It's a different goal behind it.
But with the same objectives maybe.
I think it's a great example.
The clips Foundation is one of the best run open source governance projects there are.
It's really well run.
It's very professional, very competent.
And it was interesting to us because it is a very diverse set of projects.
It's not a Python ecosystem where Ruby won.
It is a bunch of projects that have a variety of the same inequalities across those projects
that are there too.
And we are staffing a security team in that organization to drive cross-foundation security infrastructure
and practices, vulnerability management and so forth.
If you haven't seen Marta's talks on vulnerability management, I encourage you to do so.
She's amazing.
And we were very interested in, you know, the work that we did, the funding that we produced
and the team that was in there, ultimately put them in a position to go to the EU
and to put themselves in a sort of leadership position with respect to the CRA and drive significant changes.
We part of that conversation and that I think empowered them to go and ask for resources and get them both
from the EU and STA towards, you know, the ORC and other entities that are trying to
essentially solve this non-trivial problem of how do we do open source in a increasingly sort of regulated context.
And so it all comes back to the end of the day.
If you make it someone's job, humans, if you make it their job to worry about a problem,
good things are going to happen.
So should we start taking some questions?
Okay, we're going to do that.
So we're going to take questions.
This gentleman here has had his hand virtually up since the beginning of the talk,
wanted to say something.
So go for it.
Thank you.
I want to first of all, I want to say thank you for dropping a seat with culture.
Yes.
Culture, especially in regards to audit and rather moving the fear and embracing,
it has actually been a wonderful feedback loop, I think that's essential.
So I wanted to ask both of you what element of culture impacts of open source communities,
impacts elements such as maintenance.
So because independent of size of user base of open source projects,
to me, the culture of each project is just so essential to its future,
sustainability, its future use patterns, user types, et cetera.
Could you repeat the question part of that?
Sorry, what?
What is your opinion on the relationship between culture and what it processes that it's
that it's exposed dependent on?
Fair enough.
So repeating the question as concise as I can, what is sort of the relationship between the culture of an organization
and the audits that happen in that organization?
You should start.
I'm going to make up a bit.
Thank you so much, Mark.
No, I think for us it's a we have a very holistic approach,
as you talk about agencies.
That might be also different because our format is very, very much focused on security.
And that's also for us, a very, very interesting and important objective and outcome.
But we look at the whole thing a little bit more holistically.
And you know, a good maintain code, happy maintainers,
for my perspective perspective automatically lead into like a better security posture.
That's what we're doing.
And then we're not only investing in audits,
but we're also investing community management for instance.
And in things where we pay people to get into, you know,
work with their communities and, you know,
bring new more diverse collaborators on board and all that stuff.
So that's also important for us.
Which gave me time to think of a good answer as well.
No, I think it really depends on that community,
but also the state of the project and where they are.
And one of the hardest problems today is that on the smaller projects,
an audit is an overwhelming thought in the first place.
And the first responses, you do an audit and you're going to find all kinds of problems.
I don't have any time.
And one of the lessons we've learned is that if you've got people who are working,
and you talked about this in another context,
nights and weekends, to do this work,
you can't pay them to have more nights and weekends.
And so even when we've talked about coordinating security workshops,
to help them do basic hygiene of their repo,
and they just see it as more work.
Now the cultural problem there, if I may be a little bit sort of paternalistic,
right, is that they're focused on growth goals
or contain sustainability of their popularity,
their project, everybody's essentially competing for attention
or resources in some way.
It's the world we live in for better and for worse.
And so in that world, security tends to take a second,
sort of footing compared to growth and other success metrics,
things like the CRA are intentionally creating pressure on that dynamic.
And it's not easy and it's going to be messy.
There are projects that probably need to work on an end of life strategy.
And that's, by the way, super hard because it's all very well for someone to say,
I'm done. There's a bunch of downstream consequences of that.
But also if you're looking somewhere up here,
you're like, I want to get rid of this thing here because it's dead,
but it's 16 layers between you and that.
You can't go and get rid of that. You have a long list of problems.
We're being pressed for time and we're answering so much time
of questions. So that gentleman there is next on my list. Sorry, sir.
It's Christmas. You said talk to me and we have conversation.
I tried and think of a make-up site doesn't.
There's a form. I feel within, didn't hear anything back.
So maybe it did something wrong. There's no other contact information.
So the question was, how do I apply it out for a make-up?
And there's a form. And you fill it in and maybe you didn't get a response back.
I will admit that since you're talking to the entire operations team of the alpha-mega process,
that it's almost certainly my fault. In practice,
we are, we benefit from a very short cycle of attention and focus and decision-making.
And occasionally things fall through the cracks.
And knock again on the door. I'll be outside after this talk.
But we also, we make decisions quickly in terms of not even worth our time.
And I'm not saying that's the case. If you have no idea which one you are,
but we are very clear and crisp and what we're able to do.
Because we are that way we can do things in ways that I think my colleague here
would sometimes envy, but I also envy his process as well.
And I'd love to have a government funding me one day that would be great.
So we're going to take one more question from this gentleman here and then we'll probably run
out of time and kick them out. Oh, he's in charge of the room, so he can take one.
But the other is the solution, the solution about that you talk to each other on a regular basis.
And my question here is, did you identify things where it's okay?
You look like you do that, but it's impossible for both of you, like organizations to actually go in there.
And you know, is there something that would tell you, like, we need to put,
you get these or these partners on the board to do certain other things that you can do now.
So we focus, the question was, is there anything in our, like, things that be discussed,
which could be possible that are not possible right away?
And then we focus very much on the things that are might be possible in the future.
It's a one thing that is discussed is security audits for instance to, you know,
to some match, jointly fund them matching together and helping, you know,
leveraging the thing.
Positive or other partners as well in that field. That was one thing.
And then on the other hand, like, I for mega has some experience around, like,
funding folks like Mike over here, or like Cecil Austin, and we are also doing, you know,
this fellowship program, we will also fund folks directly.
And then there was a conversation around how can we combine things to do something together,
and that field as well. So we are more focusing on the things that might be possible,
but still it's in the early stages of this collaboration.
And we, yeah, just in the beginning.
Should we do one more? Yeah, one more.
So a lot of sexes of the project depends on how the sexes of a project depends on how good their business is.
How much do you spend time with these funded projects in helping improve their business?
So they are, you know, not depending on external funding,
because sometimes you need extra funding to manage a funding.
So.
How do you help?
I love this question.
I can quickly answer because there's one gentleman in the room where we had exactly this conversation,
and we'd be discussed this.
And one thing that we are, that we've done for the first time is kind of a business model workshop kind of thing,
where we try to sit down with the open source project and identify things that might be, you know,
making them possible to monetize the thing in the future,
which is only a tiny little thing to organize such a workshop,
but we are in the starting of of of this one as well.
So I have two answers.
One is when we do work with a project,
especially if it's sort of a build a thing that does a thing.
Like, you know, the home brew work, we funded trail of bits that you do the engineering work,
but the, you know, I was like, well, who's going to own this at the end of the day?
And obviously it was the home brew collective is doing it.
And so although the proposal came from William at trail of bits, he's amazing.
I sat down with Mike from home brew and said,
I'd rather fund you and then have you own this and he's like,
no, don't fund me.
My collective is a nightmare.
I don't want to deal with that.
I shouldn't have said that.
Yeah, he didn't really say that, but he wanted to go direct to them,
walking it back.
We still recorded.
Oh gosh.
What am I going to do?
So, so, but I made a very clear to Mike,
you have to own the code afterwards, right?
Because one of the, and we talked about this one of the previous sessions,
one and done funding creates another interesting set of problems, right?
We actually keep funds in reserve for our staffing things,
so that if I don't play the opx politics well with Microsoft Amazon and Google,
that I have, hopefully, around a year's worth and reserve,
so that I don't pull the rug out because the worst thing you could do is hire somebody to work on security
and then take them away, right?
So, but that sustainability of a project really matters.
And one of the conversations that I was having with Mike earlier today is,
notably, some of our critical open source infrastructure has a revenue stream based on
descending growth, descending donations, and has a transactional operational cost
and security needs that look like this.
And so, rethinking the economics of those systems is part of what I do.
We're being given the times up signal, which seems, you want to add something to this, right?
Yeah, so to the level down, and we have, we're trying to engage with it.
Just on one aspect, because we talked about funding and resources,
and just to make you very clear that with some of the tech pages we're trying to
make governments realize that people who work for an open source project,
often times, may take infrastructure that is a public service,
but we're not paying with the public money for it.
So, we're not understanding ourselves, like, founders,
but we want to build a new kind of organization that pays for something that's
required to rely on, as of right?
Because it's not a bit of a roller over it,
and it's not a nice to have this foundation.
So, I think when we talk about funding and flexibility on what's we need to put that
into the mix, because that's a missing part in it.
Very much agreed.
So, I'll painfully summarize that for the stream.
People maintain and critical open source are public good infrastructure
and need to be supported by public infrastructure resources.
Without, I think, thank you, Archmerco, for time today and here as well as that.