Developing in order to build smooth operating machines between the space, and so we're still
learning, and our journey with the Airflow Committee community has been really interesting.
So this is just a refresher of the threat model.
I'm not going to spend too much time on this today, it was been a days like that.
If you are, have been living under a rock, this is a quick refresher that basically every
18 months, there is an absolutely spectacular, major supply chain attack, and every
day there are all kinds of other attacks happening.
So it's just hard, right?
We're still figuring it out.
From the work that we did, one of the things that came out of this is that as a consumer
of open source, which means everybody who is building something in open source unless
it's the bottom of the stack library, right?
You have a responsibility and a need to look upstream and to make one of these choices.
There's a different version of the last F. You have a choice.
Either you're going to get involved and engage in fixing the things the problems upstream,
you're going to fork it and copy the bits that you need and reduce the surface air of your
risk, or you're going to stop using something.
There is a fourth F, and that's about funding, that's what my project does.
I can tell you from three years of experience and spending close to $10 million, this
is much harder than it looks.
In particular, in those smaller projects, it is all but impossible to just turn money into
security.
They're already working nights and weekends.
You can't buy more nights and weekends at their time.
You can't turn them into security experts.
You can't hire another person in that org.
It doesn't work that way.
So there are real challenges to doing so.
AlphaMega has a mission.
Our goal is to go off and protect society by fixing this problem.
We see it as a very long-standing mission.
We are not funded to do all the security work or to fund all security work.
The word catalyze is critical to how we do things.
We do so by investing across four separate types of investment categories.
The first turns out if you make it someone's job to worry about security, things get
done.
In particular, in the Python software foundation in Rust, in Eclipse, Ruby, I'm forgetting
many.
We've been able to hire the first security engineer in residence in those organizations.
We pay for those roles.
They hire them.
They manage them.
They set their goals and targets.
We just say, have somebody whose job is to care about this and the impact has been astounding.
The surprise impact is when we bring them into a room and have them talk to each other
even more great things happen.
The second category is around package repositories.
These are among the most leveraged points of influence over the STLC software development
lifecycle everywhere.
Every developer, when faced with the problem of getting tab A and to slot B and Google's
the problem, finds an out-of-date stack overflow article that says, do not use this and the
fourth common in the bottom says, MPM install food worked for me and they're going to do
it.
And unless you have checks, policies and gates in place, you're going to end up with food
and it's entire transitive dependency graph till the end of time in your project.
Category C is about audits and I can't say this often enough when we invest by putting an audit
into a project and helping them audit their processes, their culture, their tooling, their
code, we learn an awful lot about that organization's readiness to actually become focused
on security.
And so we tend to start with that because how they respond tells us a lot about whether
they're ready to do more or not.
And then finally, deeply acknowledging this is a new space and none of us know what the
hell we're doing.
We keep trying things and see how we can screw things up probably and where it goes.
Making mistakes, trying new things is a key part of what we do.
This is just some numbers.
I'll skip over it.
We spend a lot of money.
This particular journey, I think, is worth wasting a little bit of time on because of how
awesome it is.
Is this supposed to move in some way, as we click it first?
Right.
So that engineering Python I talked about, Seth Larson, was hired by the Python Software
Foundation.
He was a longstanding member of the community, had an interest in security, we able
to fund his role and he did amazing work and continues to do amazing work and is really
a leader across all of these organizations.
I think that's not because I know him and I talk about him, but he and I did a talk
at Python and because that talk, Eric came up with the talk, this was a great talk.
I want you to do the same talk at the Apache Airflow Conference, Summit and I'm like,
I'm not doing the same talk again and we got to do something new.
What are we going to do?
He's like, have I got an idea for you?
We had been doing things with Muno are already around scaled approach towards vulnerability
discovery and we did it by scanning things and then the team came back and said, look,
we scanned.
Things we found, this many percentage of vulnerabilities, we fixed this many, that many
were accepted and it was like rowing a boat out to the Pacific garbage patch, coming
back with a pile of plastic.
Yes, the garbage patch is smaller.
I can't do anything useful with that information.
And so we've been switching to this mindset of like, can we find a section of the beach that
is well defined and I ask, you know, Eric, do you have a section of the beaches?
Well, I have the exact list.
I have 719 that numbers they've asked in my brain these days, dependencies and I know
the exact list and I have these tools.
I'm like, we're going to do something together.
And so that is what happened and so Eric is now going to do all the interesting stuff.
I just talked about the setup, but it is very interesting about how we got here.
It's about people.
It is always about people and change happens when you find the right people and do things
together.
Now you get the microphone.
Can you get this?
That's under the picture.
Yeah, so that's the beach that we are going to clean together and we are already doing
that.
Okay.
So yes, let's talk about air flow security.
That's what it all started.
So air flow is already very, very active in its security.
And there are some people here who can confirm that because for example, I work with
Arno who is our Apache software foundation security response engineer and we've built the
security team together and we work on it and it's actually super active and works very well.
But our flow itself is a very active and big project.
So we have like 150 active pull requests in a week, which we're attached, or 76 active
issues which we're opened.
We have 102 commits merged every week or 124 commits merged on all branches within
like just around the week.
We have a lot of people working on it.
We have an up very big number of users for this 14,000 is just the GitHub.
But we have a lot of pretty much everyone who is processing data, uses air flow air flow
is an or data orchestrator in case you didn't know.
We have 3,200 almost contributors that's the start from today.
I always update this slide before talking because it keeps on going and very, very, very fast.
Something happened and this is the active air flow in security.
If you look at the history of air flow here is like a timeline and you can see like those
are commits and versions of air flow.
But at the top, these are the vulnerabilities that were discovered in air flow and you would
see that at some point of time they start, of course the vulnerabilities were going down
because this is like the how many open vulnerabilities in the history were.
But at some point of time they started to go down faster because we had invested in our
security ourselves in the security of air flow as a project.
So what we've done, we have a dedicated security team.
I go very quickly because this was just a base.
We have created and documented the whole detail process how we are dealing with the security.
We introduced the security model describing our researchers and users how the security works in
air flow.
We prepared canned responses to issues because of course we are flat with a number of issues,
but that's another story that we've heard recently generated by different kinds of people
and we are responding to them in the same way and we know how to.
We've disabled some inherently insecure features.
We hardened our CI workflows quite recently.
I removed a pull request target by the way.
If you're using pull request target in your GitHub workflows, stop doing that immediately.
It's super insecure with it.
And we introduced a reproducible builds which shows us that actually even
the release manager didn't tamper with the air flow builds when they were prepared.
Sometimes somethings that happened with XE or XE that depends.
So we've done all of that.
We have like 15 people in our security in more or less five is productive.
62 commuters, 32 PMC members, we're huge projects.
More than 2,000 contributors.
And the important thing is like we are big enough to attract funding.
The fourth half that Michael mentioned.
Yes, yes.
Yes.
It's hard to make it turn into people.
People working on projects?
Yes.
It's what matters.
Yes, and we have stakeholders who are supporting our work.
Like I'm an individual contributor.
I'm full-time, or I tend to say 150% time open source contributor.
And I'm fully paid by Google, by Astronomer,
and few other customers, or users of our flow, who just chose to pay need.
Because they think it's great if I'm a contributor or a committer there, or PMC member.
So we have a lot of sources of this money, people, energy, and focus.
We also have like sovereign tech fund interested in us.
So the part of the security improvements in the last year we've done.
Or the 2023, even, were financed by the sovereign tech fund investments.
And they paid for this reproducible bills, for example.
And now we have our mega fund, but are we secure?
Can we trade that we are secure?
If you look at our dependency tree, the number is much smaller,
because we've already removed some of those.
And this is not full number, not 719.
But right now, if you look at the dependency tree of airflow,
if you look down, is 579, does the number of lines?
Does the number of lines you cannot see, actually?
Well, we have a lot of them.
And we have, of course, security regulations.
We have more less than two years.
Everyone is impacted.
Everyone needs to be involved.
We know all of that.
I don't want to be repeating that.
So this is an F-bomb room.
And I was thinking, why the hell did I submit?
Well, why the hell did I submit talk to the F-bomb room?
Because it's, I see, at the S-bomb room,
most of the people are talking about the creating S-bombs,
about attributes, about number of things.
Nobody talks how to use them.
Like, nobody actually says, like, how to make them useful,
not only for regulators who required them,
but how to use them, and we did.
So experiments start, because we are in the fourth part
of what Michael mentioned.
Yeah, we have an experiment.
So we are the experimental effort, which might not fail.
Apache software foundation, PMC members of our flow,
Python software foundation is involved.
Our front agafand is involved.
Some users are indirectly involved, because like Amazon, Google,
they have thrown on their day of helping with that.
We also have open refactory.
And we are using CDXN for generating our S-bombs,
which is great help, because from those S-bombs,
we were actually able to draw very, very interesting conclusions.
We use the, we have some security out this from S,
as well, that we looked at.
We have some external researchers.
The idea that we have, we want to know our dependencies.
We want to review and talk to all 700 of our dependencies.
We will do that, I'm pretty sure of that.
We are learning how to do that.
We learn to how to automate that work of communicating
or initializing the communication,
because we believe that we should talk to the humans,
to the maintainers there.
And we think that it's like how we as a product look at them
is important as we as a user.
And we always remember the people.
So the biggest part of it is not to automate, not to show numbers,
not to show spreadsheets.
This is very useful, and we do that as a starting point,
then we talk to people, because this is where security starts.
So we thought it's some kind of our analysis on those 700 dependencies.
And we came up with like 16 of them.
And that was all automated.
The whole thing that I show you, this is automated step.
So we have some all open PSF score cards and results.
We had some like description of those results.
And finally, from that, we did design our automation,
how to use the S1 to generate actions that we can make
based on that information.
So we are not looking at numbers.
We are looking like what we can do to help those dependencies.
Then I'll pass to you, and you can continue.
Yeah, I'll just thank you.
Sure. So when you think of the three tenors,
you remember, I'm very about you.
You remember Placida domingo, and then there's a third guy.
So I'll be the third guy.
I'll just make two quick points.
So I'll make two quick points.
Number one is, when you look at 719 dependencies,
what we are doing is we are looking at creating a capability
to proactively go and scan into all of those 719 dependencies
or a few hundred in your projects,
find previously undetected security bugs in them,
and then work with the maintainers as well as you
in order to manage those dependencies.
So think of that.
This is beyond the signal that is provided
by software composition analysis tools,
which is only giving you known vulnerabilities.
We are talking about previously undetected security bugs.
So in this case, out of the 719 packages,
we looked at that in about six weeks,
and identified 14 new bugs, three of them, high severity,
and they have been right now mitigated by airflow and by others.
Point number two is, we all know that there's no love laws
between the security researchers and the open source maintainers.
And there's the drive by PRRs, there's AI slops,
these are like, have become very familiar.
So this is an actual story that I want to tell.
So this is a vulnerability that we identified
and we reported to the maintainers,
and then nothing was happening.
Then after a few days, Yerick,
he posted that he is involved in the air flow project,
and he was like, he's conduming that particular,
and air flow is conduming that particular package,
and it would be interesting for them if they have an eye on that.
And then the response was, oh, I thought that this was an AI slopp,
and it was just generated by an AI.
We don't generate anything by AI.
They're generated by computers,
because we can't type everything,
but they're not AI generated,
but then people have perception of that.
And then the story was very sweet,
like then once it was explained to them,
soon vulnerability was fixed.
The point of the story is when you involve,
security is a human problem.
When you involve the downstream consumers
into this conversation between the security researchers
and the open source maintainers,
then we'll have a much amicable conversation,
and oftentimes many of these vulnerabilities would be fixed.
Thank you.
Okay.
So let me quickly continue what the experiment has done.
We identify 16 projects to start with,
and we propose them to add security policies,
follow up on secure workflows,
propose an able-interested publishing,
following up on unpubsed vulnerabilities they had,
and we propose them to introduce mandatory code review.
So those are the actions that we wanted to talk to them,
but really, it was about the conversation.
We really are about talking to them,
security is important to us as our users,
and they are responding.
The responses were very, very different,
so we had a bad, vast majority,
if like, happy to get help.
Actually, you know, I didn't put numbers here,
but I have now more than half of the maintainers I talked to.
They are either eager to get help,
and they are looking forward to it,
or they are almost as big security fixes me,
which is pretty impossible or very difficult, actually.
But they are very, very, very, very security fixed,
few of them didn't response,
but I didn't follow up yet.
I didn't have bandwidth so far for that.
Some of those developers could go away CRA is bad for me,
and there was one coordinator,
I don't know if you've heard about that,
this is the story,
so the guy which I contacted,
who was following a layer communication of mine on the CRA,
he said, I'm removing all my repositories,
and I will remove it from pipeline.
There is a very, very interesting case that you should look at,
we are still discussing with him what's going to happen in March,
because he still wants to remove,
because he's scared about CRA, that's another story.
You want to say about availability?
Yeah, exactly.
So, we hope it's not going to happen,
but the fact that we started this conversation,
triggered the whole thing,
and if you haven't heard that of, you should look at that.
So, longer-term targets,
we want to do full automation and coverage
of all our dependencies.
We want to run some targeted outputs and projects
in the projects that we'll be willing to do so after talking to them.
All projects, regular incremental process,
so we want to have it in the way that if we add new projects,
we don't have to review all of them,
we just get the new ones and then talk to them.
And we want to spread the methodology and findings to others,
so that others could do very similar thing.
I don't want to contribute to PSF,
maybe hopefully this will become a pipeline,
kind of tool or feature or whatever comes out of that.
Conversation with another project is Python based.
You might try to review this, right?
So, exactly.
So, we are repeating that.
So, what you can do, things that security is,
if you are an open source maintainer,
start similar efforts and do similar things,
know your dependency, talk to your guys,
talk to people that you work with,
you use, so, know your dependencies.
You might want to support similar security initiatives,
like if people ask for help,
like I helping security of others,
don't hesitate just help.
You can contribute security reports to us,
any other maintainers they will also welcome that.
And learnings, go, thank you.
I haven't seen this slide in a long time,
so I'm going to make up words.
I think I've already said this a few times though, right?
You as a consumer of projects,
and as someone who's being consumed, right,
are part of a human graph, a web of significant stickiness.
It is hard to make you go away.
It is hard to fix your problems.
It's hard to you to fix other people's problems.
If you don't have a relationship with the people
consuming your software and the people you are consuming,
you are a victim of whatever happens to you in that process.
And it will happen, right?
We see it every day.
And so your project security matters to people,
downstream of you.
If you're working on this because you're pay two,
then you should care.
If you're working on two because you do it for the passionate
and the love of the software, then you should care, right?
We are no longer in a world where people can write software
without worrying about these things.
We don't expect everybody to be a security expert.
That's not the goal, right?
And I would be the first to acknowledge that there's a lot of work
to be done to make this secure by default,
easier and so forth.
But it really does start with actually knowing
that you are in this graph and playing attention to it.
We have a few other small takeaways.
I don't think I'm even going to speak to this slide, right?
Human problem.
The risk is growing exponentially.
We have sustainability's part of this,
and everybody has to have this be a priority.
Okay, we'll take questions.
Do we have time for questions?
We actually allow it.
Thank you for the awesome.
Okay, let's start.
I'm just going to arbitrarily go over here.
Yeah.
How do we get those projects?
Can white be security?
So the question was, how do we get our project scanned
by the security sort of scanning processes that we have?
It's a great question.
And the answer is, look downstream into your dependence
and find one of them that has engineering resources and money,
unless you already have the resources and money
to get that work done.
Also, the rate of change in this space.
I mean, their team has been doing amazing work.
There's so much velocity happening in this space now, right?
Go look at the things that are out there and figure it out.
You know, and it doesn't have to be perfect.
It just has to be more than you're doing right now,
which tends to be very close to zero.
Again, I don't know your project.
I'm sure it's been great.
There was a gentleman here, yeah.
I'm just going to question, how do you expect
of the tooling as well, especially the dependencies?
So for now, we are just,
are we using specific tooling for it up?
There is a spreadsheet there.
So we are for now developing as a part of our project,
as a flow that that's part of breeze,
which is a development, release management,
and as bomb management tooling that I have.
But the end goal is to be able to produce similar tooling
for others to use.
That's what we want to do.
We want to improve that, iterate on that,
and then when it's going to be ready,
that when we are going to release it,
but basically it takes a number of signals from other,
from different places, like starting from as bomb,
going through the OSPF scorecards, and so on,
and so on, just putting it together,
it's a glorified Python script at this moment.
But yes, it will plan to make it kind of publicly available
and reusable, but it's three times more expensive
to make something reusable after you make it usable first for you.
So it will take a little bit of time.
I think we can.
I also hope to have some case studies written,
or work together to sort of talk about the methodologies as well.
So a lot of this is not rocket science.
You just have to do it.
I'm getting that sense that we have a room for maybe one more question.
Are there any more questions lurking in the back of the room?
Going once, going twice.
Thank you very much.
It was a real pleasure to be with you guys today.
Thank you very much.