Okay, hello, we are on time. Please vote for Mayba.
Fire, Ron.
So, I'm Eva's Hurricane New. I'm excited to be here, and for the next 20 minutes, we will
talk about open-source dependencies and how to monitor vulnerabilities. Before we start
up, she thinks about myself. I leave the product security to matter most, matter most
in case of some of you don't know, it's a collaboration platform for mission critical work.
I was previously adult zero, acquired by all the principal product security engineer,
they're living a few different places I'm currently in Athens, Greece. And with that, this
is our agenda. So, we'll start by explaining why third-party dependencies and their vulnerability
is matter. We'll evaluate different software. Well, we will explain a process to evaluate
different software composition analysis tools. We will see how to roll out such a tool
and finally, how to create the findings. The reason we have this agenda, it's because it's mapped
to what I call a dependency management journey or a secure, a secure dependency management
journey. Before we start talking about any SCH solution, we first need to understand
why we need to do that and we need to understand the risk that vulnerabilities in third-party
code carries. Once we can do that, we can start saying, okay, this is a problem we need
to solve and we can start gathering our requirements and based on those, evaluate some tools.
Once this is done, then we can start evaluating a tool, pick one, roll it out and then
do something with the findings. And we will cover this journey today. So, why do we have
this talk today? Why do we talk about third-party dependencies and their vulnerabilities?
I'll start by saying that it's important to keep in mind how a software is comprised. I would
say that in big projects, it could be half-and-half in smaller ones, the third-party code can
even be higher than the in-house code. What does this mean? It means increase the tax surface.
Now, there is a valid point to be made that any time you add new code, you add new
tax surface, either that's in-house code or third-party code. There is however one significant
difference between those two, which is control. And in case of third-party code, lack of control.
So, for your in-house code, you can control your security processes, a security review. You can have
static analysis tools scanning your PRs. So, you can be proactive around the code that you
ended up having in your software. You don't have the option in third-party code,
the other use it or you don't use it. And this is where the this increase that tax surface
comes into play. Now, it's also important to understand that managing vulnerabilities and third-party
dependencies is not just a security best practice, they carry real risk. And the best way to
remember is a log-for-J issue back in 2021, where I create a vulnerability in a library
actually called severe issues in software using this library. So, usually we spend more time
into our in-house code and the vulnerabilities what we call in our software. And we often like
make a second-class citizen, the dependencies, and their vulnerabilities. And examples like
LLK4J remind us how important it is to also keep track of our third-party dependencies and
their vulnerabilities. Now, without setting stone, before we go into evaluating different
solutions, we need to define what are our goals. And I split them into two categories,
proctivate and reactivity. And for me, these are the two pillars of a security programming
general. Again, if you think about your in-house code, you usually do some proctivate,
as you have a proctivate approach, security reviews, some security testing maybe. And because
you are also released, they can you know that things will get missed. You also have a reactive
approach, we usually call this vulnerability management. It's a similar process here. So,
for reactivity, what we're looking is it's pretty obvious. Address dependencies that the vulnerabilities
that we have from our third-party dependencies. And for the practice part, we want to make sure
that once we start working on this like building a process and a program around this, that we don't
keep adding vulnerable dependencies. We will come back to this slide a bit altered to explain
more in detail how we can do that. But this is something to keep in mind going forward.
Without said, let's talk about, well, I forgot to mention, SEA is software composition analysis,
I'll just be using SEA going forward. Why to evaluate such a tools for this solution? Well,
imagine like going over all your dependencies, all their transitive dependencies, and then
finding out if they have vulnerabilities. There are tools out there that can help us with that.
So, let's figure out what's best for us. Now, one thing that often, I think there is
we often do something which I don't agree with with different tooling. As we have a problem,
we go out, we find we're the different popular tools, open source commercial, I'm not going to
suggest any. And then we try to understand which fits better for us. I'm actually suggesting the exact
opposite so that we remove these initial bias of looking online, which is, understand your problem,
define and document your requirements and be clear about them. And then in the next two slides,
I'll actually show what kind of requirements these could be. Now, it also helps your evaluation.
If you assign them a label, this will help when you actually evaluate tools and you know,
well, this is a must-love. That's pretty nice. I could leave without it. And once you've done
what I call your homework and you're clear about what it is that you really need,
then you can start looking for different tools to evaluate. Usually, the requirements you've created,
you can answer them from the docs for the biggest part. So, you can do that and then I say pick three.
The reason I do, I say this is time and effort like restrictions for further evaluation.
Now, an important aspect, what we mean about evaluation. Another common mistake we're doing is
we evaluate the details, but not the core. What I mean by that is SEO solutions give us a
very hard promise that they can give us complete an accurate results about all the vulnerabilities
of all the transitive dependencies. This is a hard promise. So, when you evaluate, I would say
get your source code, include dependencies, ideally dependencies that have a transitive dependency
with availability and test if the tool will find it. Like, really test the SEO engine,
you can also test all the other requirements, but pay attention to that part.
These are some examples about requirements. I'm not going to go over them, but you can get an idea
the core goes to the spreadsheet with the requirements. Just a few notes, usually languages and
package managers are your must have requirements. Your organization uses some like package
management, there are some languages. If a tool doesn't cover them, it doesn't really cover your needs.
And what I have in the third one, it's zero to minimal findings in consistency. This goes
back to the robustness of the engine that actually gives you the results. You can have some
like more details depending on your requirements. And again, one common for the reachability.
It's very difficult to get it right. If you think that you need reachability, which means making sure
that your code reaches a vulnerable function or the vulnerable dependency, test it out.
So, we're always so far. We discuss what is the risk. We discussed what are our requirements
and what is a process to evaluate tools out there. Now, what are we going to do about rolling,
we selected a tool and now we're ready to roll this out. How can we do that? I'm listening to
different options here. The first is roll everything at once. What I mean by that is you can
integrate your tool with all the repositories and then start re-ausing. There is one caveat
if you have many repositories and they have many dependencies and many vulnerabilities,
you'll get a huge number. It feels chaotic. I've been there. You see where I'm starting from.
Another option is what I call a phase approach. I'm going to give an example here and then I'll
actually tell you what we did at a mile or most. A phase approach could look like the following.
You make a list of your more important repositories. Max5 is just a random number. You need to
again, like time box is set for it and then you can have maybe a second phase.
When you start the roll out, you integrate your first repository. You wait X amount of days,
X depends on how many efforts you can put, how many vulnerabilities it ends up having.
After that time, you can integrate your second repository and by that time, you make sure that
your critical and high findings are mitigated. You go on. Some factors that you can
consider to when designing those two options are different teams responsible for different
repositories. If so, you can do them in parallel. Yes, as a security team, the security team will
have some a bit more work to help the engineering teams, but engineering teams working on mitigating
the finding if they're different, we can parallelize that. How many vulnerable dependencies is
repository has? If there is a culture of regular maintenance updates and you have very few findings,
again, you can either group them in the phase roll out or you can just do it all at once.
All at once also means how many repositories do you plan to integrate? Are there a few?
Maybe it's okay if you do it like the all at once option. You have too many.
Somehow group them and then phase them, like do them in phases. And again, to be realistic,
was a team like capacity in workload both the security team and the engineering teams. So,
depending on those factors, you may go for all at once, a phase. We went for a phase, but not like
the one by one. We actually, like, stank ranked our repositories in terms of
priority and the most important ones. Then we said, okay, which team owns which repository we found
out that we can do some parallel work and we actually ended up having three small phases,
but if you think of phases is, well, not phase. It's actually one. It's a phase approach where
we did three buckets of repositories. Three works for us because of how our teams are structured
and how the repositories own to those teams of structure, but that's an example.
I'll go back to this slide, the reactive approach and the proactive approach.
Because this is when we roll out the tool and we need to think about those two goals we
discussed in the beginning. In the proactive approach, once you integrate, make sure that
use can the PR level. So, one thing is to scan the your main bronze and see, okay, I have this
dependencies, I need to fix them, which is actually the reactive approach I just said.
But for the proactive one, you need to make sure that you start working on what you have as
vulnerabilities, but you don't add more. And to do that, you can have a check at your PR.
It's easy if your manifest file has a change inversion. If it has, it calls your SEA tool.
If there is a vulnerable dependency, I would say fail to check and discuss, like, the security
and with engineering team and find this solution. Otherwise, you would end up
keeping reactive as more vulnerabilities get added. Now, for the reactive approach,
if you solve a phased approach, I said, I showed before, I said highs and criticals.
This is just for our risk-based approach. Like adding a few repositories and going through all the
findings, depending on the regardless of severity, means that you are leaving other repositories
with highs and criticals to wait. This is why here, it mentions a risk-based approach.
Now, for let's go to where are the phase of preusing the findings?
How to do it? I'm usually suggesting a time box investigation that aims to answer two different questions.
The first one is very simple. Is this a test dependency? If yes, we can skip it.
And the second one, which is difficult to answer and goes back to the retability discussion we
had earlier, is it exploitable in our systems? Is it exploitable dependency or even better the
vulnerable function of the dependency reachable from our code? Now, the second question can be
part answer, especially if you think of transitive dependencies of a N-depth, and that's why I'm
saying time boxed. And also, this is why I suggest that you balance the time between doing the
investigation and actually going and updating the dependency. Like, it may not make sense to even
do this time box if it's, I don't know, a minor version that you're going to update it.
Another thing to keep in mind is that updating is not always an option. Well, there are many
times where there is no fix available. What can you do there? Notify the maintainer. We are in an
open source conference, let's work with the maintainer, notify them and see what's their appetite.
Maybe they would like to work on, they don't have time contribute. Like, you can work on addressing
the finding, contribute, and then both your project and all the other projects will benefit from that.
Sometimes, maintainers are not responsive. Maybe there is an unmetane library. It's possible that
you may need to explore alternative dependencies, offering the same functionality. But these are
things that we can keep in mind. Now, slowly closing, how to do this dependency management
effort, like a sustained thing. Track the findings and track their resolution for two reasons,
first to make sure that you work on them. Secondly, because you give a signal to your organization
that those findings are equally important to your in-product vulnerabilities. Like, make them a
first class citizen. Whatever process you have for your in-house vulnerabilities, use a very similar
one for your vulnerabilities and dependencies. The second one is that once you do a live initial
rollout, you may think it's your done. Because if you think about it, I mitigate all the existing
vulnerabilities. I added a PR check, so no new ones, but wait, the existing ones, we may find
new vulnerabilities. So this reactive approach never really ends. The initial rollout is usually
the heavily-fting, but you will still get new vulnerabilities that you would need to mitigate.
And that's actually why it's good to have a regular maintenance schedule. This will also help
you a lot with those lows that is very difficult to get any buying to fake, especially their
dependencies. So having a regular maintenance schedule of updating dependencies in general across
your repositories will help a lot. So this is all the journey we covered. As you see, you can
do the first three phases once. The third one is a repeatable process. I have a lifting in the
pro in the beginning, but a repeatable process. And with that, some key takeaways, choose the tool
wisely. If you need to do some initial integration between your tool and your systems to it,
like in the initial rollout, the more integrated your tools are, the more easy it is for the
engineering teams to feel like to be better, like to see the tool better, like to feel it to
feel it better. The other thing is, what we discussed before, when you roll this out, have a clear
strategy for addressing the findings and for having this proactive approach. But our lymphocytes
before you do anything like have a plan. It can become stressful when you get all the findings.
And as you probably saw in this presentation, this is an org wide effort. It's not a security
team only. It's not engineering team only. It's an org wide effort. It needs time. It needs investment,
especially in the beginning. Get agreement, get buying that for the next few months, we will need
to allocate some time and effort in this initiative. That's all I had.
Seven months for questions. Regarding reporting to a tool like zero, I have made some
observations first, the nature of the findings. I mean, to yourself see me numbers.
And really being slow, et cetera, the very often replaced by a OSB or a DSB first.
But so there was a dynamic nature of those findings. And then if you report to zero in such tools
like a fire and progressive approach, it might happen in the complex environment. It might happen
that people talk about various identities. If in case you don't keep those tickets dynamic
and the second thing is if you have a real complex application, it pays off to have a look at.
So vulnerability tickets, which have a supervisory approach. Both management and developers,
because they can see how far some teams are already fixing things. In case it's not trivial,
both from management perspective and let's look which colleagues we can ask for help.
Yeah, that's true.
And I think one thing that this reminded me is,
Girish tickets should not be a fire for a get. They should come with some sort of SLAs.
If you have some SLAs for your in-house product vulnerabilities, maybe you can relax them a bit
again because you need a cream in from engineering, but don't forget them.
Yep.
Two small questions is, what is your opinion on the state of the furnace available,
but not the needs to be part of the digital state of the furnace on relatively diaspora
to the degree of the automobile industry and dependencies. Also, have you considered,
you can see open it is a scorecard based use of dependencies in the strategy.
With, I used to, like a year ago, I was in the NSSF one working group.
With the evaluated scorecard, we haven't had the time to roll this out.
For tools and retability, I'm very, very skeptical. I'm yet to find a tool that does what it promises.
Yeah, it's a very difficult problem with retability. I can understand direct dependencies
and retability. I'm yet to become convinced about the transitive ones.
That's a good question. They often don't understand the risk.
So they don't understand why they need to spend time on this.
When you have an in-house product vulnerability, you have steps to reproduce,
you can, like, they can actually see things happening. When there is a transitive dependency,
especially the transitive, we're proving their risk is very time consuming for a security team.
Making, like, persuading and engineering teams to work on that, it creates friction.
And the other one is, if there are no, like, major upgrades is a friction.
Like, if it's a minor or a patch, usually people are fine, when you tell them to upgrade,
like, to two major versions happen, they need to do loads of code refackering, they don't like it.
Or, if there is no available fix, so they need to go and see alternative dependencies,
and then maintain it as we said, is not responsive. So they need to go final alternative dependencies.
This means that they also need to do some code changes. In any, in any, like, the more effort,
the more friction. Because the risk involved is not that clear as with a in-house vulnerabilities.
I mean, at the source, like, say, like, hey, you know, get talk and go talk to them,
like, maybe go to use so much for the sort of, you know, a fair part, you look for a little east version of that.
Is that even a gadget? Can you repeat the question?
Well, I'm so sorry, I'm so sorry. As I get my gadget, I've had a chance to, maybe,
um, thank you for what fancy to use? Yes, this is a very good one. That's a very good point.
The question is that addressing vulnerabilities and dependencies in what is one part,
but shouldn't we really educate engineers or on what dependencies to use and what not to use?
And this, actually, I think goes back to the scorecard where, like, the open is a set,
the OSSF1 where vulnerabilities, like, dependencies, you get a scorecard depending on your, on different
metrics. They're higher, the, like, it can be maintained, um, it can have security advisors or
many different factors in there. So yes, this is a very valid point. This, like, dependency management,
security dependency management is one part, uh, but educating what kind of dependencies you introduce
your code is, it's, it's, it's equally important. Yeah, that, thanks for the point.
Yes, uh, so the, if understood correctly, the question is,
at matter most, um, how many people need to be involved in this process that I described.
It's definitely our team, which is the product security team. We do the trius,
we at some times do also some heavy lifting, like, if there's no fix available, we may do some
research on this and the, again, the maintainer is not responsive. What else can we use? We,
some, and because we're, like, we started the, we did a more systematic approach to this a
year ago, um, we do some initial work. So it's the product security team. And then we do regular
maintenance of dates. This helps a lot. Um, and the other thing is the way we, uh, our engineering
team operates, usually there are specific teams for specific, specific repos. And within those teams,
there is usually a person who's has taken, like, the, the ownership of, of taking the
dependant. The good that, um, there is less context switching for the whole team if there's
one person. So the most fun thing to do, uh, let's, that's how we do it. Okay, let's thank
you.