7,000 enterprise applications and IT services, 60,000 servers and containers, and the same amount
of cultural positries, and several hundreds of thousands open source components that we
rely on in our supply chain.
And for every one of these components, we need to know where it's running, how it's used,
in which version, and so on, and in which context.
And how are we going to do that?
Well, probably with an expected amount of 500,000 S-bombs created and rich then analyzed
per day.
That's the size of the program that a few colleagues and I have been conducting for the last
two years.
Yesterday, I already gave a presentation on how we set up this program, how the strategy
worked, how we did that.
So I would cover only a small part of that.
And today, the focus is rather on the tooling.
So what are we using to scale these operations and to get to a state that is good enough
for us?
First of all, who of you knows, I know, otherwise, who of you doesn't know Deutsche Bahn?
Thank you.
No, thank you.
OK, so I won't talk much about this.
The big thing, or the important thing is the organization is large.
It's diverse.
We have 500 different professions, 200,000 employees, and so on.
So at the same size, it's our IT, more or less, like with the complications, and also
the different scenarios that we have to think about here.
So I will cover a little bit about that later.
But most importantly, Deutsche Bahn is not a software company.
So our business is making or running trains, and somewhat maintaining the railway
infrastructure in Germany.
Talking about IT now and the scope of this talk.
If we think about transparent supply chains, we have to also think about the diverse
streams of software, how software is coming into our organization.
And this is building software, of course, and for the most different applications, like
on many of your phones, probably, but also on web servers, but also on embedded devices
somewhere sitting in between stations on the railway tracks or somewhere else.
We also buy software, obviously, and we operate software, not only in the cloud, not only
containers, and so on, but also on edge devices in operational technology.
And if we have to challenge that we also know which component is used, where we have
to consider all these sourcing streams.
So how can we find a common language to describe the composition of all of that?
I mean, we are in the S-bomb, it's of course S-bombs.
But we don't see S-bomb as a tool, we rather see it as a common methodology to describe
software, not only to describe it, but actually to do something with it.
So I won't go through all the use cases that we came up with it, and this is only
subset.
But this is just more than license compliance or security vulnerabilities or CRA compliance.
This can also be more forward-looking.
How do we understand and analyze our whole supply chain to be in rest engaged somewhere
strategically or to be in rest from somewhere?
So S-bombs must become a shared infrastructure.
But again, it's the only about S-bombs, it's about also thinking of these systems in
these more open methodologies.
And one important part of that, which is perfectly integrated with this Vex, also mentioned
a few times here already, the vulnerability exploitability exchange.
As a great way to cope with a challenge that we will have and already have, the more transparency
we get into our supply chains, the more frightening findings we will have.
Especially from the security point of view.
And Vex is a great opportunity for teams to not double and triple their rub on describing
whether they are factored or not by certain security vulnerability.
That may not even be exploitable in their specific context.
So S-bomb Vex, but also S-bomb signing and so on are all both in flux that we are trying
to get into our organization.
And you can imagine given from the size that we have, that this is not an easy task.
So yesterday I talked a bit more on how we set this up and with which principles I will
not go into detail, but I would just like to highlight one or two points here, because
we as a small group that started this, we set up a few principles.
And one of the most important principles was to not talk and tools, but in capabilities.
So once we started to leave out the exact tool names that we had or tools that were on
the market and started rather thinking in systems, the whole picture became much clearer
and we could also think big.
On the other hand, on the technical side, we said, well, we have to consider all the different
sourcing scenarios.
We cannot only look for built-espons of our pipelines.
We have to think about something that works for every workflow and there are many workflows
within our company.
So what was really helpful for us is our mental model of an Espa lifecycle.
And this is the simplifying result.
So you will see on the left the different sourcing streams of Espaums, really showing where
do Espaums come from, from the most diverse systems, from the most diverse subsidiaries
that we have within our company and other raceic vendors and so on.
And the ideas that these Espaums enriched on a baseline quality are then from decentralized
sourcing coming into a central Espaum database, the Espaum inventory, how we call it.
And this is very important here.
We cannot live in the world where we have multiple Espaums storages.
We need to know in one place with one query what we are using where and how.
And given the centralized nature of the Espaum storage, we can also which with rather
further information that cannot be described in the Espaum itself, like in which IT enterprise
architecture, the application is the Espaum actually used, who is responsible for its content.
Who do I have to call out at night from their well-deserved sleep if something goes
really wrong and like what's the team responsible for it?
But then in the other hand if I look at the analysis then of the Espaums, this again is decentralized.
We are not having a single UI to put in all the dashboards and so on.
So you can imagine this Espaum inventory is like a database with quite performance APIs.
This is the simplified version that I'm just showing the system more complex things, showing
the streams of Espaums and Vex information, going through our organization and systems.
Again, this isn't tools, I mean you could also write some tools here but in the end
is systems.
You don't have to take a picture of that, these slides I think are already uploaded and you will find
that. So how do we translate all of this into reality?
It's clear that this cannot happen overnight.
So we need to have this step by step and our strategy here was to first focus on Espaum adoption
rather than Espaum quality. So we wanted to know fast which software components we are using
so we look for low hanging food, we look for priorities for instance set by the server resilience act
and start with getting Espaums for all source repositories and for all pipelines where the developers
onboarded this. And in the course of this year we will add the Espaums from runtime so from our
service, physical and cloud service and the containers. So 60,000 roughly and also following up
from operational technology. I mentioned that earlier you can imagine as an infrastructure owner
we have a lot of OT that is running. I have been running since decades and it's probably still
running for decades but we still need to know what IT is running there. Now let's get into tools
and more concrete. We looked at this overarching architecture before. So what we came up with
or what the idea was is that we also create Espaum set for a certain degree of baseline quality.
And this baseline quality has to be good enough to be working for our use cases so there aren't
perfect. So we came up with a default modular tool chain you would call it like a default
process that developers can just lock into the pipelines for instance or that is automatically running.
And the steps are similar to what Victor mentioned earlier like we come from a generation of
Espaums over in enrichment of Espaums for instance regarding a license or a licensing information
to an analysis of Espaums regarding security but also regarding license compliance. So for
instance checking it against licensing policies that we have internally based on the usage scenario
and but also for creating for instance copyright notices based on that. On the horizontal
you see like that we have internally and external information systems. So from internally of course
we need to have some context information but I go go through step by step. So how do we generate
the Espaums? Let's talk in tools now. Our default tool chain and this is important. This is like
the default. This is modular like developers can choose something else if you think that is better
for the ecosystem. We generate an Espaums currently using sift because they do a really good job
and integrating with various ecosystems and doing some post processing. Then we find and off the
Espaums especially from the licensing point of view they rely on clearly defined. That's a
great project also been mentioned a few times you're already it's a basically a database of already
scan packages using scan code. So we don't do not have to rescan all these packages with
scan code looking for copyright and licensing information because given our size of the supply chain
that will literally burn the planet. Then for the analysis of the regarding the security we are
using KRIP also from Ankara and connecting this with our RECS information. To be honest this isn't
perfect yet so KRIP and we are also contributing to make KRIP better with RECS information but
we have this RECS pretty good and for instance that already had a really good effect that can
I can say that when we had the RECS to shell vulnerability a few weeks months ago we were able to
identify the effect of products within our companies 12 hours before our proprietary
very expensive security scanner said we had elsewhere. So this is doing a great job it's not
perfect but it's well good enough in order to respond to such things. To check against the licensing
policies we are currently using KRIP also from Ankara also not perfect but we are also
contributing to it and working together with Ankara and regarding the copyright notices or
how we call it like compliance artifacts generation again clearly finds us a great candidate for
doing so we are not calling here the the public API by the way about having set up an internal
clearly defined instance that again was the simplified version now comes to complex one
again look at the slides but this is really again it's talking in tools of course but it's
more or less systems and workflows like how do certain steps have to integrate with each other
how the flows of RECS and S-bomb information which systems internally and externally are effective
basically or need to be included and what do we have to yeah still work on in order to
get this whole picture working for all of our engineers. I said this tools read how we call it
it's modular and the idea was to have a really good adoption from our developers so this default
2 chain it's working in pipelines we have internally a product called pipe shift created by a
great team by us coming up with a good lap templates for their CI and there's just a one liner
that adds all of this magical many of these magic steps creating the s-bomb enriching it and
storing it into our central s-bomb inventory and it just needs I don't know two or three rare
others for configuration but it's also important to note that not all products within our company
and our organization are using pipelines yeah they are good reasons to not do that or where it's not
possible so the idea here is that it's also available as binaries as home proof packages and we have
a home proof tab for that by mason top and mason plus and other functional so developers can basically
integrated in all their workflows that they have for instance also on an s-bomb that has been delivered
by a vendor and they want to increase the quality because they do not directly have a leverage
to do so now we have a okay good s-bomb it can become better and we're working on that but what to do
with it right reset it's going into our into our central s-bomb database but how do I look into it
and this is where the compliance read comes into play here the compliance read is an internal service
that we have it's like rep your eye but also with an API and that helps teams with understanding
their assets better so and their compliance stages compliance cannot only be security findings but
also other things that we have internally but it has a direct connection to the s-bomb inventory
it helps the developers and teams but also service owners to understand their assets better so it
can work on single repositories on on GitLab groups but also on assets that can spend multiple
groups and multiple repositories so this is a great way this service also crawls through all
our GitLab repositories for instance and accepts the s-bomb that are either coming from itself
or fetched or was sent to by pipeline integrations and other steps and technically this is
a plug-in to backstage does anyone of you do backstage by the way a few hands 30% create so
we are also working well together I can only recommend it we're using it internally as a developer
portal so teams can do much more with it like setting up new repositories and whole projects quite quickly
regarding s-bomb I mentioned already this compliance read can interact directly with the s-bomb
inventory it's basically more or less the same database or close to it so developers can directly
see the s-bomb that they sent to it they can also manually upload it different for instance
get it from a manufacturer and they can also inspect the quality they can re-upload or download
the s-bomb so that it works quite nicely based on these s-bombs we mentioned before
there are also findings we have security findings of course like from the scribe scanner running constantly
on all the s-bombs finding on an up to date security or vulnerability database rather there are
findings and for the licensing the same applies so we have license checks integrated and so
developers can see what's the status whether it complies with my own licensing policy or with
the company's default depending on the usage scenario of the project or the product over there
the idea here is that we have a machine readable and come close to an automated compliance
so not only throw warnings at developers but also prioritize these warnings and that works quite
well already technically I think the last slide on the compliance suite how it's working
from the architectural side I mentioned already the compliance read itself it's a plug-in for
backstage that is doing much more than just these compliance things the compliance within instead
has this runo or the service that quads to get lab and also our edge of DevOps and on the deployment
side how this is deployed of course Kubernetes and with a big support of cross plane and so these
are really important open source projects that we rely on and also contribute to now we have all
these s-bombs and findings what do we do with that and here I would like to showcase one of these
use cases that we have if we have aggregated information over our supply chains and this shows just
two stats of the many that we couldn't can generate first is front and frame works so we can quickly
see overall what the Deutsche Bahn are we using as front and network what are the rising stars
rare as technology that might be hidden champions do we for instance want to strategically use
only one or two frame works instead of a whole mix of it or do we are we using front ends
of frame works or also programming languages that we actually do not want to use anymore because
they are deprecated also and so this way having this aggregated information a rare federated
organization we can make better decisions so s-bombs are not only good for security compliance
and so on but also for forward thinking investments into ecosystem and the understanding
I would like to show and now we are running out of time a few stats that are just compiled a few days
ago very roughly I went through over 80,000 s-bombs basically it's natural of a second 80,000
is the current size we are having more s-bombs coming in basically every second that would
place old s-bombs and yeah interesting numbers here I mean I found that we are using more than 100,000
open source components so without versions if we add versions and architectures this will be tripled
on factor 5 I think so this is great insight that we have and the questions how do we actually
act on these stats at the end and the important part here is it's people so we have to take
people with us not only developers not only operators but also those who are as responsible
for security like governance owners so make it easy for them integrate into their workflows
consider their different ways how software is developed and operated that's very important and
surely governance owners and those who run these projects like I do our important part
yeah but we have to take people with us and these developers so concluding
s-bombs are not a product they are a mythology and therefore it can only be implemented incrementally
it has to be pleasant for the users and it has to be modular we cannot expect a single tool
to solve our other issues so internalize this knowledge into your organizations do not externalize
it do not rely on single manufacturers for that and let's collaborate not only on the tools
but also on how we set this up with our organizations and communities so thank you very much for the
you