So it's, oh, we have to wear this.
It's great to be back.
We see not standing front of the slide.
So one of our marketing managers calls it the capital crime, not to introduce yourself.
So that's me and one slide.
I spent, I don't know how many years in product management must be 20 by now.
And they're all in open source products, almost all.
And before that, I was an engineer and a better engineer.
And I was the maintainer of Manhattan, as you see at the bottom of the list of infamy.
I wrote a book for a Riley, and that's why I'm surrounded by clouds in the picture.
And that's kind of the short version.
Hi, everybody.
I'm Sage McTaggart.
I use They Them pronouns, and I work on cybersecurity at IBM.
I did my undergrad at UMass Amherst, and graduate school at UC Santa Cruz.
I've done research in a lot of areas, ranging from programming languages, files,
systems, tons of other stuff.
And I've worked on security for Suffat Red Hat, and now IBM.
I also work on incident response for Safanodef at IBM.
And in my free time, I like to hike with my dog and garden and work towards a better world.
So we're going to skip the interest lights because you know what Safin is.
It's great, believe us, and Rook is even better.
That's just go to the security points.
So they raise expensive in so many ways when things go wrong.
It's costly to collect, curate, storing it, ensuring regulatory compliance.
And success is not optional.
Failure to meet these requirements results in big fines at the least.
Out of the spectrum, and it's loss of customers, loss of confidence.
And services going down at the worst end of the spectrum.
Actually going out of business would be the worst end of the spectrum,
but let's not be too melodramatic.
I like this interesting point at the end of the slide, which is, you want your systems,
so that you are not the easiest target of those available.
That goes for almost every product, and surprisingly, especially for an open source vendor.
So many of my customers pay for upgrades, and then they don't install them.
But anyway, so we're going to go over what are the best practices to secure Saf.
And then also look at what's the newest, what's changed recently, and what we have improved.
And here is a little bit of an agenda of what we're going to cover.
So, track models.
Security practices harden a specific point of the infrastructure.
We have a security guide that's linked at the end,
but this is a bigger point about how you use such a security guide.
Cherry picking security practices without the model of the threat,
and of the attacker, is not a viable strategy.
The usual joke goes that if you want to protect from all possible threats,
you need to turn the computer off, bury it in concrete, and then drop it at the bottom of the mariana trench.
In other words, absolute security is not very usable.
It needs to be defined in the context of the threat model.
Are you facing script-key these, or the GRU,
or the dry that privileged insider, is your share point administrator,
siphoning off all of your files.
These are very different threats.
Some of these want to steal your data.
Others want to crypto-lock you and hold you for ransom.
Others may be just satisfied creating disruption.
You need to define what threats you're protecting against,
and pick your battles.
If everything is a priority, which is something that some managers like to throw around,
then nothing is.
So, focus in the resources you have, where they can make the most impact.
With that we've been moving beyond a simple inventory of regards points and hardening those,
thinking of future threats to our software.
This underlines our principles with security design.
We're also working on general improvements,
a key part of this is ensuring that we do not deviate from standard protocols.
We do this because we trust the community, such as cryptographers,
to have the best practices, or at least practices that are better than our ideas.
Don't write your own crypto.
Think you probably heard that one before.
Let's now talk about one way we build security into our design,
and recent improvements we made here.
Here is the network layout, so how the different security sections of the network look like.
The public security zone is an entirely untrusted area of the cloud.
Could be the internet as a whole, just networks external to your cluster,
that you have no authority over.
Data transmissions crossing this zone should make use of encryption.
Note that the public zone, as I just defined it,
does not include the storage cluster front end,
which in SEF is confusingly called the public underscore network,
which defines the storage front end and properly belongs in the storage access zone.
The SEF client zone refers to network networks accessing SEF clients,
like the object gateway, the SEF file system, or block storage.
SEF clients are not always excluded from the public security zone.
For instance, it is common to expose the object gateway's front end to the liver,
the SEF API in the public security zone.
Next, the storage access zone is instead an internal network,
providing SEF clients with access to the storage cluster itself.
Finally, the cluster zone refers to the most internal network,
providing storage nodes with connectivity for replication,
heartbeat, backfill, and recovery tasks.
This zone includes the SEF cluster back end network called the cluster network in SEF.
Operators offer unclear text traffic in the cluster zone,
relying on the physical separation or villan separation of the network from all other traffic.
This would not be a valid choice going back to our threat model discussion.
If your threat model includes adversarial privileged insiders.
So again, what is the threat that you want to encrypt your internal communication
because somebody can go and tap your rack that should be part of your model.
These four zones are separately mapped or combined depending on the use case and the design of your specific cluster.
Components spanning the boundary of two security zones with different trust or authentication requirements must be carefully configured.
These are natural weak points in network architecture and should always be configured to meet requirements of the higher trust level of the zones connected.
Many cases the security controls should be a primary concern due to the likelihood of somebody probing to escalate from a lower privileged zone to a higher privileged zone.
Operators in general should consider exceeding zone requirements at integration points, which for a storage product is often easier to accomplish than for let's say compute platform.
For example, the cluster security zone can be isolated from other zones easily because there is no reason for it to connect to other zones.
Conversely, an object gateway in the client security zone will need to access the cluster security zones monitors OSDs all of them and will likely expose its S3 API to the public security zone on ports 80 and 443.
So this one is talking to three different zones at least.
Some engineers on the team have worked to improve the dashboard security bringing it all within a single security zone.
So the change here is that previously only, only authentication to the dashboard was secured and in theory you could have gone and tapped the information, the monitoring information of the cluster from previous or graphana.
Now everything all of the web sockets are secured uniformly and there is a reverse proxy in front which has other engineering benefits that are not security related.
And that is the biggest change in this kind of in the network part of our security practices.
Yeah, that about sums that one up. It's been a fantastic improvement to the dashboard security as of squid.
Going on to our next slide.
We recently moved well three years ago almost from Red Hat to IBM.
And you know the security of a product really it depends a lot on who's maintaining it.
Some open source products are not maintained by anybody or there's one person.
They maintain it in their basement in their free time.
That's fantastic, but that isn't necessarily practical for an industry or any sort of case where we're very worried about security.
So I'm going to discuss some aspects of IBM product security and how that's been going, especially with the implications for open source and upstream staff.
And also a little bit of compliance as open source since a lot of compliance models are not designed for open source.
I'll discuss some of our accomplishments and what we're planning on doing going forward in the next year.
So product security at IBM, we follow a secure development lifecycle.
This is the, this is done with the goal of reducing risk and improving security for both South and and Rook.
We suggest improvements, we pen test, we manifest all of our dependencies, we review vulnerabilities and track weaknesses and exploits.
We check every new release for these things and we just generally are trying to iterate and do better.
We don't release with known criticals unless we get an exemption, which is pretty rare, that reduces the exposure to different CVEs.
And as a result, we're improving upstream staff because all of these get ported back to the non IBM staff.
We still work on secure design there and we've worked also with IBM to implement open source principles by improving things like their call home feature.
And we're working with IBM research on how to implement TCMs and where open source quantum resistant algorithms might go.
So one thing that we do a lot of is we manifest and document dependencies within South.
We automate security scans of our code and containers.
We work heavily with upstream communities and this has sort of been a change since moving to IBM to bump dependencies and ensure that fixes are made for all affected projects when possible.
The scanning benefits the entire open source community because we'll say, hey, Kephanna, how's your go-line version and they update it and they're pretty good on their CVEs.
So we work together with different open source projects to make sure that they're getting updated whenever we notice something.
This enables us to be better contributors to the open source community.
And these dependencies are need for upgrades they cover the vast majority of our CVEs which is a little bit of a pain but updating to the latest version and automating that as much as possible and making as easy as possible has really helped us.
So it's really hard to automatically update your dependencies you're going to run into build issues.
Yeah, sure, depend upon on GitHub says update your dependency.
You do that, it might start breaking things, you might have to refactor your code, et cetera, et cetera.
Especially if it's been in our case, here's since we've upgraded some of the more obscure dependencies.
And I'm sure all of us are in that boat where that's happened where we notice we have to update something and if you have thousands of dependencies, that's what happens.
So what we do there is we're trying to reduce our attack surface in the components that have known CVEs with redesigns like we saw with the dashboard.
And we're also applying what I'm going to call a humination approach, human automation, where we create a very clear, simple, almost automatic process at every release saying this dependency.
And I'm going to blame Grafana again, even though they're a lovely project, they're wonderful people.
We're going to say, hey, Grafana has a lot of our CVEs that get reported to us.
Let's update Grafana every release.
That way we're only updating between a few releases at each time, we're following a clear process, we know when we're doing that.
And we start with our worst offenders, and then we continue updating all of them until they're all relatively recently updated and we have clear processes.
And that's been a major improvement that we've done.
Whoops, sorry.
So now briefly going into executive order, one for zero to eight.
This was done by President Biden back in 2021.
Scanners sometimes, sorry.
Sorry, let me get on the right side of my apologies.
Scanners oftentimes only detect CVEs from dependencies.
This was an executive order that basically required us to start manifesting and documenting our CVEs.
And also all of our dependencies.
So we can debate what is actually required, what isn't, but for our purposes, we need to know all of our dependencies and all known vulnerabilities and update them.
So we use scanners and they detect a lot of CVEs.
Open source has a lot of CVEs in a lot of cases, and that's a good thing.
We know our CVEs.
We ran these scanners on closed source code.
You would find even more CVEs.
But you tell somebody, hey, you can't have CVEs or you have to have a plan to fix all CVEs or you have to evaluate it all CVEs in all of your dependencies.
You run these scanners on them.
They're going to give nasty results.
And you're going to discourage people from using open source.
So how do we resolve that with this emphasis on scanning that gets all of these?
Incident responses usually focused on are you vulnerable?
Do you have a proof of concept?
Is there an exploit?
Well, we really need to work on redesigning things to reduce our attack surface, so we have time to fix these.
We have to look at things like the vex scoring.
And we have to update dependencies to reduce the burden later.
So as we know all of our vulnerabilities from these scanners, as opposed to just what we're guessing our vulnerabilities, we can say, let's redesign and fix this.
For example, at IBM, they have a feature called call home that basically helps them provide IT support almost and provide telemetry and other data metrics.
So it was not open source.
It was not designed in an open source way.
The security model originally required closed source code.
We were able to use our incident response skills to create a more secure one advocating for open source in the project.
Those both redesigned to be less vulnerable have a smaller attack surface.
And we were able to do that because we were aware of our potential vulnerabilities from all of our incident response work with open source.
So you really want to use code that's up to date, but all this scanning has really helped us and brought issues to light and given us ways to fix it.
So again, as part of this, we eventually want to start automatically updating dependencies.
But as I've touched on before, that can present challenges creating communication is definitely a very viable way to go and what we're doing.
And we've done things also including redesigning our upstream CV e-reporting to account for different processes to just try to keep things flowing in a very clear systemic and easy way.
That reduces the burden on the people behind the project.
And what does this matter? Well, we might have government customers that need this.
We might have corporate environments that say we don't want to use poorly maintained code.
And also, to really want to use code where your dependencies are years out of date, you probably don't.
So again, this automation process has really helped us a lot.
We do run into issues with things like transitive dependencies. Let's say we're using a dependency that uses going.
And I'm just going to use going as an example, because whenever they do a release, they're fixing CV e's and almost all cases.
Well, a strict reading would say we have to treat that the same and handle the automation accordingly.
So that's where we're trying to make sure that we're a good partner with the broader open source community when our scanners find this.
And isn't just hidden in corporate IBM land, but instead we're saying, hey, this found this.
Are we updating this? How can we work on this process?
And this has really helped us a lot, especially for a very small security and build team and made it a lot less manual.
And I think we're a lot of open source communities we're relying on volunteer time. This helps really reduce our burden here.
And just the processes make it just so much easier on all of us.
So now let's briefly talk about how Seth does encryption.
Server side operators overwhelmingly choose to encrypt data at risk using the Linux simplified key mechanism.
Lux all data and metadata on a storage cluster can be secured using a variety of DN crypt configurations.
And almost all of our customers choose to do this.
It's possible that somebody doesn't, but we recommend it. It's standard.
We also enable a security best practice by locating our mons on separate hosts from the OSBs.
The centers to anti affinity of the keys and the data they encrypt.
So if you see a one drive, you're not getting all the information and the key.
The objects for gateway has additional capabilities, including encryption at ingestion time, the use of per user keys as opposed to per.
Drive keys, key rotation with tools like vault, support for AWS.
SSC KMS and more. We also use Department of Defense 140 ciphers when supplied by rel appropriately.
And we also encrypt our data and transit.
We do this by turning on the Seth protocol communication for messenger version 2.1 with not allus.
We, we note that a lot of people might want to use clear text as a compatibility reasons.
There really isn't much overhead, so that's not necessarily the reason, but if you have a legacy.
If you have a need for the legacy clear text protocol, some people do.
We allow that. We also obviously encrypt all data at rest if you choose to.
And we're working on redesigning our data in transit model for quantum resistance as algorithms have been now published that are recommended by nist.
We'll be seeing what we can do there.
Again, we have the client and public security zones. We have TLS security for the object gateway to us three clients.
The TLS termination at HA Proxies a special case.
And that was recently done with squid to ensure that the link is in the appropriate security zone.
And then, of course, with the network standard practices like firewalling individual nodes,
to expose only clear list of ports supplies.
So now going into rook specifics, rook can use CRD's custom resource definitions along with several other options to encode many of these features,
such as configuring the trust certificates for the rate of skateway web server or GW.
Rook supports at rest data encryption as discussed earlier with the in-flight stuff protocol encryption as an option in 1.9.
Kubernetes permission system applies to our persistent volume.
So permissions, quotas, all that comes from Kubernetes.
Nothing rook needs to do here.
Rook supports, of course, key management systems.
And this allows individual volumes being encrypted with their own key.
All of this basically results in making it really easy for us to follow best practices such as key rotation,
replication, and limiting scope for each key.
This limits the scope of unencrypted traffic, and this limits the scope of each key.
So going briefly into the control plane, as popularized by Ansible, SSH is used by Suff admin,
Suff Ansible, and other deployment and day one tools.
To provide a secure command line path for install and upgrade installation as part of host management,
we do this using the dashboard so that the dashboard is not usually exposed to the world,
and yet it can still be accessed by the operator's workstation to be abuse.
We recently, again, did the redesign that puts it all within one security zone for the dashboard,
and the manager also then supports the whole infrastructure and needs to be reachable on the storage access zone.
Hence, why we're using SSH.
So we're always trying to improve our cryptography.
We are not cryptographers.
We can read stuff that cryptographers are doing and think that it's fantastic though.
So some things that we've been doing is we've been talking with IBM research about open source quantum computing and
confidential computing.
We're currently determining the viability where we plug things in, et cetera.
We would use open source code that from the open source quantum project,
and we're documenting all the places where we use cryptography.
Seeing if it's quantum safer, we've done the initial assessment that basically determined,
hey, we're using AES, that's safer, but it isn't what is recommended.
So now we're going forward with that and also seeing where we could plug in things like TCMs.
So our priority here have it all be open source, use open source libraries.
Don't have it become proprietary to be quantum safe.
We're working with upstream to discuss the correct use of cryptographic modules,
such as AES.
Of course, we use AES, but how are we implementing it?
What type of model are we using?
Should we use GCM, different models, et cetera.
So we're working with our upstream community to ensure that we're implementing,
according to the Kerbros model, not just rolling our own ideas of where we should plug in AES.
We're tying that in with the need to upgrade dependencies automatically,
because you need to upgrade your cryptographic libraries too.
They can have vulnerabilities. They are an infallible.
Anybody can put a bug in code anywhere.
So talking a little bit about identity and access,
Seth's use of shares keys prevents mint from clusters from having
man in the middle attacks by default.
You still want to have good practices, such as key ring, read and write.
You want to only have those for your current user and route.
And your client admin user should be really route only.
You don't really want all users to be route.
This prevents that, which is the best practice.
Talking briefly about RGW.
It supports the key and secret model of AWS S3,
and the equivalent model for OpenStack Swift.
Again, we want to treat that administrator key with appropriate respect.
Use administrator user sparingly to reduce your risk profile.
Your RGW user data is stored in Seth pools,
which should be secured again as we discussed previously regarding users at risk.
We can couple with OIDC providers such as key cloak,
which you can back with your, you basically can just use SSO here.
With your organizational IDP, or again, just use SSO.
And we support LDAP for identity and access.
We recommend security LDAP along with options to use SSO as needed.
And this has sort of been a recent improvement in Seth.
So we're using SSO more.
Just to standardize it and keep it basically per user.
And one state has deleted from a Seth cluster though.
We do have some practices around data retention.
One state has deleted from a Seth cluster.
It cannot be recovered for practical use.
There's some exceptions.
RBD has a new facility called Trashbin.
And again, in RGW, there's some specific cases where
the versioning of store buckets may result in deleted objects being preserved
until their purge per policy.
So again, configure your data storage pools accordingly.
So again, there's a lot of things that you can do there to just keep it easier.
And to sanitize your retired medium, you want to encrypt your OSD contents at
rest and then destroy the encryption key.
That's probably your quickest way to do it.
You can definitely handle it all their ways, but this is the quickest way.
And it works.
So going into infrastructure hardening, the hardening options are very
vendor-dependent.
We have some choices here with the red hat modes here.
We have SC Linux, of course.
We have PIPs 140 slash 2 support.
We definitely have our certified cryptographic options
to import this in FIPs mode setup.
And, you know, relate.force most recent certified version.
There's sometimes a delay.
So, you know, we use that version when needed.
We have some hard and binaries.
We have additional kernel or OS supplied hardening as well.
We have a few links for you about things that we have referenced or
indirectly mentioned.
You can get from the slides when we distribute that.
Yep.
I'm sorry you had a question?
Yes.
You did not talk about the set of S.
What's the ACL option?
What's the set of S that you had?
Excess control this set of S.
That's the security topic for me.
Yeah, we didn't go there because they generally are our standard
posics.
We don't really diverge much there.
I don't think.
So, the question was, what are the access control lists in
SAPFS?
Yeah, yes.
Yeah.
It's just standard.
All these permissions or public's ACLs,
or the best before ACLs is PIPs.
For example, that was at ZFS.
Posics.
Okay.
Posics.
There's nothing.
Go ahead.
Who checks the posics?
Actors?
Is it the safe client or is it the safe server?
I believe it's the client.
And sorry, the question was who checks the posics ACLs?
Okay.
We need one of the developers, and I don't see him here.
But, oh, there we go.
Patrick, go ahead.
It's the client.
All right.
Outside.
Thank you.