Hello, my name is Artham. I'm here with my colleague Johem. We are working at
Claiser, helping people to run safe. And we want to present our case study of declarative
object storage deployment using Surf and Rook. And I will attempt to deploy it in one
click. So yeah, the more about the project. So the case study is based on a peer
project. It is SAP project and funded by European Union in the scope of digital
sovereignty initiative. All work from this project is open source and will be
donated to Foundation. Our main goal in this project was to develop a blueprint for open cloud
to enable governments and organization to round their own cloud to host their own data
and let's rely on hyperscalers. So what do we mean by blueprint here? So it's like a distribution
or collection of open source project to host their own cloud. So all projects, all selected
project should be actually maintained. This means that if some project and it help with my
tenets with contribution, we help this project in the scope of a peer. The blueprint should
be seamlessly integrated. So all selected companies should be tested in production and
real application work together smoothly and it should be reproducible. So others could actually
save time by using this blueprint. Deploy it with minimal human intervention and for us
reproducible means that it's have to be declarative. So that's why we selected to use Rook for this
project. And of course then this blueprint will be created. It should be also maintained.
So we will have to check all the latest releases and versions of all selected
companies and continue to test it to be sure that everything still works. So this is some
number from our key study. So we had a chance to have 100 server deployments to actually test
the reproducibility part. So the project started six months ago and it's not finished yet.
So we deployed three or even four clusters right now. So our first setup was not fully declarative.
Some of things were still done using CLI tools and so on. But our latest cluster was actually
deployed in fully declarative manner and by the cluster, I mean, in a safe cluster with the
ratus gateway and object store. And the size. So yeah, if we aim to replace like to be alternative
for hyperscalers, we should also test for scale. So we should make sure that all selected
companies as scale will. And our starting size of the cluster is 100 nodes and one thousand
OSDs. And the plan is to also scale this cluster until two or four thousand OSDs.
Yeah, so about technologies. So I told that the scope of the project is bigger than just object.
So it's to provide cloud and for cloud part, the open stack was selected like the most popular
cloud software stack. And for us here today, a relevant following project from the open stack.
It's a keystone for identity and access management. Swift is the open stack.
Object storage implementation and also interface. Barbecan is responsible for key management.
Cinder is block images for VMs and the Manila is a shared file system.
So yeah, it's quite common to use it together with open stack to use it as a storage layer for
everything. So normally you use Radus Gateway or HWM to replace Swift,
CFRBD for block images and also on a CFFS. So CF was selected like as the best open source
storage for which scales really really well, which make sure and used by many companies in production
already. So yeah. And of course, since we want to have a reproducibility and want to make it
declarative, we also have a rook here and rook is a Kubernetes operator for CF. So this means that
you also have to use Kubernetes to run your CF cluster. So CFD runs as a puts in Kubernetes cluster
and orchestrated by rook. So you have object storage custom resource to configure Radus Gateway
and you also have a CF continuous storage interface to provision CF volumes to Kubernetes cluster.
So yeah. So in this talk we will focus only on object storage for now. So the idea was to replace
Swift with Radus Gateway and continue using like the same Keystone and Barbie constellation
with the same users without touching and breaking everything and see how it works. So yeah,
so we should have this intent. So part one is about CF and OpenStack. So the most important
thing you need to know here is that OpenStack Swift is eventually consistent storage and like
CF. So a CF and all of these companies being object storage block devices are strictly consistent.
So this means that in OpenStack Swift you don't have read after write guarantee but in CF you have
for us it was actually a really good thing because if you expect people to move from AWS who
using S3 already they expect objects storage to be strictly consistent. So yeah Keystone integration
and Barbie can and the system implementation exists in CF for a really long time. More something
like a 10 years already and S3 implementation in CF is really amazing. So when you move
from AWS let's say to CF you can expect that everything will work. So it's really feature each
and stable. But we discovered a couple of issues with applications which we were working with
OpenStack Swift before and we can say that Swift implementation is a bit less popular and so
less mature today. Yeah but anyway for our new cluster the customers we were already using
edge W Swift without any problems. But we had some problems to migrate like existing application
and I will explain exactly where. So you can download this presentation. I put links to all
issues which you found you can go through it later because some of them are really minor stuff but
I put everything just to use later. So I will highlight a couple of topics. So for example
public buckets or containers, CF containers, we are not working at least with the Keystone integration
in the fix was done by community quite recently and was backported to leave. Also we noticed that
we had application which was using A sales and when we started like edge W Swift with the same
Keystone we had the slightly different owners ID in ASL. So it might be important for everyone but
yeah it's broke one of the apps in testing environment. Also speaking about ASL so in Swift you can
have role-based ASL. So for example you can figure some custom role in the Keystone and in Swift API
you can tell like okay this role has red access to the bucket. So unfortunately this thing will not
work with edge W right now. Also as three conditional rights it's quite a new feature introduced by
AWS. So it's quite new and allow you to write on conditions so if objects already exist on
or not exists. So it's also not supported by edge W but I think it's soon will be done.
Yeah so another topic which we discovered so it's probably not very popular use case but for example
open stack Swift also has some subset of ST interface and there is an apps at least in our
environment which are working with both. So for example you write to Swift bucket and read from
S3 and something like this and you expect that they are pointing to the same storage everything
to the same only interface is different and it's also working with edge W when you put object into
Swift you can read it from S3 but not always. For example if you enable a server setting
encryption in S3 and put some object and you will try to read it with the Swift interface you will
get encrypted data so it's not readable and the same is true for versioning so if you enable
versioning with S3 in edge W it won't be enabled automatically in the same Swift container.
So the reason for that is that in edge W Swift and S3 implementation not share the code too much
and they are a little bit different and the goal of our project is actually to improve it
and make Swift like in par with S3 and work together better. Also we had a small problem related
to water and usage so in edge W you have a pool placement and target placements it's like a
equivalent of S3 storage classes so for example there is a idea that when you create a bucket
or put objects in a bucket you can tell S3 which storage you should use like this object goes to
SSD this one to SSD and in edge W you can track usage and water per this storage class you can only
do it per user and the bucket yeah so now yeah beside this everything worked really well so
we already have fresh clusters with new applications who use it and we have planned to like
migrate this to one. Now let's move to Rook so I mentioned that from the start we will not able
to do configuration declaratively but now it's possible so how serdeployments with Rook is
looks like so you have Kubernetes cluster with the nodes with machines which will be your
safe nodes you label this node with typology labels like there they exist in your data center like
row, red, blade and so on you declare device filters in in Rook cluster configuration because
if you don't do this Rook will try to discover all devices on the nodes and yeah and can
you're that in OSD so if you don't want this happen you can you can use filters so then you start
and cluster it's up and running monitors always these managers are here and then you can also
declare OSD pools with desired replications and you can also declare your vegetable instances
using this pool so yeah in our case everything in Yamal for the latest cluster we deployed
and what we had to fix to be there so we added target placement support because before it was not
possible to configure this target placement aka storage classes with Rook we had to do this manually
with the safe admin tool by editing zone and zone group JSON and there is also now support for
multi-instance GW deployment so we're using it to have like separate configuration for
a GW instance to sort of Swift API another one serving history and we also have
admin ops instance which is not accessible outside for admin API yeah and yeah we also work in progress
about allowing to combine internal and external resources for example having some monitor as
the arbiter outside of Kubernetes cluster and we also looking forward to observability
improvements there is a lot of going on in safe upstream already for example if you upgrade to
speed you will get a lot of new metrics for a GW for example latency user and per bucket
and yeah so the scope of the project is a couple of years so to finish it blueprint to make it
better and our plan for this year is to maintain edgew Swift implementation because we got a
bit from community that there are a lot of people working on history and not many people are working
on Swift so we want to really help here and make it in part with instrumentation we want to start
discussion in community about water and usage management improvements to support target placement there
now when we have cluster we want to test data operations we can scaleability and also do
improvements if we find that something could be improved we also want to investigate like data
migration topic if you come in from cloud intercept and you already have a lot of data there should be
a way to migrate your data yeah develop ideally without downtime and yeah the most important part
why we are here to make this blueprint work we want to build community around it and we want to
have other case studies so if someone is interested to implement it in your company
probably these two early because we need to finish it a bit and open source and then we can connect
and test it on different environments and make it better and if you interested about this
a pair of project initiative there will be another talk tomorrow it's about cloud native APIs
and integration across multiple providers for the same issue so you can also join but yeah I think
I've finished with the slides and if you have a question so we are happy to answer
so please hope that they show them for the world of work and I'm sure I haven't understood why
we involved open stock with the root thing that is for Kubernetes I mean we could have done
set directly with the root could be not involved in Swift or my computer wrong
yeah that's true but sorry yeah the question was why do we have
open stack at this slide because yeah if talk about having reproducible semi-planet so
the scope of a pair of project is to provide like the full cloud installation including
VM provisioning and other stuff and open stack was already selected to be part of this project
and if you already using for example user management and encryption stuff from open stack
and it is supported by seconds should be integrated yeah you should use it together and
integrate and yeah actually I forgot to mention that Keystone integration was not supported by
Rook until this summer so if someone was missing it please give Rook another try so now you can
also fully configure Keystone integration with Rook yeah and the one of the requirements was
that it should be completely declarative and following the cloud native way so I think that was
in Swift VM at the moment it's not supporting this and that was one of the reasons why we have
to chose Rook at this point yeah so do you use any visualization for the Kubernetes
cluster or do you dare to install the Kubernetes on the parameters of the system?
Yeah yeah the question was a regularization technology to provide the machines for Kubernetes cluster
so yeah it's on bare metal now yeah but I think the plan was to also use Key VM yeah oh no I think
for the set nodes or Rook nodes we have a separate Kubernetes cluster and this is because of
performance reasons it's running on bare metal if you want to run this in virtual machines it's
also possible but at the larger scale I think it makes no sense if you if you have smaller classes
or you want to test it I think free to test it but from my experience it's adding just another
layer of complexity so and I could talk about this I think multiple hours why I would not do it
thank you yep
yeah the question was have you considered using Chef ADM and make it working declarative way
yeah please I mean at this point Chef ADM was in a major refactoring phase it was also the
idea to think about okay at this feature to Chef ADM when the approach was started was the route
just a wrong timing for it the other topic of Chef ADM at the moment is or was how it's handling
or upgrading the nodes as well especially if you want to upgrade and how it is checking the states
of the whole cluster at each point I think I was also not the biggest fan of Rook at the beginning
I mean first time I was in 2018 when I got in touch with the team but I think the higher potential
what you also can use from Kubernetes side I think Rook is somehow yeah it started to convince me
as well that the Rook is perhaps yeah that we can use way more other features from Kubernetes and the
other CNCF projects at the end so but we as from outside it's like this we are just supporting
both so we are also using Chef ADM it's always a question back to the customer normally what kind of
technology stack they want to prefer and what kind of team knowledge they also have so you can
okay okay what kind of this to use for the cluster at the moment it's cost so we have at the moment
we have pure NVM ease that's the one set up and the other one is mixed modes or the
B1 wall on NVM ease with LVM partitions and the data is stored on normal rotation and this also with
LVM partitions can you just repeat the question if you have the question what if you have
if you have faced latency I mean we are doing this now I think since 12 years so we know the
caching topics of HDDs and and so on yeah we have seen this but we are directly
gifted wise to disable caches on HDDs so we have also a more intensive also included in Rook
an intensive test up front so we where we can identify all the devices and also make sure that
that a single device cannot slow down the whole cluster so yeah so there's more upcoming
observability is one of the big topics for this year as well and also data improving data operations
in general and that's not only focusing on your Rook it's also we are adding new features to
and also bug fixing stuff so please okay if you are using Rook for multi-cluster things
what kind of multi-cluster can you so for pull replication not now at the moment
no so just for my understanding I mean pull replication we are talking about the object storage
for multi-site or do it okay okay I'm not a big fan of multi-site so for my eye texture point of
you I always try to have no strict dependency on the cross different zones or different clusters
we have experienced a multi-site we also did some deep dives over the last past years
especially multi-site with new for in combination with features like versioning and others
it's always you always have to test everything up front if it's really working for each future
that they are trying really using the future and it's and it's a degree relationship at the end
it also cannot answer your questions like recovery point objective recovery time objective
and for this we had a talk I think two years ago with chorus so that's something else on top
another layer above how we replicate data across the world or across data centers are very
availability zones okay
do you see any benefits of using kind of EM for you all of this because I
they find the same type of use all of these all of the EM okay do we see any benefit of using
LVM for the OSDs I mean as far as I can remember if you use the raw then there was a problem
that you sometimes get at high devices detected LVM we use LVM for encryption because all the
disk are fully encrypted I think that's that's the major benefits data benefit yeah at the end
we are working we are trying to use the most common setups that we see and that's LVM with
ZF volume and another thing is also that we are also fixing back sometimes in ZF volume that I already
thought that they are already fixed 10 years ago when we use ZF deploy yeah
because the crucial mostly because we tried to do the ZF on the raw drives and the
roof and the basic works out okay out of the box yeah and then we switched to LVM because
that's more natural solution yeah we can manage and we had a disk of LVM problems so
the disk of LVM was not discovering the LVM drives so I am sorry yeah
okay we can talk about this later after this okay but it's a channel feedback on LVM and
RUK and so on yeah I just wrote a long documentation how you can replace a single damaged disk without
re-installing the whole node that's also an improvement that's perhaps artable implement and I just
provided the documentation from this and that's exactly they have to be careful in this area
when you use LVM because it's really it's different implemented if you provide the
raw device or if you already the provided LVM device that's a pre-configured okay
hope that helps so
okay okay thank you
you