Welcome everybody, just two quick announcements. Please try to say for the talk. It's
a fairly disruptive one. Someone in the middle says that. If you have questions later,
give it to questions. Discussions can happen after them. Let's give it up for our
first month's F4. Hi everyone, my name is Riga. I hope I'm audible. My name is Riga. I'm currently
working with IBM. I'm a part of SF and ODF team. And with me, I have Weshnavi. She is also
the part of the same team. So today we will be discussing about how we can make
newbar more resilient by eliminating the single point of failure. So I would like to know
how many of you I have actually heard about Nuba. Any hand. Okay, just go ahead and do. Okay,
yeah, that's great because I have included introduction to Nuba. So yeah, great. So yeah, so this is
the agenda for the today's topic. So we'll firstly go through the introduction to Nuba. We'll
then go understand the Nuba architecture and then we'll see how this single point of failure
plays role in the Nuba environment. And then what we can do to mitigate it. So what is Nuba?
Let's try to understand that. Nuba is a software defined storage platform that is used to manage
your object storage across like diverse environments. So what do I mean when I say diverse
environments, they can be any cloud public cloud platforms such as Azure, AWS, GCP, or it could
be any S3 compliant private cloud as well. It could also be any storage drives that you might
have. It just needs to have a volume and a file system on top so that they can connect to Nuba.
So what does this do is it makes the users job a lot easier. You can have your storage
spread across different environments and you can manage them at the same time at the same platform.
So Nuba has bucket policies. These are just the rules that tell where you should put the
data like you can define these policies by yourself. They are customizable. So these policies
they provide these multi cloud capabilities like, you know, replication, mirroring,
tidying, and name space. And the goal here is to again, like I said, to simplify your job,
you can manage your data across diverse environments using the same platform.
So why you shouldn't use Nuba? It is again, it's a multi cloud solution. So this eliminates
the vendor lock in, you know, you're not tied to a single vendor here. You can have your
multi clouds. You can also have a on-prem, self-cluster as a storage back in for Nuba.
You also can use your regular storage drives as well. So it's not, you know, tied to a specific
vendor. Again, it provides a S3 compatible API. So whenever you put your data, you put it through
your S3 API. So it's a standard, you know, de facto standard API. So again, not tied to any specific
vendor. Again, this increases the high availability of your data. You can have a single bucket
mirror across different platforms. So let's say one of your back in storage goes down. You still
have your data available at some other place. So you still have your data. Again, it's a storage
orchestrator tool. So it can, again, manage your storage. It consumes storage from diverse
environments. It also provides bucket policies that are highly customizable. So you can
set rules where you should put your data depending on, you know, your requirements.
Again, these, these are very highly dynamic. So a lot of times our project starts with one set of
requirements and then it ends up with something else. Your architecture changes, your demands might
change. You may start with a block storage, then you might just start requiring objects storage,
your platform requirements can also change. So in that, in those scenarios, these bucket policies
are very helpful. You can just change them dynamically and use them. And again, we on top of your
existing storage layer, we also get compression, de-deplication and encryption. So this also
adds a security, like added layer of security. Again, we also get agility in Nuba. We can migrate
the data without any disruption and services. So, you know, in imagine, you have a setup that you
used, you know, you built it on-prem. Right. Now, your client requires some cloud involvement in
that setup. Now, you have acquired cloud license, you have cloud storage, but how are you going
to migrate the data? In Nuba, you can just use a mirrored bucket. You can, you know, whatever
your existing bucket, you can just mirror it. Your data will be mirrored directly to your cloud storage.
Now, you can get rid of your on-prem bucket, that you can also do. General workloads for objects
storage, they can be media backed up or archival type of, you know, workload. This is just a generic
thing. Oh, can you do the next? So, this is just a basic overview for Nuba, like how we usually
see Nuba when you talk about your client and backend sort of a thing. So, you have your applications,
your applications will directly talk to your S3 APIs, that Nuba S3 API. Your Nuba S3 API does have,
you know, it processes the data. And in the back and you have all those storage, you know,
back and such as AWS, Nuba will, it could also be a safe or any multiple different location,
you know, your backend storage can be. This is the Nuba platform, when your client puts data
in the S3 API, the Nuba S3 endpoint, it will process the data, depending on how you have configured
your bucket policies, it will put the data across these diverse, you know, backend. So, for
example, if you have multi-cloud buckets, your data will be spread across these multiple clouds.
If you have hybrid, it would be on-prem and some other cloud and multi-site, it would be on
different multiple locations. So, for the scope of this topic, we are talking about Nuba
deployed on a Kubernetes platform, because that's where, you know, most of the agile workload,
you know, we have seen customers deploy, right? So, next. So, this is what happens to your
data when you put it in your Nuba S3 API. So, this is just a example of Mona Lisa, the picture.
So, you put it in your S3 endpoint, it will firstly get in fragmented, we fragment the data in
4 MB chunks, then it gets de-deplication, you know, if it's there is any repetitive data, it will
be eliminated, then we compress the data for better storage utilization, and then the encryption
takes place. Once you encrypt the data depending upon how you have configured the bucket
policies, it will be stored across different environments. Next. So, these are the basic core components
for Nuba, you have Nuba core, you have Nuba endpoint, and you have Nuba agent. Next. So, this is the
architecture, we will talk about the data flow here, we have your application writing data directly
to the S3 endpoint, then we again here have Nuba core, Nuba DB, and Nuba agent at demon.
So, your data once it goes inside your S3 endpoint, it will do all the heavy lifting,
it will, you know, compress the duplicate, encrypt the data, then it will send the metadata to
your Nuba core, Nuba core will then process the metadata, you know, generally metadata contains
the bucket details, where you want to put the data which bucket, and what bucket policies that
bucket has. So, for example, if I have a multi-cloud bucket, I am putting data in a bucket
B1, and it is a multi-cloud bucket, that information will go to the Nuba core, and Nuba DB
stores a lot of bucket related metadata like where your objects are stored, what bucket,
they are stored, and what is the bucket policies, all the configuration related details.
So, with the help of Nuba database, Nuba core will calculate where to put the data, and it will send
the instructions back to the Nuba endpoint, and then Nuba endpoint will actually store the data
in your back end depending on where it is. So, once the data is stored, it will send that knowledge
meant back to your Nuba core, and then Nuba core will update your Nuba database, and, you know,
the cycle that is how it is completed. We here we have, you know, scalable Nuba storage node,
which basically is a Nuba agent, it is a demon that keeps track of statuses of all these components
in case your back end storage is not available, it will try to, you know, think that and make
sure that it is available if it is not, it will market us down, and yeah, basically it keeps track
of it, and updates the status, again it goes back to Nuba core, and sends what all data,
you know, and I will fix they have gathered, and it again updates to Nuba DB with the current
statuses of the, you know, the whole cluster, and, you know, again cycle continues.
This is the data path for Nuba architecture, again Nuba DB here is, as you might have seen,
it is very important, it is a core component for Nuba, because it contains all the bucket information,
such as configuration, it contains account and access keys, bucket policies, again it contains
the metadata related to that backing store and cloud connection information is also there,
mapping of object to bucket, right, where my each object is stored at what location, that is also
stored inside your Nuba DB, it also has encryption keys, it also has system status and statistics,
so yeah, you might have gotten an idea like how important it is, so without this Nuba DB,
we cannot, you know, access our data, because again, all the access related information is stored
inside here, so what we can do to, what we can do, you know, to understand how we can fix this
situation, in case something goes wrong, right, so we have Vaishnavi discussing what we can do.
So, till now we discuss the architecture of the Nuba, why we should use the Nuba and what is the
workflow, now we will discuss the understanding and single point of failure in Nuba, so as we discussed
earlier, Nuba, Nuba metadata, Nuba DB is very important, because it knows the location of the
object, where the actual object stored in the background, whenever we write any data, it is metadata
gets written in the Nuba DB, so it has to go to the Nuba DB and when we want to read the data
from the back end storage, we need to know the location, so again we need to go to the Nuba DB,
to know the location, to read the object, so Nuba DB stores the crucial metadata, every
of some Nuba database storage is scalable to handle the increase load and ensuring the flexibility,
so at time passes, our storage might get increased along with the metadata, so if we want to scale
out the increase the size of Nuba DB, we can do that, we can just increase the size, so that's
what flexibility it is, so as mentioned, every operation of the Nuba DB has to pass to the
database, so in case of any failure happens, it leads to the disconnection between the data
and the client IO, which results in the system failure, again having repeating, if the data
base layer is corrupted, it's obviously going to impact our overall Nuba cluster, so how you
how you will identify the Nuba DB is corrupted, so was from the client side, client is unable
to write the data or unable to read the data from the cluster, and then and also if the client
is unable to create a new object claims from the cluster, so this is what customer will observe,
but from the system administrative perspective, we will see that Nuba pod is fails to come up,
how we will troubleshoot this kind of issues, so if Nuba pod is not coming up, then we can just
start with checking the describe output of the animal, and then we can check the events of that
specific namespace, in the namespace, in the events, if we observe a specific found error on the
device, specifically on they are BD0, if the back end store register, then we will have RBD,
and could not found, then there is a possibility of the database corruption, in the second scenario,
if the Nuba DB pod is failed to start, we will observe these kind of messages in the Nuba DB
pod log, that is could not start server, and from the database log, we will see could not
locate and valid the checkpoint record, so this scenario will observe if the node get crashed
unexpectedly or rebooted an expectedly, if all Nuba pods are up and running, but still object
bucket claims are not getting created or still claims are impacted, then we can check from the
backing store perspective, if the backing stores talk in a connecting phase, and if our devidum fails
with this kind of error that is invalid page in the block, then again there is a possibility of
the data loss, there might be more possibilities, but these are some general scenarios that we
have seen till now, so this is how we can identify that Nuba DB is corrupted, no one wants
are cluster in that scenario, so let us see how there are few ways we can mitigate this
single point of failure, let us take a look one by one, so first way is using an external
post-grace query, in an generally in Nuba, we use a single instance of Nuba PDC, but the external
post-grace query cluster, use Nuba DB as it provide the database lever replication, so in case
of any corruption happens of database, we will still have another copy available, so we will not
lose the access to the database, so we can use the external post-grace query cluster to avoid
the single point of failure, external post-grace query cluster also supports the SSL, let us take
look the prerequisites if we want to use the external database cluster, first obviously we need
to have the valid working post-grace query, accessible input for the external point, second
the empty details where we can store the Nuba DB's metadata, and for that we can use this command
to create specifically with this LT code locate, third one is we should have correct user name and
password to create a retables in the database, so in this method we will going to clone the
Nuba DB PVC periodically, so Kubernetes provide the feature called cloning which is pointing
thing copy of the data, so if we create a clone of the PVC in the back inside, it will create
one more PVC and it will store the point in them copy of the data on that PVC, cloning supports
CSI driver as well as dynamic provisioning, so we can clone the DB PVC periodically to take a
backup and whenever require we can just restore it, so we can to do that we can create Kubernetes
job in the Kubernetes cluster, so basically it's system administrator responsibility to decide
how many jobs they want per day and their retention period as well, because whenever as we are
creating a clones it will also going to consume the space in the cluster, let's take an example
if my current Nuba DB is 50 GB and I'm three backups are taken per day, then it will be 50
to three, 150 GB data will require to take a backup and also if the back end storage is
set then replication must be considered, so keeping all this thing in the bin we need to decide
how many backups and their retention period as well, let's take a look to step to restore the
DB using the clone, as we need to clone the DB PVC periodically to have the up to date backups,
for restoring we can just scale down the Nuba DB state for set, delete the older corrupted data
and we need to identify the latest healthy clone because that would be obviously before the data
corruption happens and then we can create a new clone with the original PVC named because whenever
we scale up the Nuba DB set a state full set it will look for that specific PVC name that's why
we need to give the original PVC named and then we can scale up the Nuba DB state full set
and we can check the port status it should be up and running next, so in the previous method
actually we took a backup of the whole Nuba DB PVC but here in this scenario we will take a backup
we will take a dump of the database, so in case of any failure happens we can just restore the
Hindi Nuba DB dump from the back of folder let's take a look at the steps how can we do that
so steps to restore the data, we should take a backup of the all Nuba related secrets first
to the local folder and then back of the post-gay school database to a local folder,
these just an example come on, we need to go to the Nuba DB port first and then we'll just
pj dump and be core to take a backup and as we took a backup inside the port we need to copy
from to the local machine, this is copy come on and if any corruption happens for the restoration
we need to stop the Nuba DB service before we restore and then verify the all components are 0
and then obviously we need to clear the older corrupted data and then restore from the local folder
again the back of folder is on local machine, so we have to copy that inside the port and then
we'll just run the pj restore command so it will just restore the healthy data from
that dump to the database and then we'll again delete the older corrupted secrets and we'll
restore them from the local folder and then we can destroy to restart the Nuba service it will
be up and running, so these are three ways that we can follow to avoid a single point of failure
in the Nuba, so anybody has any question?