The last speaker, and we returned to open segment, like we started.
Okay, thanks for joining my sessions, I'm Sonsu from South Korea, and I'm working at the
Energy Cloud based on the South Korea.
So today, I'm going to talk about my experience about the operating the open-stack shift
in the public cloud last eight years.
So as you know, the open-stack shift is a very old open source project, but it is very
difficult to find a production use case or document in the Google league.
So I hope that my presentation will help the others to operate in the open-stack shift.
So first, I will talk about the watch challenges based in a public cloud running the open-stack
shift.
So as you know, the public cloud can be used by many people.
So we can receive a lot of requests and a traffic that is unspectable and cannot control.
So if our open-stack shift is a internal service, we can execute the client to reduce
the request when our service is very getting a performance degradation.
But in the public cloud, I cannot execute the client to reduce your request to our service.
So yeah, that is the number one challenges in a public cloud service.
And then the second one is maybe something only for my productions.
Suppressing the leather than the largest size files, there are a lot of similar size files
in our productions.
The smaller size means a hundred of the kilobytes or the few megabytes object.
So it becomes a very important thing and it becomes a problem in a later on.
And then the third one is some client using our object service as their storage backend.
In their own storage service.
For example, the cortex is one of backend storage of the promitus.
The cortex also supporting the S3 API.
So if one client is using the cortex as the backend, they are the promitus.
They can stay setting up the backend storage of the cortex in our object storage.
And the last one is a very important and a very difficult thing to learning objects
storage in the public cloud.
We should be most ancient or stable service.
Even if the data, the Lebanese needs occurred and some nodes are failures,
we should ensure the stable service.
In the Swift, if the slum, some nodes are failed or adding a new node in existing cluster,
the data, the Lebanese will be occurred, but this operation will impact the service trapeze.
So it is very difficult to control the service impact when you're adding the new node in a cluster.
So let me briefly introduce the what is the OpenStack Swift?
The OpenStack Swift is the one of the initial project of the OpenStack.
So it is originally used for the backend storage of the glance component.
The glance component is storing the cloud operating system images.
But unlike the other OpenStack component, the OpenStack Swift does not have any dependency with the others.
So you can deploy your own OpenStack Swift to media out any others, OpenStack component.
So as far as I know, the Twitch, which is our live streaming service,
they using the OpenStack Swift to ask there are video storage service people,
not sure that these days.
And there are three concepts in the OpenStack Swift.
The first one is the account, account is a thumbnail concept in Swift.
So the account is just a namespace for the containers.
And the container is a namespace for the object.
So the container does not store the object physically.
It is just a logical namespace.
And the object is the rear object, the binary data things.
And then the account and the container is handling with database.
So since the Swift is supporting the HTTP base, the last API,
the basic structure is UI format.
So account select container, select object is a format of the UI.
And then when we are using the Swift,
most people using the Keystone as their authentication system.
So the account will be the Keystone project ID.
And then normally the Swift adding a prefix in account.
It is the OS on the back.
So you can see the OS on the back test,
the user will be the full name of the account.
So let's talk about the nodes that banks up the Swift.
There are the four nodes in the Swift.
The first one is a proxy server, the proxy server,
handling the incoming, the user request,
and do something additional features like authentication,
loading, encryption, like that.
So and then the account server and the container server
is a database server.
But it is not really like a database server like a bicycle.
It is a web service.
But internally, the account and container server stores
each data into the SQL, like the three database file.
So yeah.
And the last one is object server itself.
It stores a real object in every disk.
So this is the configuration when I set up a new Swift cluster.
So for the account and container node,
it is very important to ensure the very fast performance.
So I am using the SSD as their disk.
And those servers don't need too many disks.
So I normally using a four disk.
So two for the account and a two for the containers.
And when you configure the object node,
it is very important to setting up the memories.
Because when the data remains to all the data
replicate to the other node, the object server
using a lot of memories to their disk cases.
So this is my rules, not to everyone.
So I configure the one gigabytes of memories
for every one type of disk.
And then the minimum load is maybe four,
because every Swift data must have a three-ply care.
So at least the full node can handle the failures.
And then it is very important to ensure the consistent disk
size across the whole node.
Because if some node has a big or large size disk,
the data can store both disks or large.
So to ensure the consistent disk usage,
to ensure the consistent disk size is very important.
But in before the eight years ago,
it was my first year to operate in the open state Swift.
So I tried to setting up all these nodes
into those one nodes, but because we don't have any body
suitable configuring the node, but every year,
the sub-scoring of an attribute are increased.
Accounting container servers are counting container server
became a bottleneck.
Because the object server's process
spent a lot of CPU times during the service,
so accounting container process cannot
process in the userly case.
So after that, I decided to separate the accounting container node
and object node in our production.
OK, so the next one is a network architecture.
So to operating the Swift in a production,
you need to set up a three types of network.
So service network can be accessed from the public,
and handling the incoming user trapex.
And the data and the replication network is a private network.
It cannot access from the public.
So the data network is handling the userly case
to internally, and then the replication network
is only for the transferring the data replicating.
Let me talk more about the service network.
At first, I was using the Apple hardware switch
or the ADC switch to load balancing the Swift proxy servers.
But the service traffic getting increased,
the Apple switch doesn't carry on the trapex.
And it is very difficult to scale up the bandwidth
in a production hardware.
So we decided to change the load balancing method
to the HP proxy.
So in the Swift proxy, I put the HP proxy
as a software load balancer, and each HP proxy
has a virtual IP, which is managed by the capability.
And so when the client replaced our service domain,
the DNS returns the GSA B2 main and the GSA B returns
the one of the virtual IP in our production.
So after that, I can scale out our load balancer whenever I want.
So the next one is a scale out of the object nodes.
So the first concept to understanding is a ring.
The Swift is the basic concept of the consistency
of the ring, which is common in this distributed system.
And the open stack Swift has their own ring algorithms.
So the ring is a file, and this file is an extremely important.
So the all nodes in the Swift must have the same content
of the ring files.
If one node has a different ring files,
maybe the disaster will begin soon.
And there are four components in our rings.
The first one is a partition power.
The partition power is a vector,
determining the total number of the partitions.
The partition is a physical location of the where the data stored.
So the when we set up the number of the partition power
is 10, the total number of the partition will be the 1224.
So it is very important to setting out right partition power
when used create the ring.
And the other things I will tell you the later.
The next one is a device list.
The device list is a array of information
about this information like IP and port and a device.
And each index has their own device information.
And Swift using the device list create the device look table.
It is a two-dimensional table.
So you can see the columns.
The columns is a total number of the partitions.
And the rules is a number of the replicas.
So when you create the ring, the Swift spread out
the device list into this table very evenly.
So when if the number of the replicas three,
the first three row will be the primary node.
So that this will be the location where the data stored.
Since the primary node phase, the handoff node
is a temporary designated in the ring to store the data.
So we need to predefined the back on nodes.
So the back on node means the handoff node.
But the important thing is that the handoff
is not completely the spare node.
So you can see the disk index number one in the handoff node.
But it is also the primary node for the partition number 0, 1, 2,
in here.
So in Swift, every disk operate actively.
But one disk can be a primary node for the some partitions
and can be the handoff node for the other partitions.
So it is very important and difficult
to the managing the usage of disk every day.
Because if some disk phase that Swift
replicate those data to the handoff node.
So yeah, it is very important to managing the usage
of the disk.
So we are sure that this object should be stored in this table.
The process is very simple.
The first approach is over, get a request from the client.
And then the computer and developer has
you of the request you are right.
So this is the hash value of the data you are right.
And I take on the modular with the total number of the partition
like this.
And the neutral to weave is the partition number
of the dead request.
So if the value weave is the 140, the data weave is stored
the disk index 193464.
Yeah.
And physically it looks like this.
So if you look inside the mounted disk,
you usually the partition directory inside the
mounted disk.
So this is the where the data stored.
Then how to add a new object to node in the Swift.
So actually, there are one more items in a device list.
It is a weight and weight is a proportion of the disk
in the Swift push.
So the larger weight means that this can store
more data than others.
So I normally setting the weight as the disk side
in the database.
So if the weight is at 4,000, this disk
physical size is a 4 terabyte.
So adding a new node means the adding disk
information to the device list.
And the newly added device must be placed in a device looker
table.
That means that some disk will be assigned to the partitions.
That means the data migration will be occurred.
So data migration from one node to the other node
spend a lot of the disk IO and a network three peaks.
So the open-stack Swift demand using the replication
that talk to reduce the service traffic in degree
two to avoiding the impact of service traffic.
And this is the how the Swift's link looks like.
You can see the device ID and IP and port and the weight
and the how many petitions in each disk in here.
So I heard someone's, yes, the someone's told me
that it is great to be the, you should,
the 100 to the 150 is the minimum size of the petitions.
So you can setting up one, one disk can has less than 100.
But it becomes a nightmare when a sub-strippy
is getting increased and a pull is getting adding a new node.
So 150 is a minimum size of the partition.
So as you keep adding nodes,
you will even eventually reach a limit.
So for example, if you setting up the partition power as a 10,
the link creates the 1204 petitions,
that means your cluster can have only 1204 disk.
Because one partition cannot spread out the multiple disk.
So yeah, so the one partition is one disk.
So, so to solve this problem, the open-text
Swift provides the increase partition power features.
But this process creates the handling for all objects.
So if the one partition has a lot of objects in a single partition,
there are a lot of disk I use.
And the total time to increase the partition power is a
maybe the one month or two months.
So it is very important to set the partition power correctly
from the beginning. But if you're unsure the what value to choose,
the you can use this calculator, just put the number of the disk.
And the disk calculator, let you know the what the partition power
will be good to your cluster.
And this is an experience I had a while adding on your object node.
So even the submission to this utilization and the
replication network is very low.
And we have enough bandwidth in a replication network.
But the replicator replicates the object very slow.
So after analyzing the cloud, the wider replicators
replicate the object very slow.
So the I found that if a single partition has a huge number of the
file, updating the file that stores that hash value for the partition
took a lot of times.
For in my case, the each partition contains the tens of millions of
files, very, very similar files.
And then the processing of single disk with the 100 partition
took about the three days.
So if you're setting up the or 10GB pairs or 4GB pairs in the
replication network, but open text waves to cannot
using the whole bandwidth of that because of this problem.
And then the next one is a container database, three copies,
mismatches the container database stores the object list and
contain the information and the container database is also
has a three copies for the replication.
And this database file is a SQL legacy.
And each the SQL legacy database file will be created each
containers.
So this is the process of updating the object in container
the first the proxy server is the uploading the object
and storing the object in the object server.
And then the sending updating the object in the container
asynchronously and then the container server also updating the
number of objects and a size in the account server
or synchronously.
So why this happens?
I don't know why, so because the Swift must ensure
all of the three copy database data is the same.
But I don't know why why this problem
happens in my production.
In my production, there are a lot of
requests in the only one container.
And in the case that container servers
cannot handle the other request and at the point that
maybe the database data is a mismatches.
So what happens if the database
three copy is the mismatches.
So when some client died downloading the very
large object using the multi-part download.
So the file, I'm not if you had a dialog become
on download over because when the user downloading the object
inside the Swift, the object listing in the container
DB, because the proxy servers looking for the
list of the part of the object.
But if the container DB is unstable, the list of part
is not stable, so the object cannot be downloaded.
Then how we solve this problem?
I'm not sure that this is a good solution, but I
performed the scan of all object listing in the
three copies and manually correct that if the
sum object in the A and B, but not in C, I put it
put that object in the C manually.
It is very hard job.
Then the last topic is how to monitoring the open
Swift.
There are no metrics over the how to open
the Swift in our website.
So there are full main metrics.
So system resource and API is a very familiar with the other
system.
I'm using the parameters to collect the system resource
in the app, in the all of the nodes.
And API is a collector load to the monitoring the request
trend.
And then the recall is only in the Swift
open Swift, then you can check the status of the Swift
Demons.
So API monitoring, I'm using the electricity and a
logistacy and a graphana.
So I'm monitoring the how many requests in the production
and a response code trend and the other things.
And the recall is what happening inside the open
Swift demons, like how many partitions
is a replicated successfully or failures.
And is the all nodes has the same mean files like that.
So you can call it using the SwiftScire.comment.
And it allows you to view the Demons status for the account
container and object.
But it is the one-time data.
So I made the open text Swift.
The recalling exporters, and so promulgative collect
these recall data every five seconds.
And I do a graph to the monitoring these things.
Yeah, so this is how the open text Swift
recalls looked like.
And the last one is a steadily timing monitoring.
Yeah, it is the timing data.
So inside the open text Swift.
And then that there are new features in
progress opting to better understand the Swift
internal operations.
The opting developers are working on integrating
it with the open telemetry tools.
OK, so to conclude my sessions, in fact, unlike the other
sub-tative finance story service, the open text
Swift does not have much production of use case
and a document.
So it is very difficult to define what was the problem
in my production.
So I think the operator should know much about the code
levels understanding.
And then there are less the monitoring tools
and operational tools.
And the operators should make their own tools
to operate in their open text Swift.
And in South Korea, some companies
want to security things, like encryption or IPAsia
like that.
But pure open text Swift doesn't
support very good these features.
So we have to develop our own bidder error
and our own features in the open text Swift.
OK, thanks for listening my talk.
And if you have any questions, this is my LinkedIn.
So please send a message to me.
Thank you.