Thank you, everyone. Thank you for having me today. It's a great opportunity, great honor
to be with you all today. Take your leads and open source answers, yes. Yeah, so thank you.
My name is Fred Wake for the Knust. I'm a 7-bass driver for France, and I've been working
on Cess since 2014 at French University in Estifloin, and I joined Clyseau last year recently.
And so our talk, my talk today, will be on Cess S3, as three storage, with dynamic placement
and optimized retention. So we'll go through over the different points here, so we'll
start with storage glasses and get to know how we can use a Lewis scripting of the radios
gate weight for dynamic placement, and also life cycle policies for optimized retention.
And then we'll get to see the configuration to set this all up, and then we'll go through
some demo. So storage glasses, there can be multiple storage glasses, set closer, link
to varied data placement schemes, replication, and a rager coding. And we can also use
some compression if we want, with some different storage glasses. And also I mentioned here
Intel QAT, because that's how you can offload the process of the compression, for example,
when you don't want your radios gate weights being too much of a loaded with compute
compressions. So this enables us three clients to place objects based on criteria, like
performance, access frequency, durability, and cost. You can so have multiple storage
classes with multiple data placement scheme, and you won't have the same latency depending
on whether this is a rager coding or replication. So yeah, and regarding cost, then you could
use less storage if you get to those objects being recorded on a rager coding that pulls
rather than replication for example. So there are three clients, set out of the storage
upon writing an object, but they often choose incorrectly or sometimes do not specify any
storage class at all. So the idea is, what if we chose for them? When the client does not
specify it or incongruously forces it, we could then automatically assign a storage class to an
object based on the object type, for example, his name, the object size, the tenant, or even
the user, the bucket to where the object is going to, and the upload method. So by doing
this, storage efficiency and excess performance would be insured right from the moment the
data is written, that's the idea with Lua scripting. So as three gateways can execute Lua
scripts on the fly, so when request arrived, put request arrives, based on the context,
pre request, post request, background, get data, put data. Those different contexts you can
have examples and explications on this page down below here, Lua scripting in the documentation.
So the Lua scripts can read and modify object metadata on the fly, and therefore we can
dynamically set or modify an object storage class upon writing according to certain criteria.
Where Lua scripting can help, this is an example of one, for example, solutions,
expecting uniform performance, regardless of the size and the number of objects that they write
to S3 buckets, to a single buckets. A typical example that has been backup in recovery,
which stores data objects, but also metadata objects, backups, catalogs, index, locks,
pointers, and in the same very same bucket, but there are all from very different size,
very very small objects and also big objects, the actual content of the backups.
So the challenge is how do we get that to be performance? Keep the performance,
even though the bucket gets to millions of objects. So the idea here is to differentiate
that those objects and automatically direct small objects to replicate it pool and
replicated pools and bigger ones to EC pools. So since VBR, a version 12, for example,
being back in recovery now groups a small metadata time to single S3 objects to address this
kind of design flow. So there are other applications that could benefit from this kind of scenario
LOS groups. So LOS groups could provide an additional protection layer also,
beyond bucket ACLs, to block requests. Based on specific criteria, for example, user, tenants.
For example, this could enforce read only or write only bucket access on publicly exposed S3
gateways as opposite to internal gateways that you could use to write or to read.
So I mean, my colleagues here and also Marshall have been working on this PR to
eventually add this ability for LOS groups to block requests. And there are other use case
that you can find on in the upstream dock and also some code samples to trace requests for specific
bucket applied default metadata would not specified by the client. Log operations only when
errors appear on capture operation traces for an analytics, for example, for instance.
What about LOOA performance and reliability, of course, for a request, now we add a
stripped that needs to be run. So that might have some consequences, of course. So what about
CPU and RAM conceptions? So initial test, that way made out of course, so should that LOOA
does not add much latency, like some sort of tens of microseconds. Also in squid, there's
execution of a script in a context, can you use up to 1 and 20k of RAM? This is configurable.
And what if the script fails on a timeout, which by default is 1 second or syntax error? Well,
it script fails on non-fattle for specific lines. Clients will receive a normal response,
as if the script was implied at all. So it won't break the upload, the activity of the client.
There are also some things, there's also something to notice here, there's a performance improvement
as being worked on recently in the last two weeks to improve the performance of
bycaching the LOOA bytecode, which also reduces the number of network calls to read the
rate of subject containing the LOOA script. So that was for the data placement optimization.
But now we'll be talking about lifecycle policies, and that's the optimized retain retention.
The idea is to move some, based on some criteria to move some objects to other storage classes.
That's the first point transition, objects between storage classes, based on criteria that
can be days for how long the industrial objects have been here in the cluster, or object size
greater than or less than. That's interesting because you can do both at the same time.
Just to make sure that if you move the threshold, the size of the object,
some of them will move to the other pool and the other way around. So that's good to know.
So that's since we've squeezed. Yes, the idea is to optimize storage for railway use data.
Also, they let non-curren versions of objects, eventually retaining some versions of the objects,
and also free up some space by cleaning up incomplete multipart uploads.
You know, that's some, when some MPU fails, and the client never comes back to restart,
resume its workload, and it will leave a lot of parts in the cluster. Well, in the bucket.
That will, you will have to take care of removing. So lifecycle rules can also use tags or
prefixes to apply only to specific objects. So this is an example of a rule that you apply
per bucket on a specific bucket that will clean up multipart upload parts of the 10 days.
Move objects to deep archive storage class, as we've shared it to EC pool,
raise your code pool, 8 plus 3, after 30 days, and the objects after 3, 6, 5 days.
So I just added this be careful because I will empty your cluster of bucket, of course,
from all of your objects too. So that may not be the best move to do what this is just for
proof of concept. So the configurations look, the configuration looks like this.
We start by creating set pools of different kinds, so replicate it, raise your coding,
and another type of raise your coding, 8 plus 3, and we will also add storage class
and default storage class. And you can verify the configuration with this command on the right.
So yeah, configuring the data index and on EC pools in the zone.
And is it here that, yes, we set the compression also, we can set the compression LZ4 here on
ZSTD for example, and then apply the configuration.
We also need to send a configuration file for the different rules that we will have for the
LUS strips. To say for example, choose this storage class if the file of the object has this size
or match this pattern, for example, or is this from this bucket, oh, going to this bucket,
or a connection from this tenant, for example. So these are the rules that the LUS strips
will be using. So we can add this configuration file to the global configuration service
of RGW. And then we will apply the service and then redeploy the rate as gateways.
Yes, and then we can check that on inside the container and we can actually list the five configuration
file. You could also prefer to just mount a file that would be somehow static on the rate
as gateways. So you don't have to redeploy the service to apply a new rule, right? So you
would just log into the rate as gateways and modify the file. Just make sure it's not used
because the VM is using a temporary file and the end, sorry, the, I know the link will be
broken and so inside the container, you won't see the modifications that you've been doing
on the host. That's a typical thing that I've been encountering. So you could do the configuration
here or directly on the host, mounting the file. So here we are giving our rate as gateways the
Louis scripts. Here is an example script that I've made. You can find it on my GitHub if you want to
use it. Just make sure I've been also using it. I've been also using it to produce this script. So just
make sure to double check double read the script before we're running it. Please, yep. So
then here we apply a lifecycle policy. So I'm not going to deepen to that. Oh, by the way, if you
download the presentation, the slides, you will get those our text box. So you don't want to
copy past some of the comments in the course easier to do. So we can check the lifecycle status.
It's a background process that will run from time to time. Looking over at the different objects
and get to know, see what needs to be done over these different objects. So at the beginning it's
an initial state. And then you can start the process manually with this command and then
list again and see that the lifecycle. Yeah, so Louis scripts logs everything at debug
suitable you level 20. So if you want to see how your script is working, if your roles are
being matched or not, just make sure to increase up to this higher like one of the highest
probably highest debug level to see what it does. So yes, the idea is to monitor the pull activity
create a different size objects from 16k, 4.7 megabyte here on this example and push object to
S3 storage and get to see our roles match or not. Yeah, so we can also list every objects in the bucket
with this below command here and get to know which class storage class was associated to them.
Yeah, so here are the logs. For example, object size bigger than two megabytes gets this one
above object size above two megabytes as a sign of the default storage class which by default
anything in this example is it's standard and frequent access and it goes to the EC4 plus two
pool. Another one here is a 475 kilobytes so below two megabytes is a sign that the SC storage
class standard and goes to the three times the replicate pool. We can see here how the match goes
no match here so storage class by default here there's a match so storage class route for rule one here
so it will go to the standard storage class. Yes and every multiple uploads goes to a specific
storage class here that I named deep archive which is forced. Why is it forced? It's because when
an object is uploaded with an MPU method so multi-part upload, Lewis scripts won't be able to
identify the size of the object at the moment it starts the client starts sending the object.
So you have to predefined pre-choose somehow some storage class or MPU upload.
So since this this is MPU it's probably a big object right so the idea is to send it to
deep archive but then again if you want to read if it's not a big object it's small one. You
can also have this lifecycle policy that you will set on your bucket and then the object will be
yet again removed like not removed the move transition to another storage class.
Okay so how to check how can we check the optimizer retention. You know this if you're using
staff you know this command which is rate is df or sfdf it will show you the different object how many
objects you have in the different pools and also what place they are using in your cluster
in the different pools. But of course if you have to wait like 30 days or 365 days to
know and to check whether the object has been transitioned from one storage class to another you would
take a lot of time to just to make sure that it works properly. So there's this configuration
that you can use the setting or you will see d bug interval. This is a dev setting that was
meant just to on this purpose like for this purpose. To check if our configuration or lifecycle
policy applies correctly. So by doing this you would say for example that it will turn
any day like 24 hours into a single second. Okay so if you set a policy that will do
something transition objects after 15 days sorry 15 days you would get the results in 15 seconds
rather than having to wait for two weeks. Yeah and so on this example we set a transition at 15
after 15 days so 15 seconds and expiration after 30 days or here seconds.
So here the object is written at 8 o'clock 12 minutes and 5 to 6 seconds and like 15 seconds later
at 13 minutes and 11 seconds. Okay so lifecycle is checking if the object is expired. If the
30 days if it's here for more than 30 days and it says is expired no 0. 15 days says yes it has expired.
So I'm friends in front of the file to the deep archive which is the rule that we set.
And so like 15 seconds later here at 26 the objects match like the matches both is expired one
is expired one. Okay so now the object is deleted. Right so that's how you read the laws.
Like but this is yeah that's the lifecycle standard log. So we can have this quick demo
here we have a set cluster set status on the left and the pool the sorry the pools here
only 30 rupees PGZ each this is a small really lab cluster so nothing big here.
I didn't expect me that quick. Yeah cool.
So yes we've made this series of files different size and we will upload them here using our
clone. I will not run this with a bunch of content. So now we see that our pools are empty zero objects
nothing stored nothing used in those space used in our cluster. So I'll just start
what is the what? 2000. Okay well so I'll send different objects from different size and then we'll get
to see that some data is now being stored here. I'm going to surprise that this was not
the numbers that I was expecting but anyway that's the demo effect even. So yes and you see that
okay so by default they went to the warm data over to megabytes and went to the hot data
below this threshold of two megabytes. And then after some time of 15 minutes we've seen the different
the data moving to the the archive pool. So now it has been fast but after 15 seconds the data
has already been moved to the the S3 archive data pool which uses 8 plus 3 the displacement scheme.
So the idea is again is to for data the pool probably that is there for 15 days here in this
example. If it's not expected to be accessed anymore or maybe less frequency for equivalently
and then you should better move the data to another storage class. Then you could ask but why
do we still have some some data here? It should have moved right because this policy moves any
data to the archive data pool after 15 days. That's the reason for that is that the lifecycle
policy only copies object from one storage class to another so one pool to another. So it doesn't
move it does copy right. So what we'll get rid of the data here is the garbage collector.
The garbage collector will see that does no point in keeping this data.
So if we run the garbage collector we'll see almost all data and probably all data being removed
why because there also has been 30 seconds already. So that's expected but yeah if I run the tool again
run the alkaline command again we'll see the data going to oh I'm not in the dd files folder
and that's the reason why I was uploading more than expected. Yeah so let's do
now the gc will come into play and remove some of the data but not all because the 30 seconds are
not there but yet. Let's remove this to this manually. So now all pools would be
empty.
Yeah demo I've been doing this like four times this morning or at least that's why.
See some of the data goes to the hot pool and some goes to the one pool depending on the size
and then yeah so well that's a demo effect that should have worked better than that but
still that's the idea. Yeah I used to optimize data placement right from the start I mean right
from the data when the data is injected in the cluster and also then get old data to occupy less space
in our cluster. Yes as they get less access. Yeah five minutes left. Well it's time to say thank you.
So yes I acknowledge means to you've all of the shifts he's the one that has a coded
lieu as support in rate as gateways instead. Steven and Biocker for his work on RWOTO tiering
and also inton he Dutchman Kirk Bruns for that talk on RWW lieu is scripting that you can find on
YouTube and also this French guy a little on Bob which is a friend of mine. Casper is you might find
a good stuff on his blog for sharing his experience on using VIM backup and recovery or
replication with S3 storage so the idea is but that this idea that idea is good for any
kind of storage class and stuff as many software expect Amazon named after Amazon storage class
so we better go with Amazon storage class names yes so use a standard storage class for for example
and VIM drives for objects that are less in size than 64K and then use standard infrequent access
storage class on HDE's or hybrid drives for objects above this threshold and set the default
storage class for standard that's the right way to use VIM on on on S3 storage okay thank you
any questions
thank you