So welcome, everyone.
In this talk, we will actually go through our experience
in provisioning stuff of us.
So my name is Matiba Lucco.
I work for the certificate services at TTH Zurich,
which is a section of the TTH Zurich IT services.
We will go through a little bit what was the context,
in which a surface provisioning started,
or this idea started, how we provision the system,
the benchmarks that we run, and then the production phase,
and also what we would do differently
if we were to do everything again.
So regarding the context,
as a fast TTH Zurich serves the learn-and-med
trusted research environment,
it's a platform used by around five other researchers
that stores three petabytes of secure data.
And as you can see from the graph,
it has been a web option as been quite sustained
and growing from year to year.
So we needed a solution that could actually accommodate this growth.
What's the starting point of our surface journey?
So we needed a storage system with a usable capacity
of around the two petabytes.
It had to be possible compliant and validated
to be reliable, scalable, affordable, reasonably performant.
And the idea was to have it accessed by virtualized workers
of various size and capabilities.
This last requirement is mostly due to the security
requirements of the infrastructure
where surface would actually be placed.
What does it mean with reasonably performant?
We are looking for something that has a, for a system
that has predictable performance,
smooth metadata operations,
most of the system users experience it's really bad,
gets really bad if the metadata operation,
like least indirectories and very manipulation is as low
as low or doesn't work at all.
And also, we would like to have the system
that describes as much as possible
performance posed by bad behaviors
to the specific project creating such behaviors.
So what did we do?
So we started with Blueprint,
the initial configuration quite a standard one.
So 16 was OK, so as the servers distributed
in four distinct racks.
Each server would have 24, are described, of 18 terabyte each.
And for NVMe, of two terabyte each.
We would have also predicated mode servers
that would host mode and manager services
and PM gas servers that would host the AM gas services.
Already we've had year hosting more than one AM gas demand
per server.
We will see why later.
All these servers had the same networking connectivity
based on LSEP bond with two links at 25 gigabit.
The guidelines that we followed for the deployment
were the usual ones that are available in the community
in the documentation.
We're using spindles, so we provision fast devices
for wall and db, so for the right-hand log and the rocks db.
We provisioned for OSDs per NVMe device
to try to get a little bit more performance out of them.
We put the metadata pool on fast devices.
Our NVMe is, if I want, but we saw before.
And we pick a IR clocked the CPU model
for the server hosting AMG-Ss.
This is due to the mostly single-treated nature of AMG-S processes.
So again, trying to get a little bit more performance
out of the file system.
And then we switched to the benchmarking phase.
So as most of you know, benchmarking requires time
and the benchmarking data are generally
gathered and stored, but sometimes never
properly analyzed, or also like gathered,
but only used for coarse-grained estimation,
or in many cases, it was not gathered at all.
So with the wisdom gathered from previous deployment
experience, we decided to actually bring in some external
out, in this case in the personal mountains,
some fast HPC, 12-pass benchmarking process.
Our plan was to characterize the hardware
to use a 16 virtualized worker nodes as client.
Each virtual node, in this case, uses an entire advisor.
And the reason for that is that you wanted something
that would mimic the final layout of the system,
while dispensing from the fact of having
collocated virtual machines on the same hypervisor.
We also aim the comparing replica,
or replications, the replication strategy
with a ratio coding 2 plus 2, and a ratio coding
of 4 plus 2 schemes.
From the very beginning, we encounter some other issue.
I guess the most interesting is this one.
We're benchmarking one of the SD nodes.
All of a sudden, we add what appeared to be a device
that performed twice as fast as we have a device is,
except the actual explanation was the other way
around.
So actually, we snowed that the 4-MPM devices,
one of those was actually performing as he should,
and free devices were performing half speed and half IOPS.
So this node actually got a screw
from the benchmark phase, and we got the MBMU replace by the vendor.
So it's always nice to have an initial characterization
of the hardware, because these issues can actually
lurk, and then actually review themselves at a later stage.
The initial data that we gathered from the benchmark
excluded the EC4 plus 2.
We did a rough evaluation, and EC4 plus 2 actually
had the sizeable performance gap compared
both to replica and the EC2 plus 2.
And the economy of space that you would have actually
granted us was not as interesting as to give away
more performance.
The first data comes in, and usually, at this point,
the benchmarking effort would be over and by other priorities.
In this case, fortunately, we managed to actually
go a little bit further, and obtain some data
for what we thought would be the most interesting case for the 5-System,
that is actually a small file random read.
So the main goal, the main use cases for this 5-System,
was to store data to be analyzed by means of a batch
system, or that are set by means of multiple batch
system.
The workload, the anticipated workload would be
read most if not read only, with a random read pattern.
So we picked this use case as kind of the most
interesting to benchmark against.
And here we see the number of clients active on the x-axis,
vis-à-vis compute nodes, and the attainable bandwidth, of course,
a value are quite low, because they are kept by the maximum number
of IOPS, the 5-System, can cope with.
And in fact, if we actually switch to a viewer,
we have the IOPS on the y-axis.
We see that we reach around 80,000 IOPS with this configuration,
and we're actually looking at the aggregated IOPS for a replicated pool.
So in this case, we have our 5-System that has a flash,
so an MVM unit, a data pool.
So SD with the based on MVM devices,
and then replicated the full data pool underneath
where the test were run.
The scaling is, so we see that we reach around the maximum
performance at 5 clients' active clients,
that we're running actually 16 threads each of IOPS.
And then, like we decrease somehow not linearly,
but with some bumps along the way,
but overall, like in spite of the results.
So here you see that the maximum value is 80,000.
Now, if we switch to the same test, but then,
with the regular coding, so again, this is the bandwidth,
which is not super interesting, because it's again limited by IOPS.
But if we go to IOPS, we see that now the maximum value,
it's a little bit more than 50,000 IOPS,
and we can actually max it out already with three clients.
But this is, remember, like everyone is a random read,
so we are actually really paying the regular coding policy,
that needs to actually read from multiple devices
to provide the data back, as opposed to the replica.
That just needs to read from the primary SD.
This shows also that in the regular coding case,
actually the trend seems to be a little bit more regular,
with the increase of in number of the clients.
And it also gives us hope that if we not actually
keep this trend and not go down linearly,
very steeply, after we added a few more clients.
Here, I believe it may be a little bit small,
so I'll try to actually describe what this graph shows.
While the initial idea, or the initial goal,
was to have a file system for read most workloads,
we wanted to also see how it behaved, mixing IO types,
so read and write, read is read here,
and write it in blue.
And then we have like the columns,
are different, sorry, the row are different block sizes.
We start from 32k to 1 megabyte, the middle one,
and 4 megabyte, the bottom one.
And then we actually changes a few parameters,
so we started with the first column,
where we have all sequential read and write.
And where we can see that actually the absence or write,
create quite a big effect on the file system,
if the block size is more, a little less,
so when the block size increases.
And then what we see is also that
starting to any of the other columns,
which means actually adding some random,
into the mix, the performance would actually remain stable,
but decreased drastically, in particular in the first case.
So in the first case, we see that this random IO would actually
create a lot, like decrease a lot with performance.
This actually it's a bandwidth graph,
but the IO scraps looks exactly the same,
the same kind of behavior.
As you can see, the maximum IO, so we can obtain
with a four-under megabytes, in this case,
is then limited by the bandwidth and no longer by the number of IOs there.
So this is where we stop with a benchmarking part.
We were quite happy, the system that we had to replace,
had some parameters, or had some performance target,
but we're similar to the one we saw with benchmarking part.
So we decided to go in production.
Here we're talking about three years ago,
where it is, we went in production with was specific.
And we started with a single active and yes,
and all the default settings except for a little bit more cache for the MDS,
so the active and yes, MDS.
I believe the default was the eight of the time,
and we actually gave the MDS, so we started with a 30 gigabytes of cache.
We actually managed to migrate more than one bit
about a bit of data from the previous storage system,
and ramp up the usage by migrating individual project
to the new system once the data migration was completed.
But it became a sum clear that a single MDS divone
could not cope with the amount of request originating from the clients.
So just a super brief look at the results architecture.
The client has a direct data path to the Rados pool,
but the metadata path is mediated by the NDS.
In our case, with a single active MDS,
the clients could talk with the vicinity,
but would actually hold the cache and the journals
with the five system, and then write the metadata to the metadata pool.
When we decided, when we saw that this approach was
in a longer feasible, we decided to actually increase the number
of active MDSs, which in the case of Seth provides a coherent caching,
a coherent cache for the metadata part,
and also a coherent journal from point of view of the client.
So the initial step of our corrective measures
to try and cope with the actual workload that the cluster
was experiencing, was to increase gradually in maximum DAS,
and we started from one and increasing in steps,
until we found an optimal value that in our case is 7.
The first increase was actually to just free,
but we noticed that that brought a sudden increase
of the quest per second counter counters,
and initially we thought that everything was working fantastically,
because before the numbers were much lower,
but we later realized that it was mostly to be an MDS balancer,
exporting some trees between ranks,
because some of the workloads were actually insisting
on very specific directories, and before the balancer would actually
try to export them to less BZMDS.
We went after that, we pinned the most active project directories
to dedicated MDS demons, and the scene that it was working quite well,
we landed decided for strategies to pin them all.
Namely, we decided whether we want to pin a project directories
to dedicated MDS, if actually story sizeable amount of files,
and they actually also are quite active in analyzing those.
Or on the other hand, if we have like a small project,
but have a few files that sit quietly,
we just put multiple of those in a single MDS.
This actually gives us also a way to understand
which project is actually creating the most traffic,
which is a metric to use the also to track the file system status,
and eventually potential MISB heavier,
or MISB heavier, or lack of, let's say,
appropriate knowledge to how to use such a fast features,
like still trying to use the U-1s of a fast, or doing some other things,
but instead of asking them to do that differently.
So one of the other things that we tuned was the cache of the MDS,
based on the documentation, the release
that we added at the time.
We had some of these changes were based on the documentation,
on the explanation provided by documentation.
We'll over the number of capabilities,
a single client can hold it from a million to a million.
We have around 600 clients right now registered to a file system,
and we wanted a way to keep that in track.
We incrementally increase the MDS cache memory limit,
and right now we're currently having a value of 100 and 10 gigabytes.
This is actually not needed for all the MDS demons,
but unfortunately our pinning strategy is not too,
it's a little bit coarse-grained.
So there are some MDSs that have a lot of the files,
I have to say a lot about it, and before we had to have a slightly higher value there.
We did some changes, but helped us to understand a little bit better,
and prepare the file system to the varsity use case that arise
from having a batch system insisting on the file system.
Namely, we increase a little bit the cache reservation,
bringing it from 5% to 10%.
And we also lowered the MDS cache threshold,
which is the threshold by which the monitor would actually complain
about an oversized cache in the MDSs.
So we wanted to make sure that we were warned early enough,
that something was going wrong with our settings.
What we also did was to lower the cache trim decay rate,
and increase the cache trim threshold.
So the value that we used, so one is still the default for both
Pacific and Quincy, the value that we used for the cache trim threshold,
which is a 3123K, is actually a little bit higher than what Quincy has right now,
but it's way higher than what Pacific has as default.
So Pacific had 64K as a default, and already Quincy has it two times higher.
This before doing all these changes, aside from driving a little less control
on the cache size, we would also have the MDS complaining
about being behind on trimming.
This change has actually helped a lot on that records.
And a very issue that we had was the deletion operation
that some user trigger on millions of files at once,
or even like, because they wanted to think faster,
inside is worm jobs.
So we would actually have worm jobs trying to clean up things,
and this would actually really put the MDS demo that was serving
was requested to his knees.
So we decided that taking longer, it's fine, as long as it doesn't affect
the world cluster stability, with a increase to that.
So we decrease the max, and yes, we'd file to 64, and also the
opsperpG20.5.
It takes longer to delete, but deletion are no longer affecting the world system.
When this is something that I find out preparing this slide,
as well with the value that we saw before,
we actually tune a few of this value that are
the tunable for how TMDS recall the capabilities from the client,
or when it decided to record them from the client.
And interestingly enough, for me, our values are now in between
Pacific and Quincy.
So we had the value of 20k,
while it was 5k in Pacific, and it's now 30k in Quincy.
The max decay rate that we set was 2, which was 2.5 in Pacific,
1.5 in Quincy.
The decay rate here is actually works on the other way around,
so it's actually part of the formula as a logarithm inside,
so the higher the decay rate, the less the less,
the first example, the less capabilities are recalled per second,
and in this case, the less caps are recalled there.
The threshold is again in between,
so we have the value of 65k in our cluster,
and the value of the current value is 120k for Quincy.
All these values were actually, when we started with Pacific,
were a little bit difficult to tune,
but the documentation explains what they do,
but then it's difficult to actually map them to the operation going on
in the file system, and there were like a few posts
either in the mailing list or even like on some step storage product,
and we were advising just picking a set of these,
max caps, decay free sholes, and I think also of the max decay free sholes,
and just double them until you found a stable situation.
So we went a little bit of way there, like halfway through these strategies,
so we decided that we would actually try to use a multiplier,
but that seems sensible, so not like 10x, but yes,
now but we're really happy that also the defaults are actually being increased.
Warning, threshold is again something that we change and it's in between.
So one thing that we learned the hard way is that in our case,
having a busy system with a large cache,
when an MGS demon fails, and it can fail either because an operator needs to free up an MGS server,
and once the MGS to manually fail, so you can reboot the hardware,
use maintenance, or like the MGS gets killed because there is a misconfiguration
in the cache value, and the out of memory killer takes it out.
The new LES, MGS assigned to the fail rank, in our case needs more time than the default value,
but I believe it's 60 seconds. And after several attempts, we find out that if we wanted to
be sure that the MGS could take up the, could join the cluster, again, we added 12 at least
30 seconds, probably like a little bit less would work as well, but we were afraid that if anything
would happen, that would actually be recovered automatically by the system, if we had the lower
value here, I would not have just wait for an operator to do something about it.
So what we would do differently?
So I would now set up the defaulted set that up, it's on spindle.
So the defaulted app will need to be replicated since nowty loss, but it's recommended to
be replicated since nowty loss, because of the tiny objects that are created on the route of the
file system, but contain the trace information and not link references. So the idea is that there
is one of these objects per line order in the system, and currently we have like 560 million
nine nodes in our system, and doing some rough calculation, it means that we have 3.5 million of
this tiny object per device. And if a device actually needs breaks and needs to be balanced,
that's like 3.5 million objects to move and with a relative observation, which are not so cheap
in a system that are disk-based. So in our system it takes as much time to the balance, but the
full time I'm the surface data pool, so the folder replicated one, as it takes to the balance,
like the actual data contained on the disk, so we're talking about 12 terabytes of data,
as opposed to like 350 million tiny objects. So the idea, if we had to do it again, would actually
to make this like a flash pool as well, so to leverage the kind of free IOPS, but those devices
can give. And everything that I feel we did not put enough, let's say, effort on was to evaluate
at the other SC algorithm and other than the default. So there is a very nice talker from
Jimmy Pride, from IBM that was given at the last several of them, but actually go through
a few of these algorithms. Also the banking, the fact that the eyes are national is not working
only on Intel CPUs, but also works in AMD CPUs. And it presented a few benchmarks, but make
us like wish we had done that part as well, because the actual performance gain is more than two
acts from certain workloads. So depending on the use case, it would have been interesting to
actually look at. So this actually concludes my presentation. It's very question.
Yeah, every benchmark that you saw was everything on, sorry, yes. If like the graph that I showed
before for the IOPS and numbers were on a spindle or not. All the benchmark that you saw
all the graphs that are in the presentation are always benchmarking spindle, because in
in the end that's what we have to work with.
I think my question was, what did you actually use to process the benchmark, so I didn't actually
look at this data? Oh, that was just, so to be honest, I don't remember. Sorry, the question was
what we actually used to process the data from the benchmark, and the answer is that I don't
remember, I'm sorry about that. The processing was done by StackHPC, so I don't have actually
the actual script for the dynamic processing part. Yes.
Now, we're currently in the process of designing a new set cluster. We've got the advice
for the so-called R&O of the servers who use single socket, I suggest that it's an
single socket. Yes, so indeed, so we never really managed to benchmark the difference between
dual socket and I'm sorry about that. So the question was, we used dual socket servers,
and the isn't it better to actually use a single socket, the server for this use case.
There was a test done on dual socket system with everything separated, we've like dedicated storage
controller attached to both CPUs and mix for each CPUs, but it didn't really seem to have
such an advantage over like having like a dual socket mix case. So I'm not sure if this
actually is also true for a single socket server, but yeah, like this is something that I heard
several times, but then when we actually went to the vendor and asked for actually an hardware
that could fit the like the video, the space or the badge that we had, single socket was always
out of reach for us, so we never really managed to that. So thanks everyone, and I'm a little bit