Good evening and welcome to the stock again. Nice to see some of you again. All right. This talk
is about the sex-self-final system experience in umbrella release, which is the upcoming
self-release this year. My name is Minkie, I work for IBM and I'm literally for the
falsacist team. Okay, you probably would have seen this 10,000 times, but I'll just include
this for for brevity. Self-self-fascist is a post-excompliant file system, distributed file
system, and these have metodata server, and clients cooperated, we maintain a set of distributed
cache, including I-nodes and directories. So, clients cache a lot. India sends out something
all the scabs to clients, so that to allow them to delegate a certain part of the tree,
so that the clients don't have to get in touch with the metadata self-requently,
thereby improving performance and throughput, and the file data, the clients actually talk
directly to the jarapolent loss. This is just a highlight of you, I think, super interesting here.
Okay, before we go on into umbrella and whatnot, this is a history of how self-and-self-fascist
has come up. It started around 2006, Sage's project, PhD thesis, scalable hyperformance,
distributed file system, somewhere around 2011, Ink Time was formed, and around 2014,
is been product acquired, Ink Time. Most of the times, self-fascist was marked as tech review,
it was not really production ready, reason being the development of recovery tools,
and things like that. So, if you happen to recall, to lose a file system, there was no way to
rebuild it. So, once we introduced that, we officially marked self-refascist stable in a single
MDSK, somewhere around 2016-ish, dual release, and near down the line, we introduced multi-MDS
as stable, multi-proactive MDSK, with real-fragged static fragmentation and also sub-direct
propending. Some were on not-last 2019, and first clusters, management was introduced in
the manager module, and the concept of self-refascist sub-volume came into existence,
and we will talk about that and what we are doing. Pacific was a big release, because we
introduced lots of features, self-refascist stop, snap schedule, let's see multi-MDSK
crabbing, limited window supported with self-token, there is a client with four windows,
self-refascist mirror demon, that is the asynchronous app, short application for the file system,
and also FSK. The kernel FSK script, I mean here, and we will talk about what FSK script
in the user space, with top-coming releases. And somewhere on 2024, 2023-ish, the squid release was also
pretty decent in terms of features, we have the choirs, and were improved MDS lock trimming,
which was pretty nice, because now you don't get the spurious MDS lagging behind trimming
warning, and we also disable the automatic balance, because it was misbehaving a lot.
So that was more or less in squid, I will talk about tentacle in the next slide, there is an
opportunity in a space to fit here, and finally we will go into the umbrella release, and what's
coming up. So tentacle was released last year, so let me just do a pretty quick recap of
on what was introduced. Case-in-sensitive directory trees and sub-volume, this is something that
my colleague Patrick worked on at last year, this is to support somebody use cases with surface,
because surface is a post-exampline file system, file names are case sensitive, while soundbar works
the other way around, case-in-sensivity, shoot-impro performance, and bring it on par with other
storage systems, we had to do some kind of strengthening on the MDS and the client side, so we
introduce a case-in-sensiv knowledge in the MDS and the client for some of your cases.
There was also a nice feature for blog, a five-log difference is between snapshots,
so we introduce a blog-tif-a-p-c, where you can ask the self-married RS server, give it a
file name, and to snapshots, and it tells you what offset file, offset, and lens, excuse me,
has changed between that file, between two snapshots, and those snapshots mean not be considered,
it can be further spaced, and the metadata server calculates all that has changed and gives you back.
The blog-tif-a-p-i, already obviously, is being used by the surface-married human,
and now it's much more faster. And we also had some requests from the community as part of
self-local on 2024 to years back regarding log replay times, and how much there are a bunch of
asynchronous jobs that run on the MDS, so there was no way to know how much time that would take,
and there were some couple of features, so if you fail over a self-married RS server, and the
fail over MDS now takes more than 30 seconds to replay its log, you get warnings and also an estimated
log completion time, that was putting it. Now let's talk about umbrella.
The main features, these are main features, I have a couple of them, slides for these,
a self-efferves are volume metrics, eager, mechanically-carious, being actively working on this,
we'll talk about this, snapshot visibility, mechanically-impair as well, don't it recently,
directory quarantineing, you know, you'll talk about this, but basically it's like
quarantining and adpress so that nobody had access to it, and you need a special key to access
that particular directory. Use a space encryption support, so I told Kurnitland that already
had the encryption support, all that is imported back to the user space, the main uses of this
is NFSkNation and SMB. Again, more on the usability side, estimated completion playing for
disaster recovery tools, you know, whoever has use a DR2 with self-efferves knows how painful it is,
because it doesn't tell you anything, absolutely nothing. It just keeps on running, if you have
parabytes of data and self-efferves, it probably will run for days, so you have no idea whether
it's making progress or doing anything. So now, with this feature, you know,
self-status will actually show you the exact amount of time it will take for a disaster recovery
finish. A couple of features here, the Kurnitland driver support is answered and dissipated,
somewhere down the line in the future. The Kurnitland driver obviously doesn't follow the
user space timelines or the September line, so that works a bit differently.
Come on audit logging, we can collect the areas spoke about this earlier today.
That's also upcoming, Meridim and improvements. We'll talk about this,
major improvements in reporting metrics, and a bit of the last low-hanging performance sting.
MDS tracing framework, again, colleague, Igor has been working on this and stabilization
of a couple of APIs that's used by an SSN Samba, and also the long-standing QS quality of
surface based on DM clock. That's going as a tech preview, probably. So let's start talking about
each of these features, so volume metrics. So currently, Kurnitland user space driver, forward
pipeline metrics to the MDS. So the client, you know, when it's doing IO, is trying to
see how much of a deal response from the MDS or the darapolis coming, and it aggregates this,
and sends it to the MDS. These are per client, right? These are not per share. So if you're mounted
a particular subvolume, like if NFS has mounted a particular share, which is actually a direct repart.
You don't get per subvolume data, per subvolume metrics. So that is changing now.
Now, first share subvolume will be reported for an unspawnetering. The way this is being handled
is, as a total MDS has some concept of a cap, a capability for an I node, and we who
want to that. So whenever the MDS hands out a cap for an I node, it sends along the subvolume
ID on the widget lives. So that's basically the I node number and the client has everything to
track all the metrics for this particular I node number, right? Anything, when all I was
will start with a cap request. So essentially the client knows which subvolume the file or direct
presender, and it starts tracking that particular those metrics. And the whole logic of forwarding
it to the MDS still works. So now you get like per share metrics. This is a sample, a dump of
how it will look like. So this are available with Perf dumps. And if you are learning other things
like node expert demons, these are plummeted into parameters. But you know Perf dump shows you now
the actual subvolume path and the counters for those path.
Okay, snapshot visibility. This has been mostly driven by IBM. The users for this is an FS
primarily where it was desired for the NFS clients to not be able to navigate snapshots.
All those snapshots can be created using you know every directory in SFFS has a special dot snap
directory and you can create snapshots by just laying an MKDI under that special dot snap.
This dot snap doesn't show up in listing but it's probable so you can see it into it but not listed.
This snapshot visibility you can switch the strict access to the dot snap directory.
So once you enable this which is actually a per client and a per subvolume basis you know the
clients do not have dot access to dot snap. With the odd cap restricted dot cap they can actually
not snap shut the directory itself but when you switch this on they can't even traverse dot snap.
Okay, this is the feature I was talking on which will go as a tech preview, directory quarantine.
It allows the operator to quarantine a SFFS subvolume they were a stricting access.
The use case for this is basically safeguarding data during security incidents such as ransomware.
Once a directory is marked as quarantine new mounts attempts on the subvolume are are denied.
Existing mount points that mount the subvolume path will be evicted.
They are blocked listed so they will have to remont which will be disallow.
For existing mounts of the parent directory so if some client has mounted in the root of the file
system and you mark a sub directory as quarantine all i operations on files within the quarantine
directory you start return errors and the only way to have access to the quarantine directory is for
a cluster operator to hand over a special auth key which is like a keyring to the user which is also
path restricted and whoever has that keyring is the only one who can actually access it.
So if you don't have that keyring you are a direct placement of quarantine.
This will be as a technical preview for the upcoming umbrella release.
Use of space encryption support as a set the kernel driver has support for encrypted subvolumes from
six of six. The entry names are encrypted using the fscript library file.is and scripted with
per file keys store n i no matter that now. This is compatible with all the user space tools.
So the fscript tools are you know they work with the xt and the xfo file system.
They transparently work with with with self-refestue.
It's essentially a port of the kernel client's implementation of fscript and the same
primitives and protocols are in use and cross-compatible with the kernel.
So fscript enable clients has to present a key to the log in a log x directory.
If you don't have the key you can't log it. fscript univered clients will see the encrypted version
of the file they were making no sense. My colleague Chris has done an extensive talk on it
on this couple of years back in self-local and I'll link that at the end of this of the
presentation that will give you extremely detailed view of how it's implemented.
Okay, some UI improvements disaster recovery ETC.
So as I said offline disaster recovery tools offer no or limited progress.
lack of estimated time completion for deploying tools such as data scan and the journal tool.
You know if you have ever run it you will know the pain but with now with umbrella the progress
and ETC estimated time of completion is reported by a sub-stress. So has a sample one.
I've omitted the other health okay things. So this is like a self-refest data scan scan
extents sub-task or maybe let's call it a step that's being run on a distributed on on on on
the file system basically scan extents is going to scan the data pool
going to each object into data pool and strength recovery of the metadata and with this you
you know how much time is going to take. I hope you never have to run this but if you do
you actually get a lot of you know information on how much time is takes. So in this case it takes
27 minutes whatever. Come on all it's logging. This is the presentation me and my colleague
very, very early on the day. A lack of historical record for you know was making things like
complications in cases you don't know what exactly happened. So with now with the implementation
of the command audit logging system you'll get a structured journal logical order of what
commands for run on the file system management commands you know when was maximum
the exchange, when was a particular config was switched on every single history is therefore you
to query and for us to be able to see what accurately was done.
Improve Meridiman, Federico was talking about this in morning. Thank you and this is going
in the umbrella list release. The tentacle release significantly improved this
after synchronization performance by using plug diff. There have been some performance deficiencies
for corner cases that has been addressed that will be addressed with the umbrellas and those
will probably be back potty to it and tickle release. But it may in focus here is for you know metrics
and feedback how much time is going to take for snapshots to get a synchronously replicated
to a different cluster. So that was something that was entirely missing and this it has been
actively gone. Integration with dashboard is also another thing that will happen.
Facing framework. So the design implements the hierarchical tracing framework so you know open
open telemetry is one of the popular tracing you know addition frameworks that that is widely used
and the MDS tracing framework implementation or the framework will actually you know the goal
is to integrate to integrate with hotel however for umbrella release we are not totally integrating
with open telemetry but linked on the foundation stood with the future. So the way it will
work is each of those MDS a client request will be tracked inside the MDS and each of those metrics
will be exported via NMDS command. So this is how it what will look like this particular
jation or the metadata is hotel compatible. So when in the future we do an open telemetry integration
all of this will be nicely variable in the in the yogurt UI or the open telemetry UI. So in this
case if you see real quick there's a link from the client that is being handled by the MDS and these
are the various stages of the request inside the MDS and you can see the MDS is not tracking
at a very granular level how much each of those steps are taking and some of these these
stages are asynchronous same on that is taken care of like the journal weight is
as the MDS somitting a journaling record and waiting not really waiting for it but you know waiting
for a call by for the journal to journal IO2 finish and the time metry tracing MDS tracing takes care
of all of it. Okay lips of this asynchronous and 0 copy API is that was there in the
safer for users based client but they were not much stable. Main users of this is again
NFS Ganesha and SMB the primary reason they were not being being being used were because of the
lack of testing. So with now a tumbler release at least workload testing has been started to
do with Ganesha clients so we test fews which is uses this client the call of client and now the
Ganesha client so all of the workloads will run with all of three and NFS and Samba I believe
Samba 2 has integration of to use the asynchronous IO interfaces with lips of FS and those
probably will also be called as stable NFS has integration but they have switched it off
once we market as stable as my NFS will to market as usable. Okay last but not the least I guess
long standing request CFMDS quality of service this feature that has been contributed by line
cooperation this spoke about this in CFL account 2023 and I'm sure I might believe they
video is out there they have been using it for long and they did submit a poll request however
there was you know couldn't make to review the poll request but now we have taken efforts to do that
at a test at a documentation which was actually lacking in the poll request however it will be a
technological preview take preview again so we have to like enable it it will be disabled by default
and once we are like you know happy with our tests and documentation mostly test is when we
will make it open for use. Okay here are some links for some other things I spoke about
the tracing changes the QS changes documentation to how to extract some volume metrics
and a detailed user space as a script talk from CFL account couple of years back.
Yeah I'll do happy with that first.
Please, I think you know if you start the right place as this question but
to say I know how to administer set of tests and I want to learn how to build on
this sort of something that's designed for AI workflows like Lester, I'd like to know
how hackable is set fast how to use it to get closer to something that can service
a GPU cluster for high performance. Okay so your question is basically how do I tune a
SFL system to work well with say an AI workflow? Okay so what typically would be the
AI workflow you're looking at? Let's say you have a cluster with CCGBUs and you want to be able to
reach like some of this I can see is and you know very high throughput
and the accuracy is very accurate and you know I know it's a bit special that we have
with no production with through communication does not reach that because of all the overhand
of replication and you know in the S overhead and in the view but like I've heard of certain
numbers from other people who do the last year and I was wondering if I can get them up like
build a customs set fast with sort of two in the weeks just to you know get closer to them.
Okay I don't have an answer right now frankly but you know you know if you have specific cases
we can talk about it you know but but supporting some kind of a Lester base and and
achieving Lester kind of throughput because Lester is pretty much you know everything is hard
to be based and you know to quite special things this is not that surface and not that so you
have you know that requires a special kind of you might want some special configurations
which we can talk about but I don't have anything on top of my head to give you an answer
but AI workloads are nasty I mean I recently saw an AI workload that was trying to list
a directory with one million files or 10 million files that is not going to scale with any
files system leave the surface right not even Lester I think maybe not even Lester
yeah I mean it's it's just management to know that they don't bucket us for you know
all these new AI workloads and our customers you know tell you all so we have this from this provider
and you can you you know provide this kind of phones it's just like I'm trying to understand
if if if it's like a technological limitation or is it something else like do I need to
think about hardware or different kinds of you know well it's not a technical technological
limitation but at least for the you know sometimes the applications have to be written
sincerely right listing a 10 million files some directory is not going to lead to nice things
so sometimes you have to just just just write the application in a more sensible way
and I have seen in the past that folks try to run an application that was previously working well
with a local file system and expect to work the same way in a distributed file system that might
not be innitiable target you know I mean you see projects and you see my own and everything is fine
but behind the scenes things are very difficult there are things that local files systems can
do because it's local and distributed files systems cannot I'm going to come across as
kind of a lemon and maybe but I've heard a coin word called GPU direct direct like something
that basically by a message city you to read from this which is you know storage engine
there are two GPUs without father and CPUs is it something you've heard of no I didn't like
yeah like I didn't but not really I mean
right
yeah pretty much yes oh sorry your question is surface pretty much having to to the internet yeah pretty much
so
thank you