All right, can you hear me?
Yes.
Excellent.
Okay.
Yeah.
So we want to give a presentation about our project that we have.
I've been working on for a while now.
And we love to have some more input from others.
And hopefully I have some good discussion about it.
Which is about building a multi-product stack based on open source components.
Just very briefly about ourselves, my name is Goodadationer.
I work for IBM.
I'm also part of the summer team since a long time.
And yeah, I've been dealing with all kind of aspects with some of the past.
Mostly with active directory integration, these sorts of things.
But just recently also now with building an SMB product based on class of file systems.
And in the beginning we were doing this with cluster.
We also have a loop played in a port control.
And now we are building an SMB layer on top of CFFS, which just got G8.
And I manage a team of engineers looking into these integration aspects.
Yeah, for IBM.
Yeah.
I'm basically working on that.
So as I mentioned, I'm also as somebody member in upstream.
And focus on the VFS layer of things.
Basically the integration to the distributed file systems.
Okay.
So what is mighty protocol?
What do we really understand with this term?
Even the term itself, sometimes it's written with a white space.
Sometimes we're not, I just have my own convention.
So it's all about a concurrent access to storage,
or the standardized network access protocols.
So you have one storage solution.
And you have a well established list of network access protocols,
which you all know.
And you want to create an architecture where it can basically
access the same storage of a multiple protocols at the same time.
Which is of course challenging.
And we're going to see.
So for now, we will only looking at NFS and SMB.
Of course, there has been, when we start the discussions around that.
A lot of interest also in adding S3, right, which is kind of that the new
of famous access protocol used by so many cloud components.
A bit of something that we do not get considered for our purposes.
So we only focus really on NFS and for SMB.
I try to paint a very poorly looking image that kind of visualizes
what we have to mind.
So we have some entity of storage.
It could be a share, it could be a directory.
It could be a storage system of some sort.
But at least some small granular entity of storage.
And then you have this multiple protocol heads sitting on top of that.
Accessing the same storage state at the same time.
And then you have a myriad of possible client combinations.
I just put Linux and Windows clients here as the front end consumers.
Yeah, we were going to detail the details of all these architecture.
So what is really the use case?
Why are we actually really doing it?
Why are we looking into this kind of infrastructure?
So it's really about supporting just the whole suite of classic office use cases.
Like like work like Leap office and so forth.
This is one of the messages that we, I mean, I've got from our product managers.
Even if you then consider word as just a simple use case.
If you're familiar with the pay load that really the word application puts on top of a network
system.
It actually explores immediately all sorts of very complex aspects of these protocols.
So it's everything but for sure not an easy use case, but nonetheless this is what people
who were starting this discussion ahead in mind.
Of course, we want to also allow the whole suite of developer tools.
Make it Python from pilots.
You name it.
These need to be supported by the architecture.
Also even web service that may be even simple databases.
In particular databases are going to be very challenging as we're going to hear.
But it's on the list as well.
So if we start to allow supporting these use cases, we certainly really need to have a lot of focus on
data consistency, right?
Whatever we kind of allow between these protocols, we need to absolutely make sure that we don't
put the storage and the central location at any risk.
And so we need to look very deeply at all aspects of file locking.
We need to also make sure that we have a consistent understanding of how to deal with
caching.
Be it the caching that the file system does.
Be it the caching algorithms, which are exposed over these network protocols.
And of course, all things related to authorization data must be very consistent.
Acles in particular.
At no means we may we may ever slip in.
Yeah, access permissions in these kind of scenarios.
Metadata, of course, needs to be dealt with as well.
So has actually this been done before.
And yes, of course, there are many proprietary solutions that provides this functionality
to some sort, of course, always.
In the fields, there's net up mix mode, vast, wick, nasuni, dellmc.
There's an ongoing list, there are many new products,
who just look at this as well.
Tuxera isn't considering doing this, I think, with a new NFS server.
But these are all fully proprietary solutions, meaning they have their own file system,
which they have under full control.
And they also implement their own set of protocols service on top of that.
But there are also examples of, at least a partly open-based solution.
Cetera is one that I started looking into.
I don't have many details on it right now.
But there is the currently named IBM Storage scale product,
which I've already had a different name over the past.
It's also an older product for IBM that uses the open source,
components, server, and NFS Ganesha for the protocols,
but provides its own proprietary file system,
called GPS, which is a very mature and very well-performing file system.
And their architecture, and we're going to touch a little bit on that as well,
is all based on the kernel-based GPS file system that they mount.
So they mount really a file system, as if it would be a local file system.
And then they have these protocols just operating on the local files,
as if they were really local.
And this is only possible to, while leveraging a couple of integration pieces,
that in particular, sound provides, to offload some of the complexity to the kernel itself.
The three examples I gave here are kernel share modes, kernel uploads,
and post-exlocking, so aspects for locking mostly, and also for caching.
But if you use this infrastructure, and whatever let the kernel,
the Linux kernel take the important decisions about access control,
and the locking, in that example, then you will quickly identify a couple of limitations,
these kernel APIs, they have been designed at some time.
So they kind of didn't catch up with the ongoing extensions of many of these algorithms,
like for kernel-leases, for example.
So they didn't really change a lot, they are hard to implement,
and then sometimes they are not implemented in a way that we can easily consume them,
or they are even not always following the specification fully.
So it's a very inflexible way to kind of do any changes at the kernel level.
This will either have an impact on what we are planning to do.
So there are examples, but is there also a standard, is there kind of any attempt being made ever
to specify this behavior or this architecture in some sort?
The network protocols per se, as you know, they are well standardized,
and we are defined and we are documented for the NFS specifications.
There are various RFCs, there are industry-wide entities that really just look into defining all aspects
and all extensions and improvements for the protocols over a long process of iterations.
And so there's a lot of specification going on there as well for SMB not so much,
because it was mostly Microsoft owned protocol,
and only with the core decision that has been made Microsoft was actually forced
to finally document their protocols.
But they did this in a really beautiful manner, not only specifying the properties
that you will see on the network, but also document the client behavior.
And that gives a person to implement the protocol a very good hint.
What is really the expectation?
The future ABC, is it optional?
Is it really enforced?
Will all Windows clients use it?
And what happens if they are not?
If these features are not supported and so forth,
so it kind of not only specifies the protocol per se,
but also the client and server expectations and the behavior.
Very different from the NFS specifications.
They leave a lot of room for interpretation and for the option character of implementing these features
that makes it way more difficult to come up with the standard way of implementing a specific feature
unlike SMB.
But no one really made an attempt to kind of create an overlap and create a specification for all these overlaying aspects.
And there are so many similarities in the protocols as you will see that this sounds to be very likely
and easy thing to do, but it is not.
There's a lot of detail which makes this very hard.
So all these solutions that I've mentioned, they all came up with their own way of doing a multiple protocol implementation, right?
There's no specification that you just follow and write down implement all things.
It's really a lot about what can the infrastructure provide, what use cases do you want to support, what clients do you have even.
But there is also the Microsoft Windows server and we as we both have a very strong sample focus obviously.
So we always look okay how what does Windows do, how does Windows behave?
And maybe this is a good opportunity to consider this to be a reference implementation and learn from that and see if things can be derived from that.
They do support this concurrent sharing over SMB and NFS for many years now.
But of course we want to be 100% open source based, so we will have to look also on what the servers can provide that we have.
Yeah accessible.
Along with all these discussions, I mean these are not new things.
There is a for calendar, sitting behind over there, he made a wonderful paper, a great presentation at the storage developer conference two years ago,
to really explain all the different concepts, right?
So the concept of his session, for example, how does that map in these two worlds,
apart from all the more obvious things like locking and directory names and these sorts of things?
So that's really highly recommended start point to really start overseeing the complexity of really mapping these two worlds to one thing.
Of course when we now want to implement something, we want to learn from that.
We want to build up on top of that, we don't want to kind of bring in any new protocol or protocol extensions to some sort.
We would ideally really just use the completely unmodified NFS and SMB clients out there, build in Windows, build in Linux or wherever,
and let them operate to one server without really being aware that there's really a multiple infrastructure on the server side.
So the server really needs to make sure that it understands all these problems that has a solution for all of them,
or at least kind of has everything under control, so that there is no exposure of any kind of,
yeah, undefined field, let's put it that way.
So that's that we need to look at the components that we have at hand to really see what is available already,
and see how these user space implementations of SMB and NFS can be modified appropriately if they don't already provide APIs that help us,
you know, join in to implement it.
And it is of course obvious that the file system which we are going to use really plays a key role,
and as you know, we also part of the safe organization.
So of course our file system of choice, our storage system is the safe infrastructure,
where we have a lot of ways to bring in new changes and to plan out supporting new scenarios.
There's also an important understanding that we are not planning to put a lot of authority on the kernel,
but to really use the different API that we use in order to communicate with our storage system in user space.
So we have an library which is easy to be extended, which is currently used by NFS connection,
also by server as an open, going to more details into, and this is what we plan on building on.
Okay, so I pass it up to you.
All right, so let's look at some of the building blocks here,
and as you can see, here is where you see,
for the first time in this slide.
As you all know, some of you might have already attended the earlier talk.
It is an open source storage ecosystem which provides file,
block, object, everything, and it is highly available and fault tolerant.
We, as Gunther just mentioned, we make use, we do not rely on the kernel space.
We rely mostly on the user space, because that's if most of the things majority of these logic happens in the user space.
So we do have a library called LibsFFS with high level and lower level APIs.
Right now, we make use of the lower level APIs to connect, connect with the underlying file system from a protocol site,
like if it is from SMB or if it is from NFS, we make use of this LibsFFS library.
And on the SMB site, obviously, some by search or choice here,
and we do have a layer called, which we have a, we have a layer in Samba,
where we have a module called, module dedicated for self.
You'll see the keywords have underscore new, which is a new module, but not very new.
It is at least one or two years old.
So that's the module that we use to integrate with the underlying self-file system.
It is capable of using this lower level APIs from LibsFFS.
It can do asynchronous IO calls, and of course, the FS script support,
which was discussed in the earlier talk by Vengi.
There is a kernel integration kernel client also from SFS site, but we are not using it.
We have purely going through the LibsFFS user space.
And from Samba site, there are some kernel level parameters that we could configure within Samba,
for oblogs, shy modes, and locking.
But at the same time, there are some limitations, like there are a particular set of oblogs that we have to disable internally
to make some part of it clear.
So the kernel level approach is kind of limited, and coming to the NFS world,
so the open source approach is via the NFS Ganesha.
In a similar way, where we have a VFS layer in Samba, we have a file system abstraction layer in NFS Ganesha,
which allows us to talk to underlying file systems.
And we do have a self-file module in NFS Ganesha, which handles this integration.
The common approach here is that both these integrations, like whether it is from Samba or from NFS Ganesha,
both of these make use of LibsFFS, it goes through the lower level API calls from the C native API library.
A little bit about how the self storage tag looks like in the current upstream world.
We did have some protocol enhancements, like we did not have a native deployment scenario for Sambi with SF.
But recently with the technical release, we had this integrated service, so we can manage both NFS and Sambi in a similar way,
from the same orchestration component of SF, which is a fedium.
And we can do monitoring IO retlimiting and the first script as I mentioned earlier.
There is also dashboard support for both NFS and Sambi, which you can basically do make use of to do the monitoring and IO retlimiting live stats.
And from Samba's idea, obviously, for active directory integration, we need windbind.
And we also do, we have integrated this aspect also, the active directory integration also with this storage tag.
Yeah, and the common IO path here is LibsFFS, which is the library that we make use of.
There is no kernel client involvement here.
Now, I will quickly mention some of the technical differences.
In some of the key aspects, when we consider the multiple protocol approach to such file systems.
First and foremost, the echoes, we know how complex the echoes can be.
But when we compare, for it is mostly or similar to some extent for Sambi and NFS, in terms that the permissions that we can do,
we can configure the LODI aspects of the echoes. Those are all kind of similar concepts.
But there are a subtle differences in both.
In Sambi world, it is called a security descriptors.
And in security descriptors, we can have different kind ofacle and risk like we can have a discretionary access control as cycles and stuff like that.
It is not just the normal file directory echoes.
And in Sambi, we can store these echoes as such in the file system using X Actors.
And at the same time, and at the same time, Sambi can map these echoes into the posics world, which is obviously a lossy mapping.
So right now, in the storage stack, in upstream, I think we have the touring of these securities descriptors as it is, using a different VFS module in Sambi calledaclexactor.
And with NFS Genesh, I also, I think, have a silent integration for NFS Genesh users, posic cycles.
So I don't think there is same app. Of course, yeah, I have linked draft the regards to how the mapping from NFS VFORacle, how low-ZGs and related details you can find it in that link.
But NFS Genesh can do NFS VFORacles. So that's already that.
When it comes to leaps of FFS, which is the common IOPA, yeah, leaps of FFS only provides posic cycles, right now.
Yeah, that's it with echoes. And regarding the exclusive locking, there is a concept of share modes in Sambi, and a similar concept called share reservations in NFS.
You can see a few of the flags that we use in both of these protocols like, there are some common things like read, write, but there is a, but in Sambi, we see a file share delete, which is not present in NFS Gen, NFS protocol.
So here also there is a difference in how this export of all exclusive locking is considered.
When it comes to leaps of FFS, we don't have anything right now. So for a multi protocol approach, we need to have some kind of interface from leaps of FFS to accept and accept all these semantics.
And by trying to locking, in posics world, the locking is always adversary, but in Sambi world, it is considered to be a mandatory one.
So if an application, which is opened and processed accessed by an Sambi client, now it is supposed to be done in a mandatory locking behavior that any other client is expected to check whether this is in use.
And then respond accordingly. But in posics world, there is no such an invitation. Like, even if there is an lock held by other process, you can just simply access the file.
Unless there are some file mode level, file mode bits are set and some kernel level involvement is, otherwise it is all adversary locks.
And since FFS compliance to a posics file system, I see a typo that we can do only adversary locks right now.
All right. And locking is one thing. The other thing is also caching. Of course, they are very similar concepts in the FFS and Sambi when it comes to caching, Sambi leases and NFS delegations have very similar functionality, even some similarity in the calls.
But they also differences that need to be taken into account. But foremost, just by kind of creating this mapping, this is just one aspect, but we also need to kind of find a solution.
How do we actually deal with conflicts? So what happens if an Sambi client has a lease and an NFS client wants to have a delegation.
And there is no kind of indication. What is the server behavior? How does it look? How does it need to react to these kind of conflicts? How does it resolve them?
These are all things that we as an implementation then need to define. And we need to really be very careful in particular these caching aspects. They have callback, callback spilt in. So they are calledbacks to the client.
For least, there is a lease break request, lease break reply and so forth. And they are even carries specific error messages and these error messages then triggers some specific expectations and behavior on the clients.
So it's a very sensitive area. And as said, there is just not one solution to kind of implement these things, but it's always the need for having a lot of testing and to really watch a monitor and understand how the clients react to specific reactions.
So this is going to be a very difficult and tedious area, which we will then need to look at next, maybe not for the first prototype that we are planning to create.
I right now, which is used by NFS Ganesha, to implement NFS delegations. That may be can serve as a vehicle to implement a mapping and a proper client server behavior that we have in mind.
Then of course, there are big differences in how high availability failovers are dealt with with NFS. You have a handling with grace periods and S&D. You have a much more complex way of doing it.
To be has some built-in ways to provide that, and there's even in the S&D 3 protocol and use service invented by Microsoft that will allow transparent failover, which is now also fully implemented in server by the way.
So there's a lot gap on that side that needs to find an answer.
Obviously, you have different file handle types in the S&D world, a durable resilient and persistent file handles, which don't have a similar concept on the NFS side, so that's also something to be taken care of.
And then of course, a whole meal read of small details like file system attributes, case sensitive file names, and so forth permissions.
So there's a lot of lists that we could easily extend to to a longer call, but there's not only the technical challenges.
There are also organizational challenges, right? To quote, forker again, he wants not maybe in this exact terms, but since he said no individual can oversee the design and resolve the implementation complexity alone.
So it can't just be done by one talented engineer or one talented team. It really requires a team effort of all these components to really have the same goal in mind and to work towards the same project and then also open up the server implementations so that it can be fully and rightfully implemented.
And yeah, nonetheless, as often just by looking at the problem space and painting it in a bigger and bigger picture, one could easily give up and never really start, but we actually want to start and we actually did start.
So the first thing that we do is to really invest into test cases and try to find simply use cases, try to find test cases that we have reproducible way to first really explore the the currently existing problems.
And so then really scale down what we actually all need to do on the implementation perspective to overcome these things and to really design the solution that we have in mind.
So one engineer on the sev team has looked into creating a testing framework based on Python with a defined list of use cases.
So that we can explore the the actual behavior first of all to really see if looking at actual as an isolated component is maybe a good first start point to really see what all needs to be done and what is already there.
And to build up and test infrastructure which is ideally then publicly available allowing us to to verify and to to to see if we're all doing the right thing in the turn of coming changes.
One could also think to leverage the existing sea libraries which are present and can be used for that.
We could also build up and invest more using these libraries and we need to of course design the implementations that we have in mind.
But also not really kind of trying to create the super multi protocol specification and really then end up with a monster that no one will ever be able to implement.
So we really want to make this very use case driven and to always motivated by what we can actually really use in the products and what are really possible use cases for this approach.
So it has to be all following a certain stage implementation as a set we look at the occult first.
The mandatory file locking next which was kind of a good way to probably look out the other protocol entirely so that we don't need to resolve complex overlaps.
But we can just kind of if someone keeps a mandatory lock on a file then the other protocol won't be able to access it in the meantime.
It's a very simplistic and simple thing to do but maybe this could be one of the first things to accomplish.
Currently we have a set we have a earmarking feature that basically this allows you to deploy the both services on the same sub volume.
Which is kind of our way to prevent basically running into multi protocol scenarios right now.
And then of course we need to look at the much more complex word of creating a ping of pipe range locking and so forth.
Again this is can then be endlessly continued and of course we want to and love to involve the community.
And just because we have chosen NFS and NFS Ganesha and Samba as use based choices.
This I think without really any big alternatives in the open source base.
But there could be also some interest maybe from other classed file systems to to to help building up such an infrastructure and discuss with us like maybe from cluster maybe from cluster.
I don't know but at least right now we are focusing all our all our things on on surface because there we can really easily bring in or easier bring in changes for the required interfaces like lip surface but.
Yeah so we would love to hear more input from the community and from from others outside.
And I think yeah I gave a list of some some very interesting talks the one talk that I mentioned from forker about creating this NFS and S&P common infrastructure highly recommended but there others as well.
There was a good talk from Tom tell P it's last year some exp conference comparing S&P 3 and NFS 4 also in light of the history also very very good talk.
There's actually one feature that we gave a presentation last year together with my colleague Patrick about case incentive.
File name support in surface which could be considered as maybe one first baby step towards market protocol support because it kind of harmonizes the the excess over multiple different layers.
And yeah that is all that we have thank you very much.
Thank you.
Sorry done.
I have a question so catch you.