Next session is up. We were staying with Seth for a little bit, but slowly move inside
waste. Yeah. Give it up. Hi. Thank you. My name is Sachin Prabhu and I work with IBM and
what we are discussing here is one of the major problems we had when implementing the SMB service
for Seth. So a quick introduction on part of IBM came from Red Hat after the acquisition.
I'm part of the Seth team and we work on the SMB service as described before and we also have
a GitHub page for our standard loan projects which we do as part of the team. Some of the stuff
I'm talking here is part of the Seth repo, but some of the bits I use for testing are
hosted at this location. So quick, the Seth SMB service, this is an SMB manager module. We
export a Seth FS volume over SMB to do this. We use Samba in a container which is part of the
Samba container project. This is one of these projects which are hosted on a GitHub page.
And we use a new Samba VFS module or VFS Seth new. Now before we get to the problem,
I need to talk a bit about the Foking model in Samba. Now every time you have a new connection
coming in to Samba, a new client connection coming in, we fork a new process. The UIDGID of the
process is switched to the authenticated user. Now there are many reasons for doing this and some
the main reasons are portability. It makes easier to write code which is run on several platforms.
We do not have to, if we do not have to keep switching the UIDGID or the process, we just switch
the UIDGID at the authentication time and just let it run and robustness. So if one of the
client connection dies, it does not take the whole server down with it. But this Foking model also
leads us to this problem. We describe here. Now imagine we have a large number of simultaneous
clients connecting to the Samba server. Each of the connection leads to a new process. Each
process then has to connect to the backend SFFFS volume. It uses the lip SFFFS library.
Now each lip SFFFS connection has its own metadata and data cache. Now once you have an
IO with starts, each of these connections, the cache keeps growing. Eventually, it leads to
memory depletion causing the server to die. So to reproduce this problem, the reproducer is part
of another project, other of our projects which is the SIT test cases. It is a simple Python
script which uses the SFFFS protocol Python module and all it does is it opens up multiple threads.
Each thread opens up a new client connection on the Samba server. We then open and close
multiple files and we perform IO on it. And what we noticed is this is on a test
SFF cluster with three nodes, a four node was used to run the client tests. And what we notice
is after 100 simultaneous connections, we could bring down the server because of this memory pressure.
So the solution which we propose is the lip SFFFS proxy. Now this has just been
added to the SFFFS app, upstream SFFFS app, we also have a design document available in the SFFFS
app at that location. Now the main objective for this particular project is to avoid independent
connections, cache connections for each client connection. So in this particular case, with the proxy
enabled, we have run a test which on the same test test cluster, we were able to simulate
1000 simultaneous connections. The proxy solution itself has two parts, two parts,
one is the lip SFFFS D demon process. And the second is a proxy library. Now this proxy library
is it sits in the same location where your lip SFFFS library sits. So clients are linked to the
proxy library instead of the actual lip SFFFS library.
The demon itself, the demon connects to the SFFFS volume using the lip SFFFS library. And what it does is
it centralizes all the requests. It listens on a unique socket and the clients use this unique socket to
connect to the demon. All the requests are funneled through the demon process. And what we do is
we end up limiting the cache to this particular process itself. So the lip SFFFS proxy library
we provide a subset of low level SFFFS API calls. And as mentioned, it is used in place of lip
SFFFS today. So in this case, there is no caching done on the client itself. All it does is
forwards a request which is coming in through the the the demon process over the unique socket.
The same configurations, client configurations share the same connection. Every time you have
a new client connection coming in with a different configuration, a new connection to the SFFFS volume
is created. Now because the clients can mount different sub-directories within the volume,
so that that means some calls require special handling. And these are the getCWDCHDR.
Finally, for testing, our our colleagues in QE decided to test this. We used or the QE team
using the product protocol spec storage, which is used to perform this test. The test were done
on a cluster, which had CTDB enabled. But all the testing was done on a single summer server.
So we do not actually use a cluster. We would be testing against a single summer server.
The mount was using a SFFS kernel mount and those are the product versions we used.
Now as expected, sorry, there are two two different workloads we use here. One is the software
build which simulates a make on a software project. This is very metadata heavy. And the second
is a video data acquisition where we simulate reading data from a stream a device like a camera
and writing to a single file. And as expected, we have higher latency, which is higher for
the build process, the process which requires a lot more metadata calls.
And also the throughput decreases. So this is something we are still working on. Now for future plans,
we are planning to reintroduce the metadata cache on the client end because service like
Samba is very metadata heavy. So we think that the performance can be improved by adding a metadata
cache on the client end. However, this right now is blocked because we require we have invalidation
calls which are allowed from SFF, but these are asynchronous. These are just kept in the queue
and it is invalidated on the SFF end which opens a window for data corruption. So we are in talks
with the SFF developers to have a synchronous invalid invalidation call back calls added to
SFF. So once that is in there, we would be able to implement a metadata cache and hopefully
improve performance. There are also considering other options for the connection between the
proxy library and the demon. Just last week, we tested using shad memory and a mutex for serialization.
However, the performance gains we noticed was quite marginal. So it wasn't too good. So
we are still considering other ideas, but that's still under development right now.
And finally, we only support those low-level APIs which are used by the VFS SFF new model.
So going forward, we expect to add more of these low-level API calls.
Yep, that's it. Thank you very much.
Oh, yeah, any questions please?