Hey folks, one welcome to this talk, good afternoon.
My name is Venki.
I am the tech lead for the file system stream and I have data, my colleague.
We both work with IBM on the file system stream and this talk is about command auditing
framework for quicker cluster rescue.
Let's see what this is.
Okay, so anybody who has operated or served cluster knows there are debugging challenges.
As long as things are working fine, it's good, but as you hit a roadblock, cluster warnings
and things like that, that's the time that you want to kind of see what's going on with
the cluster.
You want to be able to debug it, you want to know what's going on.
And served debugging is not straightforward.
It presents significant challenges.
Sometimes it requires just going through the talks, sometimes not.
You want at times when the issues are complex, you would want to get in touch with the community
and the actual developers and see what's going on.
So it's not straightforward, right.
And on top of that, coming from a file systems point of view background, sets of various
challenges are even more challenging.
There are lots of ways to change cluster state, to play at one with different bits like
the MDS map and different files as a components.
There are not a handful, but like a kitchensync of commands, I would say.
And each command has arguments, many arguments, and if you were able to mistakenly swap,
you don't know what you did.
We have placed in save gods, in commands like, which are riskier to operate, something
like, yes, I really, really mean it, and that deliberately makes the user or the cluster
operator to think that this is dangerous, or should I still execute.
But it's still abused.
We have seen, you know, send reports from folks who have, do you copy-based commands
and it happens all the time.
The other self-refest challenges, you know, we have a set of disaster recovery tools.
So for some reason, if you lost a PG in self, you know, a placement group, and that
placement group was probably holding, like, some of the MDS meta-tara, you know, your toast,
because, you know, you recover the PG, lost PGs, and then you, MDS meta-tata is gone,
your file system is gone, right.
So you have to now run a set of disaster recovery tools to rebuild them.
Unfortunately, those don't do not leave any trace of what's going on, you know, so it
becomes extremely challenging, even with the disaster recovery tools, the order of execution
matters.
So, you know, our dogs are well laid out, yes, but sometimes, you know, you know, folks
just execute things out of order and that leads to loss of complexity.
Okay, so, let's see, let's see a scenario, like a self file system issue is reported
on the user's list.
How does one go about it?
How would I go about it?
Or, you know, when I see an issue in the mailing list, whatever I do, some of the reports
are detailed, some are not, unfortunately, you know, we can't expect everyone to have, like,
the most continuous, more details in the, in the, in the, in the mailing list, or in a
slack, or I see a whatever.
So, we have to go on a, uh, back and forth with users to see what was done, right, what
commands were run, or the hardware's correct, uh, and not to blame the user, not blaming
anyone, you know, uh, sometimes the, the, the, the, the detail we get back is very, very
tells, uh, so we have to work with that, um, so once we have that all, all that information
what we, we, we, we, we do some kind of a post-mortem, yes, uh, we tried to help the user.
So, for that, what I, I would look, right, as a, as a self, as developer would be the exact
sort of commands that were run and their completion state.
The completion state is very important, because if a particular, uh, command was run, say
manipulating the MDS size, right, the number of MDS is, and that didn't run to completion,
I want to know that, right, because it makes the debugging, um, my, my debugging is based
on that context.
Um, and at times, I also want to know the MDS map, MDS map is basically, uh, you know,
a metadata about what MDS is, are in the cluster, which is handling the file system, and
what are the state of each of the MDS's, and many other bits, but I also want to know
that, right.
So, my debugging is now, I'm building up the context for debugging.
Um, I also, more, most importantly, as I mentioned, want to know if disaster recovery was
a way executed on the file system.
Um, you know, there have been cases where like some, uh, someone got a broken self of
cluster, because of a PG loss, ran disaster recovery, recovered it, uh, and possibly a couple
of months down the line.
Some issue happened, but, you know, it was related to disaster recovery.
So, I want to know that, right, and who remembers like a couple of, you know, months down
the line, what they did.
Okay.
So, especially for like, you know, uh, performance issues, I also want to have like a periodic
set of problems.
So, this is what I'm looking at, you know, at a broader sense.
Um, so, question is, how much of history am I looking at or anyone looking at?
cluster commands, whatever commands were, and like recently executed, right.
MDS map, I wanted at bounded time frame, maybe, you know, like when upper particular
issue started showing up, I, I want from that.
Of dumps, again, I want to know only, I want to know the perv dumps, uh, when, when, and the
certain performance issues started appearing.
Disaster recovery, three, I want to know it as far as possible.
I want to know if it was ever run, maybe five years ago, maybe ten years ago, but I
want to know that, because that changes my entire deeper gain context.
Okay.
So, we have some kind of an audit logging in, in, in, in, in, in, in, in, in, in, in,
and it's implemented in self-monitor.
So, let's see if, you know, talk a bit about that to see if it's enough or not.
So, the, so, this, this, this component in the monitoring, self-monitor is called the
monitor logs, and this is what it does, or it provides.
It provides a service where self-demons can set custom logs to the monitor.
These are not, these are like audit logs.
They can be like small strings, uh, to the monitor for, for, according, um, you know, uh,
all the demons in self have, you know, make use of monitor client to talk to the monitor,
to authenticate, that's the very first step.
So, the same monitor client can use this in these audit logs.
These are it logs when the monitor receives it, it is stored in Paxos, uh, so the Paxos
service for, in monitor that deals with these audit logs is called the log monitor.
And these are it logs are stored in the monitor DB store.
These are like local storage, right.
And crucial part is these are periodically trimmed.
Um, you know, if the monitor did not, do not trim it store, it would be, if it
is growing forever, and consume this space, and eventually run out of this space, we
don't want that.
So, they speed on the trimming for the, uh, for the audit logs and the monitor.
So, these are the downsides, right.
The use of local storage for storing these audit logs is counterproductive.
You know, I don't want it if I lose it, I, I lose all the audits.
Logs are periodically truncated to mitigate on bounded growth as I told, um, you can't
leave it on bounded growth.
So, we have to trim it, um, you know, uh, periodically.
Uh, so if say disaster recovery tools started to send these audit logs to self monitor, it
would be recorded, yes.
But I won't know if it was ever, disaster recovery was a very secure 10 years ago, because
the monitor would have trimmed it.
That is crucial for my debugging, but we don't have that information.
Another idea that came up was why doesn't the monitor just store the audit logs in
radios, right?
Um, that is a possibility, however, you know, monitor cannot really talk to radios.
That will lead to like very bad, uh, feedback loops, where, you know, the monitor to,
uh, dumb something radios has to talk to itself, uh, and again talk to radios.
So, these, this is like a cyclical dependency that happens, uh, and that really cannot, you
know, work.
And we're adding the log audit log itself, uh, these can be small strings, they are acceptable.
But I don't want to just need, you know, small strings of our, of the commands.
I want to know the MDS map periodically, I want to know the Perf terms.
I want to be able to, you know, store nice JSON dumps, so that I can have access to them.
Uh, and even if I was able to do that, right, um, I don't have a mechanism to read it
back nicely.
There are commands, so that can pull off these, uh, these, these, or these logs from monitor,
uh, but they are not flexible.
I can't say, give me, give me like, the logs from last month, uh, to made of this month.
No, I can't say that.
So it's very crude.
It is there, but it's very crude.
So these are the challenges, uh, um, right, and the monitor has the log store, uh,
but commands may be distributed between the monitor and the manager, right?
We have the self manager component, some commands execute and monitors, some commands, and
self, uh, manager.
Um, so the challenges were any of the commands unsuccessful.
I don't know that.
I want to not be precise, arguments, I want to not disaster recovery state, and I also want
to somehow correlate a chronological sequence of audit of what was executed on a cluster.
So these are the challenges.
Okay, um, the proposal is, and there's a feature that we have been working on is decentralized
audit logging.
Um, so these are the audit logs that are stored in rados, in one of the rados pools,
and we'll introduce that, uh, by the self manager.
Uh, and this, and we'll provide an structured chronological order of command history.
Um, with this, it's possible to track the precise sequence of disaster recovery steps, uh,
monitoring execution of commands, you know, how much time to take, um, you know, we also
record every single information like what flags were used, what other type, uh, did the
command train to completion, and was like the dangerous flaglessly, really, really mean a flag was used.
Um, and also, you know, query mechanism to, to, to pull out commands which failed,
timestamps, uh, structured entries, uh, and also, you know, uh, with, with all this information,
right, it's, you can, you can imagine debugging becomes much more, uh, easy.
Uh, all I have to do is, you know, when a, when a user reports an issue, uh, I have to just pull
out, tell a command to the user to pull out, they say the last one month of command log history,
and probably, some other steps, uh, but I have all the information for debugging.
Uh, so just really quick log monitor, you know, uh, um, uh, uh, this already I spoke about, um,
it's there for, it's, uh, the, the audit command thing is there in the monitor, but we can't make
use of it. Um, we have all the things in log monitor like, you know, who is sending, which
demoness sending, with sequence, with channel, what time, and for every single thing.
Okay. So, now, the idea to hear is, the monitor cannot dump in, in radar. So,
there has to be a way for the monitor to relay commands to the chef manager, so that it's
chef manager can now go and, uh, record it in rados. So, there is something called an audit
for log relay. Log monitor supports subscribing to a log channel. It is already there. Okay.
Um, and the subscribers can request a one-time digest of logs, and after that, once you request
to digest of logs, the monitor will be keep on sending you incremental, uh, audits. Um, after a
access proposal is made, uh, so the simplest thing to do is for the chef manager to subscribe
to the monitor log. So, there's, so this, there's a new manager module for it. It's called audit
command, and there's a new pool, dot audit pool, um, when the auditman starts up, very first time
it, uh, subscribes itself to the log monitor channel, on the audit channel, uh, and once it subscribes,
the monitor will keep on modifying it to, for the audit logs, and the auditman, uh, module,
all it does is talks to, um, talks to rados, why, uh, SQLite, because all these audit laws are
going to end up in the pool, in a database, and the database is SQLite, which is back, which,
which, which, which, which, which the backing story story story was. Um, so the monitor, the manager
will be changed to use this particular framework, and we start recording it in SQLite.
Um, and I'm going to hand it over to Dharia, uh, for the database snippets.
I'm a audible. Thanks. So, um, to store the audit logs right, we would need, we would be
making use of the, uh, lips of SQLite, uh, audible extension. Um, it provides a, uh, to create a
SQLite via first, which would be, uh, communicating with the, uh, interfacing with the SQLite databases,
which are stored in rados. Um, and one of the reasons to use, uh, the, uh,
self-acquilite, uh, managed SQLite databases is that, uh, it's one of the suffix, suff, suffix
related to, uh, sort of, you know, allow multiple clients to, uh, access the database in a serial fashion,
uh, which would be managed by the rados logs, uh, you know, been provided by the self-acquilite
via first, for lips of SQLite, that's a link down below. So, um, let us discuss the core part of the
audit logging framework, which is to flow the commands from the demons to the databases.
The framework would be making use of a new pool called the audit pool to store the databases and
the audit logs. Um, all the self-CLI commands, whether it's the monitor command or the, uh,
manager command would be routed via the manager, the reason being that the monitors can
automatically communicate with the rados, right? So, um, from the manager, the audit logs would be
reaching the respective databases. Um, the manager would be interacting with the, uh,
invoking the SQLite APIs from the library to execute the SQL queries and, uh, record the commands
into the respective databases in the audit pool. So, here's how it would look like. Um,
we start from this FCLI from there, um, you have the command. If, if it's a M. J. Command,
and it, the, the monitor will edit it and it's over to the, uh, a self-managed demon,
and from the manager demon, it works the, uh, SQLite 3 library and it routes via the SQLite
VFS and then the monitor log is committed to the underlying M. J. Audit, TB, which is persisted in the
audit pool. Um, if it's a monitor command or any command coming from the other entities,
it's stored in the PXO store, um, and, uh, it's, uh, on demon-based this thing, um, the manager would
be requesting for the audit logs, um, making use of the log monitor, which would be fetching the
log summary from the PXO store and it, uh, the log monitor would process it to fetch the log entries
and return the audit logs to the manager and then the flow is same as the M. J. Audit logs.
So, this is how it would be. Um, so one of the, uh, as Venki discussed earlier that,
getting the, or, you know, persisting the, uh, disaster recovery usage across the cluster is very
important. So, but the disaster recovery tools are quite different from the, uh, demon commands,
um, they need to be worked directly via the binaries. So, we are going to do this with, um,
the usage of a separate database would be reserved for the, uh, the recovery tools, all the recovery
tools, whether it's the channel tool or the data scan tool would be making use of the same
of land, say, of land tools stored all the to DB and all the tools will be linked to the
lip-saccharacculate to facilitate this, um, the link to the disaster recovery tools is down below.
Um, this is how it would be for the offline tools. Um, start from the offline tool, the
information of the offline tool, um, the offline tool would try to access the underlying data
service via the SQL activity library and the, uh, SQL at VFS, the self SQL at VFS layer,
um, and once the connection has been established, we would start writing the SQL logic,
which is, which could be the suffocational tool journal or header or any event into the underlying
audit pool. And, um, once we are done writing it, uh, the tools cannot, uh, delete or close or log
the DB, they can just release the memory buffers, which would be the data. The sendals and some
other stuff, the memory, uh, the structure is that would be using. And, um, so, yeah, this is how
it would be for the offline tools. So, um, the current DB schema, the current audit log, uh,
would be consisting of six rows, uh, six fields. First one would be the, uh, sequence number,
which would be, uh, fetched from the monitor for the, uh, manager or the offline tool logs.
Then there's the command, which is executed in the in a time and the completion time, which would
be, uh, in timestamps, um, then there's the status of the command, whether it has passed or
it failed or it's pending. Um, and then there's the return value, which is the retwelled from
the command execution, uh, auditman to the rescue. Um, so the auditman is going to be a,
uh, staff manager, uh, plugin, and it would be used to access the audit logs. So, the key features
includes, uh, rich and extensive set of commands that would be able to fetch and retrieve,
uh, audit logs and different formats and different ways. Um, it is capable of querying, uh,
audit databases from, uh, for the, uh, the manager or the monitor or the offline tool DB. Um,
and there is also retention policy in place, uh, which would keep only the last key, uh, logs,
uh, in order to not breach the limit and overgrow the size of the databases. Um, you can be said
why I just command, uh, staffer, the audit attention, add or the DB and then the log, count, uh, say,
say, you just want to add, retain last 100 kilox, you can do that. So, um,
this is how it would be, uh, if you start from the self CLI, um, the, uh, the command,
a self FS, uh, MGR or whatever, it which is self MGR via monitor, um, and once that is done,
the audit plugin would be constructing the URI, because we need to ensure what that happens, we need
to access, whether it is monitor or manager or any disaster recovery. And once that is done,
we need to append some stuff like the dot audit and also the URI would look something like this,
um, file, type of slash, an audit DB and then the, uh, self which is going to be the skillet
via FS. And then it would make use of a read-only handle to, uh, communicate and establish the
connection with the, uh, audit pool, um, and once that is done, we would be executing this
led query to read from the, uh, respective database and then the audit plugin would be formatting,
uh, the, it tried data and then send it to the CLI. So, uh, this is some of the, uh, commands,
um, the command by default only fetches the last 100 records, obviously for sanity, uh, by default,
it would be just a surface audit MGR. TMGR over here is the, uh, demon, sorry,
it is a place order for the DB that you want to use here, we are trying to fetch, uh, the audit
logs for the MGR demon, uh, it would retry all the six, uh, fields and, uh, show it to you in
a JSON or a blog, um, and then there are various ways to, uh, retry commands say if you want to,
say fetch, uh, last 1000 queries, but you want to only do it after a particular sequence or before,
uh, sequence and you also have the count, which would return you, if, if, come on, what was the number
of times the command was executed, and that's just brief, just brief, the ID, the sequence number,
uh, the command is set as the recent N, which means last 100, 1000 or 1 liter cake, uh, logs,
and then if you want to fetch only a particular fields, like sequence number, the command was
the count time, you have that, and you can also audit based on the time, whether you want to do it
from a particular date or to a particular date, you can do that, um, you also have the range
from this day to the next, the, from D1 to D2, um, you can also order the logs,
via the order, by field and order, and we'll then also, by the S&U descending order,
if you want to fetch, maybe to last a day log, or maybe last a year's log, you can do that,
and we can also fetch, uh, the audit logs, only a pack, uh,
literally the field, or maybe all the past commands, or all the earning commands,
and we can fetch the audit logs by Jason, which is pretty, so we can also, uh, so, um,
there are commands, which might need no correlation, uh, the auditman makes use of log monitor to
fetch the monitor logs, of lentils, um, like direct use of the, uh, the instance with the
SQL ADB. Um, if you want to correlate commands from various sources, auditman can reconstruct
the, um, event sequence by matching the records, um, often with the criteria, maybe say any time,
which usually is the case, um, this is done by the compile command, uh, say if you want to, uh,
uh, uh, you know, compare the MGR and the monitor logs based on the initiation time,
uh, between the range, the first of 10 to 30s of 10, with delta speed being 1 seconds,
you'd be getting something like this, uh, so say there's a first scrub command, which is executed,
and which is managed by the, uh, MGR demon, you would have something like this. Um,
extending the, uh, entire audit logging framework beyond just commands, um, right now, uh,
it only, uh, stores commands in the, uh, databases, but, um, demons like MDS, uh, could make use
of this by dumping the MBS state, when the file sits system encounters on degradation or maybe
subordinates or so, both dumps can also be added, uh, by demons, if any OSD or placement group,
degradation occurs, um, this can help us not only just for surface auditing, but for, uh,
a safe wide auditing, um, but yeah, this starts with surface and, uh, in future, we'll be trying to
implement it over the demons as well. So, yeah, that's it. Any questions?
Yep. What's the timeline for the, uh, what's the, uh, what's the, uh, what's the, uh,
uh, what's the, uh, what's the, uh, what's the, uh, what's the, uh, what's the, uh, what's the, uh,
so this is currently, uh, yeah, so the, I think the question is that what's the timeline,
this would be part of the area is right, uh, so right now it's currently working progress,
but we're eye for umbrella, I guess. Yeah, so there is currently a working progress, uh,
peer that I have, it's a tough peer. I don't have the link handy, but, uh, yeah, so umbrella, yep.
Any other questions? Yeah. Sure. Yeah. Uh, um, uh, uh, uh, uh, very cool, uh, uh, uh, I like the idea.
Thank you. Um, uh, maybe you said, you talked with, could you maybe repeat this there, like,
anyway, like, I can customize what should be locked and how would that look like?
So, you want to customize the fields that we, uh, store in the database, right?
So, right now it is a, yeah. So, right now it is a fixed set that will include it is just
six fields like the sequence number the command, the initiation time, the completion time,
the status and the hardware, but in future we have been V2, we might incorporate this
but thanks, that is a nice advice. Thank you. Thank you.
We have, well, this module of the module and all the whole thing will be enabled by default
with always a module for example, at the manager in the manner of configuration.
Do you visit this care intention to enable this like the logging of all commands?
So, the question is that whether the auditing would be enabled by default, we would like
to that it stays enabled by default, but I guess we can make it an obtain as well.
The thing is it needs a deliberate switch on, we use it because there is a data space in
mode, right. There is a conjunction of cluster space. Although we expect it not, we do huge,
maybe you can use irresa code it will go back because time is not, you know, the speed is not
so it needs to be deliberately switched on. But if, you know, we see if the courage for this feature
is the whole offset, then we might like, let me get all this on, like, the one we are talking about.
I remember Patrick mentioned that the next step might be simple to repair and MBS getting
all the knowledge from possibly this database and get to motor repair at the same time.
So, yeah, that's something that I've been talking for on my own, where you can ask for my hand and just
yeah. So, Dan, do mention that one of the things that MBS would be nice and doing is to
execute some of the disaster recovery steps itself. So, if there is some failure,
it knows what the type of failure is instead of having the user run it by themselves, the MBS can
auto recover itself. But it's something we have been thinking about, but not yet planned.
This takes it like, with this, the MBS has all the required information of what could have
gone wrong. But it will only do it if it's 100% sure, you know, if it's not, if it's not even one,
like 99% if it thinks it's, you know, it knows the problem but it's still not sure,
it should still not go into it. So, and that's the thing. But all this makes it easier for the MDS
and also for us as developers and support folks to reduce the time to, you know, bring back
a file system online. So, because there's lots of back and forth. I know, yeah,
I'm so proud, somebody who ran a command with, yes, I really mean it, but never told us.
Yeah, not deliberate, but it happens.
Thanks for watching.