Ceph » CephFS

In general: yes! This is a long-desired addition to our introspection and debugging abilities.

But some questions to make sure we've looked through things:
What do the session dump commands output, and what's missing? (I know they include some of this info around pending revokes, but I think not per-cap requests?)
What does the cache dump include, and what's missing? (It should have the MDS-side state but I think not ongoing revokes.)

In general tracking MClientCaps status is a big hole; they didn't get fitted into the OpRequest tracking infrastructure and I suspect we'd need that or something similar to satisfy this.

Greg Farnum wrote:

In general: yes! This is a long-desired addition to our introspection and debugging abilities.

But some questions to make sure we've looked through things:
What do the session dump commands output, and what's missing? (I know they include some of this info around pending revokes, but I think not per-cap requests?)

I didn't see a session dump command, but there is a session ls. It has some high level counts for recall_caps and release_caps, but otherwise it doesn't tell you enough detail. You might be able to use it to ID problematic clients though.

What does the cache dump include, and what's missing? (It should have the MDS-side state but I think not ongoing revokes.)

dump cache does have info about caps, but it's going to dump out every inode in the cache, which makes the signal to noise ratio pretty bad. On my vstart cluster, it's 56M of text from one of the 3 mds's. On a large, long-running cluster we could have to sift through (and download) gigabytes of text.

What I think we need is a way to be able to look at a glance and see what's happening with cap activity. In particular, any caps where
pending != issued != wanted ?

That might indicate places where the client hasn't caught up to what the server wants (or vice versa), and those are of particular interest for debugging.

As far as what it would print, the info in the client_caps field in the dump cache output would be good, and maybe something about the lag time. Alternately, we could just have it dump a list of inodes that are of interest and use dump inode command to get the rest? IDK what would be best, really.

In general tracking MClientCaps status is a big hole; they didn't get fitted into the OpRequest tracking infrastructure and I suspect we'd need that or something similar to satisfy this.

Yeah, for the userland client in particular, that's going to be a project. The kclient has debugfs. We have some info in a "caps" file in there already, but it could be expanded (knowing about outstanding cap refs would be nice) and maybe reformatted to better highlight the ones that are undergoing changes.