Ceph » CephFS

This is with reef branch - none of the PRs seen to be responsible for this. PRs in the test branch are:

https://github.com/ceph/ceph/pull/53497 - reef: mds: fix deadlock between unlinking and linkmerge
https://github.com/ceph/ceph/pull/53548 - reef: mds: log message when exiting due to asok command
https://github.com/ceph/ceph/pull/53553 - reef: qa: lengthen shutdown timeout for thrashed MDS
https://github.com/ceph/ceph/pull/53558 - reef: mds: add event for batching getattr/lookup
https://github.com/ceph/ceph/pull/53572 - reef: mgr/volumes: Fix pending_subvolume_deletions in volume info
https://github.com/ceph/ceph/pull/53624 - reef: qa/cephfs: switch to python3 for centos stream 9
https://github.com/ceph/ceph/pull/53661 - reef: ceph: allow xlock state to be LOCK_PREXLOCK when putting it
https://github.com/ceph/ceph/pull/53666 - reef: client: move the Inode to new auth mds session when changing auth cap
https://github.com/ceph/ceph/pull/53825 - reef: qa/cephfs: fix ior project build failure
https://github.com/ceph/ceph/pull/53827 - reef: qa/cephfs: fix build failure for mdtest project
https://github.com/ceph/ceph/pull/53906 - reef: qa: move nfs (mgr/nfs) related tests to fs suite
https://github.com/ceph/ceph/pull/53918 - reef: pybind/mgr/volumes: log mutex locks to help debug deadlocks
https://github.com/ceph/ceph/pull/54035 - reef: mds: adjust pre_segments_size for MDLog when trimming segments for st
https://github.com/ceph/ceph/pull/54037 - reef: qa: assign file system affinity for replaced MDS

There seems to be evidence that the mds in question was stopping:

  -173> 2023-10-19T10:26:44.832+0000 7f9ab645e640  5 mds.beacon.b Sending beacon up:stopping seq 118
  -143> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640  7 mds.1.cache shutdown_pass
  -134> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640 10 mds.1.log trim_all: 2/0/0

We could just add another test for "mds->is_stopping()" in MDCache::dispatch_fragment_dir but I don't think that's the right place. It's already too late: we shouldn't have gotten there after we've decided to shut down and trim the log. I wonder why the balancer was allowed to queue the merging operation after the shutdown pass:

  -111> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640  7 mds.1.626 shutdown_pass=false
  -110> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640 10 mds.1.bal operator() merging [dir 0x60e.000* ~mds1/stray4/ [2,head] auth v=26894 cv=26894/26894 state=1073741825|complete f(v20 m2023-10-19T10:13:30.448411+0000) n(v33 rc2023-10-19T10:13:30.448411+0000) hs=0+0,ss=0+0 0x558d7f7b2900]
  -109> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640 10 mds.1.bal operator()   all sibs under 001* [0x558d83f87b00] should merge
  -108> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640 10 mds.1.bal operator()   all sibs under 01* [0x558d83f87f80,0x558d83f88400] should merge
  -107> 2023-10-19T10:26:45.696+0000 7f9ab6c5f640 10 mds.1.bal operator()   all sibs under 1* [0x558d83f88880,0x558d83f88d00,0x558d83f89180,0x558d83f89600] should merge

Apparently, that was called by a timer event:

void MDBalancer::queue_merge(CDir *dir)
{
...
    mds->timer.add_event_after(bal_fragment_interval,
        new LambdaContext(std::move(callback)));

We could avoid scheduling the merge if `mds->is_stopping()`, but maybe we should just cancel (or run?) all timers before calling the shutdown pass on the cache?

Leonid Usov wrote:

There seems to be evidence that the mds in question was stopping:

[...]

Right, mds_thrash is adjusting max_mds causing MDS to stop.

We could just add another test for "mds->is_stopping()" in MDCache::dispatch_fragment_dir but I don't think that's the right place. It's already too late: we shouldn't have gotten there after we've decided to shut down and trim the log. I wonder why the balancer was allowed to queue the merging operation after the shutdown pass:

[...]

Apparently, that was called by a timer event:

[...]

We could avoid scheduling the merge if `mds->is_stopping()`, but maybe we should just cancel (or run?) all timers before calling the shutdown pass on the cache?

I think adding a check in the callbacks for queue_split/queue_merge is appropriate.