Ceph » CephFS

The upgrade process uses `fail_fs` which fails the file system and upgrades the MDSs without reducing max_mds to 1. I debugged this a bit with Milind and it does seem like the MDS did not receive the updated map and failed to transition to a rank.

Milind Changire wrote:

[...]

The command for fs set joinable true when executed by the mgr reaches the mon, but the mon fails to broadcast the mdsmap update leading to all mds remaining in up:standby for this specific run.

The MDS do not receive an updated broadcast because they've not been assigned a new file system; i.e. they are up:standby.

The real question is why do the mons not assign any of the standbys to ranks.

NOTE: This is an upgrade scenario

Here's the log from the mon which is handling the fs set joinable true command from the mgr:
[...]

Few issues:

- This upgrade test is going from pacific to main. This is an N-3 to N upgrade.
- The problem seems to be FSMap::get_available_standby is failing because:

https://github.com/ceph/ceph/blob/9fedc1e062027dbce66747e5d0dc11319615ab8a/src/mds/FSMap.cc#L769-L772

The recent addition of the minor log segment incompat bit caused that check to fail.

I'll work on a fix for the second issue.