Venky Shankar wrote:
Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).
I can`t reproduce this error in new-deployed cluster, here are the debug mgr logs may help:
Aug 23 16:34:24 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518087: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(audit) log [DBG] : from='client.75954 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr handle_command module 'status' command handler threw exception:
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr handle_command module 'status' command handler threw exception:
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
assert metadata
AssertionError
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
Aug 23 16:34:26 u2 conmon[2121]: File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
Aug 23 16:34:26 u2 conmon[2121]: return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
Aug 23 16:34:26 u2 conmon[2121]: File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
Aug 23 16:34:26 u2 conmon[2121]: return self.func(mgr, **kwargs)
Aug 23 16:34:26 u2 conmon[2121]: File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
Aug 23 16:34:26 u2 conmon[2121]: assert metadata
Aug 23 16:34:26 u2 conmon[2121]: AssertionError
Aug 23 16:34:26 u2 conmon[2121]:
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518088: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s