Bug #57072: Quincy 17.2.3 pybind/mgr/status: assert metadata failed - CephFS - Ceph

Actions

Copy link

Bug #57072

open

Quincy 17.2.3 pybind/mgr/status: assert metadata failed

Added by cld slnt over 1 year ago. Updated over 1 year ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Venky Shankar

Category:

Correctness/Safety

Target version:

Ceph - v18.0.0

% Done:

Source:

Community (user)

Tags:

backport_processed

Backport:

pacific,quincy

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Component(FS):

Labels (FS):

Pull request ID:

47814

Crash signature (v1):

Crash signature (v2):

Description

`ceph fs status` return AssertionError

Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
assert metadata
AssertionError

Related issues 2 (0 open — 2 closed)

Actions

Copy link

Updated by Venky Shankar over 1 year ago

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

Actions

Copy link

Updated by cld slnt over 1 year ago

Venky Shankar wrote:

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

I can`t reproduce this error in new-deployed cluster, here are the debug mgr logs may help:

Aug 23 16:34:24 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518087: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(audit) log [DBG] : from='client.75954 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr handle_command module 'status' command handler threw exception: 
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr handle_command module 'status' command handler threw exception: 
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
                                     File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
                                       return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
                                     File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
                                       return self.func(mgr, **kwargs)
                                     File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
                                       assert metadata
                                   AssertionError
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
Aug 23 16:34:26 u2 conmon[2121]:     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
Aug 23 16:34:26 u2 conmon[2121]:     return self.func(mgr, **kwargs)
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
Aug 23 16:34:26 u2 conmon[2121]:     assert metadata
Aug 23 16:34:26 u2 conmon[2121]: AssertionError
Aug 23 16:34:26 u2 conmon[2121]: 
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518088: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s

Actions

Copy link

Updated by Venky Shankar over 1 year ago

cld slnt wrote:

Venky Shankar wrote:

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

I can`t reproduce this error in new-deployed cluster, here are the debug mgr logs may help:

[...]

I think the assert is not valid. Daemon metadata is asynchronously fetched by ceph-mgr. I'll send a fix. Thanks.

Actions

Copy link