Project

General

Profile

Bug #57072

Quincy 17.2.3 pybind/mgr/status: assert metadata failed

Added by cld slnt over 1 year ago. Updated about 1 year ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
Venky Shankar
Category:
Correctness/Safety
Target version:
Ceph - v18.0.0
% Done:

0%

Source:
Community (user)
Tags:
backport_processed
Backport:
pacific,quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
47814
Crash signature (v1):
Crash signature (v2):

Description

`ceph fs status` return AssertionError

Error EINVAL: Traceback (most recent call last):
File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
return self.func(mgr, **kwargs)
File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
assert metadata
AssertionError

Related issues

Copied to CephFS - Backport #57728: pacific: Quincy 17.2.3 pybind/mgr/status: assert metadata failed Resolved
Copied to CephFS - Backport #57729: quincy: Quincy 17.2.3 pybind/mgr/status: assert metadata failed Resolved

History

#1 Updated by Venky Shankar over 1 year ago

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

#2 Updated by cld slnt over 1 year ago

Venky Shankar wrote:

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

I can`t reproduce this error in new-deployed cluster, here are the debug mgr logs may help:

Aug 23 16:34:24 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518087: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:25 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u3.wtcnic v2:10.23.58.10:6808/2750777995; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u3.wtcnic: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server handle_open ignoring open from mds.u2.ktvcld v2:10.23.245.200:6808/1792704188; not ready for session (expect reconnect)
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr finish mon failed to return metadata for mds.u2.ktvcld: (2) No such file or directory
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(audit) log [DBG] : from='client.75954 -' entity='client.admin' cmd=[{"prefix": "fs status", "target": ["mon-mgr", ""]}]: dispatch
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr handle_command module 'status' command handler threw exception: 
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr get_metadata_python Requested missing service mds.u2.ktvcld
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr handle_command module 'status' command handler threw exception: 
Aug 23 16:34:26 u2 ceph-mgr[2208]: mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
                                     File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
                                       return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
                                     File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
                                       return self.func(mgr, **kwargs)
                                     File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
                                       assert metadata
                                   AssertionError
Aug 23 16:34:26 u2 conmon[2121]: 2022-08-23T08:34:26.105+0000 7f7ffe482700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/mgr_module.py", line 1732, in _handle_command
Aug 23 16:34:26 u2 conmon[2121]:     return CLICommand.COMMANDS[cmd['prefix']].call(self, cmd, inbuf)
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/mgr_module.py", line 462, in call
Aug 23 16:34:26 u2 conmon[2121]:     return self.func(mgr, **kwargs)
Aug 23 16:34:26 u2 conmon[2121]:   File "/usr/share/ceph/mgr/status/module.py", line 109, in handle_fs_status
Aug 23 16:34:26 u2 conmon[2121]:     assert metadata
Aug 23 16:34:26 u2 conmon[2121]: AssertionError
Aug 23 16:34:26 u2 conmon[2121]: 
Aug 23 16:34:26 u2 ceph-mgr[2208]: log_channel(cluster) log [DBG] : pgmap v518088: 305 pgs: 305 active+clean; 521 MiB data, 4.6 GiB used, 295 GiB / 300 GiB avail; 341 B/s rd, 0 op/s

#3 Updated by Venky Shankar over 1 year ago

cld slnt wrote:

Venky Shankar wrote:

Could you provide more info on how you ran into this. A reproducer would really help (or at least the debug mgr logs).

I can`t reproduce this error in new-deployed cluster, here are the debug mgr logs may help:

[...]

I think the assert is not valid. Daemon metadata is asynchronously fetched by ceph-mgr. I'll send a fix. Thanks.

#4 Updated by Venky Shankar over 1 year ago

  • Category set to Correctness/Safety
  • Status changed from New to Fix Under Review
  • Assignee set to Venky Shankar
  • Target version set to v18.0.0
  • Backport set to pacific,quincy
  • Pull request ID set to 47814

#5 Updated by Rishabh Dave about 1 year ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot about 1 year ago

  • Copied to Backport #57728: pacific: Quincy 17.2.3 pybind/mgr/status: assert metadata failed added

#7 Updated by Backport Bot about 1 year ago

  • Copied to Backport #57729: quincy: Quincy 17.2.3 pybind/mgr/status: assert metadata failed added

#8 Updated by Backport Bot about 1 year ago

  • Tags set to backport_processed

Also available in: Atom PDF