Project

General

Profile

Bug #42635

mgr: daemon state for mds not available

Added by Patrick Donnelly 3 months ago. Updated 3 months ago.

Status:
Pending Backport
Priority:
Urgent
Category:
-
Target version:
% Done:

0%

Source:
Q/A
Tags:
Backport:
nautilus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
Crash signature:

Description

2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] <== mds.? v2:172.21.15.103:6832/363807761 1 ==== mgropen(mds.a) v3 ==== 61794+0+0 (crc 0 0 0) 0x55f886447800 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 10 mgr.server handle_open from 0x55f886343680  mds.a
2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] --> [v2:172.21.15.103:6832/363807761,v1:172.21.15.103:6833/363807761] -- mgrconfigure(period=5, threshold=5) v3 -- 0x55f88653e600 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open updating existing DaemonState for a
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open  got config {auth_debug={2=true},auth_supported={2=cephx},chdir={2=},daemonize={5=false},debug_asserts_on_shutdown={2=true},debug_mds={2=20/20},debug_ms={2=1/1},keyring={0=$mds_data/keyring},lockdep={2=true},mds_bal_fragment_size_max={2=10000},mds_bal_merge_size={2=5},mds_bal_split_bits={2=3},mds_bal_split_size={2=100},mds_debug_frag={2=true},mds_debug_scatterstat={2=true},mds_op_complaint_time={2=180.000000},mds_verify_scatter={2=true},mon_allow_pool_delete={2=true},mon_clock_drift_allowed={2=1.000000},mon_cluster_log_file_level={2=debug},mon_host={2=172.21.15.103,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.112},mon_max_pg_per_osd={2=10000},mon_pg_warn_max_object_skew={2=0.000000},mon_warn_on_crush_straw_calc_version_zero={2=false},mon_warn_on_legacy_crush_tunables={2=false},mon_warn_on_osd_down_out_interval_zero={2=false},mon_warn_on_pool_pg_num_not_power_of_two={2=false},mon_warn_on_too_few_osds={2=false},ms_die_on_bug={2=true},ms_die_on_old_message={2=true},osd_crush_chooseleaf_type={2=0},osd_default_data_pool_replay_window={2=5},osd_op_complaint_time={2=180.000000},osd_pool_default_erasure_code_profile={2=plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd},osd_pool_default_pg_autoscale_mode={2=off},osd_pool_default_size={2=2},pid_file={2=/var/run/ceph/$cluster-$name.pid},rbd_default_features={0=61}} ignored {}
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open  got config_defaults_bl 60016 bytes
2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] <== mds.? v2:172.21.15.103:6832/363807761 2 ==== mgrreport(mds.a +0-0 packed 6) v8 ==== 41+0+0 (crc 0 0 0) 0x55f885f32e00 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 10 mgr.server handle_report from 0x55f886343680 mds.a
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_report updating existing DaemonState for mds.a
...
2019-11-04T13:27:56.476+0000 7fc765441700  1 -- 172.21.15.112:0/14153 <== mon.1 v2:172.21.15.112:3300/0 293 ==== mon_command_ack([{"prefix": "mds metadata", "who": "a"}]=0  v13) v1 ==== 72+0+685 (crc 0 0 0) 0x55f886385680 con 0x55f886594900
2019-11-04T13:27:56.476+0000 7fc763c3e700  4 mgr finish mon returned valid metadata JSON for mds.a
...
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr get_metadata_python Requested missing service mds.a
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr handle_command module 'status' command handler threw exception: 'NoneType' object has no attribute 'get'
2019-11-04T13:28:00.016+0000 7fc7558d5700 20 mgr ~Gil Destroying new thread state 0x55f8860cd180
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 975, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/status/module.py", line 251, in handle_command
    return self.handle_fs_status(cmd)
  File "/usr/share/ceph/mgr/status/module.py", line 176, in handle_fs_status
    mds_versions[metadata.get('ceph_version', "unknown")].append(standby['name'])
AttributeError: 'NoneType' object has no attribute 'get'

From: /ceph/teuthology-archive/pdonnell-2019-11-04_12:55:15-fs-wip-pdonnell-testing-20191104.085350-distro-basic-smithi/4472376/remote/smithi112/log/ceph-mgr.x.log.gz

This is with the recent fix by Sage in #42494.

I don't see why the `daemon_state` would be empty here. The presence of this message in the output:

https://github.com/ceph/ceph/blob/5a15148b7bd2a2b7919fb873a6c18405524151b9/src/mgr/DaemonServer.cc#L410

would indicate that it's not empty.

I don't think this is related to #41538.


Related issues

Related to fs - Bug #41538: mds: wrong compat can cause MDS to be added daemon registry on mgr but not the fsmap Resolved
Related to fs - Bug #42917: ceph: task status not available Duplicate
Copied to fs - Backport #42713: nautilus: mgr: daemon state for mds not available Need More Info

History

#1 Updated by Patrick Donnelly 3 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Patrick Donnelly
  • Backport set to nautilus
  • Pull request ID set to 31400
  • Component(FS) MDS added

#2 Updated by Nathan Cutler 3 months ago

  • Related to Bug #41538: mds: wrong compat can cause MDS to be added daemon registry on mgr but not the fsmap added

#3 Updated by Patrick Donnelly 3 months ago

  • Status changed from Fix Under Review to Pending Backport

#4 Updated by Patrick Donnelly 3 months ago

  • Copied to Backport #42713: nautilus: mgr: daemon state for mds not available added

#5 Updated by Patrick Donnelly 3 months ago

  • Related to Bug #42917: ceph: task status not available added

Also available in: Atom PDF