Ceph » CephFS

2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] <== mds.? v2:172.21.15.103:6832/363807761 1 ==== mgropen(mds.a) v3 ==== 61794+0+0 (crc 0 0 0) 0x55f886447800 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 10 mgr.server handle_open from 0x55f886343680  mds.a
2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] --> [v2:172.21.15.103:6832/363807761,v1:172.21.15.103:6833/363807761] -- mgrconfigure(period=5, threshold=5) v3 -- 0x55f88653e600 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open updating existing DaemonState for a
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open  got config {auth_debug={2=true},auth_supported={2=cephx},chdir={2=},daemonize={5=false},debug_asserts_on_shutdown={2=true},debug_mds={2=20/20},debug_ms={2=1/1},keyring={0=$mds_data/keyring},lockdep={2=true},mds_bal_fragment_size_max={2=10000},mds_bal_merge_size={2=5},mds_bal_split_bits={2=3},mds_bal_split_size={2=100},mds_debug_frag={2=true},mds_debug_scatterstat={2=true},mds_op_complaint_time={2=180.000000},mds_verify_scatter={2=true},mon_allow_pool_delete={2=true},mon_clock_drift_allowed={2=1.000000},mon_cluster_log_file_level={2=debug},mon_host={2=172.21.15.103,[v2:172.21.15.112:3301,v1:172.21.15.112:6790],172.21.15.112},mon_max_pg_per_osd={2=10000},mon_pg_warn_max_object_skew={2=0.000000},mon_warn_on_crush_straw_calc_version_zero={2=false},mon_warn_on_legacy_crush_tunables={2=false},mon_warn_on_osd_down_out_interval_zero={2=false},mon_warn_on_pool_pg_num_not_power_of_two={2=false},mon_warn_on_too_few_osds={2=false},ms_die_on_bug={2=true},ms_die_on_old_message={2=true},osd_crush_chooseleaf_type={2=0},osd_default_data_pool_replay_window={2=5},osd_op_complaint_time={2=180.000000},osd_pool_default_erasure_code_profile={2=plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd},osd_pool_default_pg_autoscale_mode={2=off},osd_pool_default_size={2=2},pid_file={2=/var/run/ceph/$cluster-$name.pid},rbd_default_features={0=61}} ignored {}
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_open  got config_defaults_bl 60016 bytes
2019-11-04T13:27:55.820+0000 7fc7550d4700  1 -- [v2:172.21.15.112:6800/14153,v1:172.21.15.112:6801/14153] <== mds.? v2:172.21.15.103:6832/363807761 2 ==== mgrreport(mds.a +0-0 packed 6) v8 ==== 41+0+0 (crc 0 0 0) 0x55f885f32e00 con 0x55f886343680
2019-11-04T13:27:55.820+0000 7fc7550d4700 10 mgr.server handle_report from 0x55f886343680 mds.a
2019-11-04T13:27:55.820+0000 7fc7550d4700 20 mgr.server handle_report updating existing DaemonState for mds.a
...
2019-11-04T13:27:56.476+0000 7fc765441700  1 -- 172.21.15.112:0/14153 <== mon.1 v2:172.21.15.112:3300/0 293 ==== mon_command_ack([{"prefix": "mds metadata", "who": "a"}]=0  v13) v1 ==== 72+0+685 (crc 0 0 0) 0x55f886385680 con 0x55f886594900
2019-11-04T13:27:56.476+0000 7fc763c3e700  4 mgr finish mon returned valid metadata JSON for mds.a
...
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr get_metadata_python Requested missing service mds.a
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr handle_command module 'status' command handler threw exception: 'NoneType' object has no attribute 'get'
2019-11-04T13:28:00.016+0000 7fc7558d5700 20 mgr ~Gil Destroying new thread state 0x55f8860cd180
2019-11-04T13:28:00.016+0000 7fc7558d5700 -1 mgr.server reply reply (22) Invalid argument Traceback (most recent call last):
  File "/usr/share/ceph/mgr/mgr_module.py", line 975, in _handle_command
    return self.handle_command(inbuf, cmd)
  File "/usr/share/ceph/mgr/status/module.py", line 251, in handle_command
    return self.handle_fs_status(cmd)
  File "/usr/share/ceph/mgr/status/module.py", line 176, in handle_fs_status
    mds_versions[metadata.get('ceph_version', "unknown")].append(standby['name'])
AttributeError: 'NoneType' object has no attribute 'get'

From: /ceph/teuthology-archive/pdonnell-2019-11-04_12:55:15-fs-wip-pdonnell-testing-20191104.085350-distro-basic-smithi/4472376/remote/smithi112/log/ceph-mgr.x.log.gz

This is with the recent fix by Sage in #42494.

I don't see why the `daemon_state` would be empty here. The presence of this message in the output:

https://github.com/ceph/ceph/blob/5a15148b7bd2a2b7919fb873a6c18405524151b9/src/mgr/DaemonServer.cc#L410

would indicate that it's not empty.

I don't think this is related to #41538.