Project

General

Profile

Bug #41549

mgr: ActivePyModules::list_servers_python() returns mds with empty hostname

Added by Kiefer Chang about 1 year ago. Updated 7 months ago.

Status:
New
Priority:
Normal
Category:
ceph-mgr
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

  • Use vstart.sh to create a testing cluster (test on master e81ef76cae66d95af4725cdd81743f68f2e0593d), following is result of cluster status:
    bin/ceph -s
    *** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
    2019-08-28T07:31:23.492+0000 7fbf1a5b2700 -1 WARNING: all dangerous and experimental features are enabled.
    2019-08-28T07:31:23.496+0000 7fbf1a5b2700 -1 WARNING: all dangerous and experimental features are enabled.
      cluster:
        id:     b7d4c8ca-9a64-4721-852e-587960f1a475
        health: HEALTH_OK
    
      services:
        mon: 3 daemons, quorum a,b,c (age 12m)
        mgr: x(active, since 12m)
        mds: a:1 {0=a=up:active} 2 up:standby
        osd: 3 osds: 3 up (since 11m), 3 in (since 11m)
        rgw: 1 daemon active (8000)
    
      task status:
        scrub status:
            mds.0: idle
    
      data:
        pools:   6 pools, 56 pgs
        objects: 225 objects, 6.4 KiB
        usage:   6.0 GiB used, 3.0 TiB / 3.0 TiB avail
        pgs:     56 active+clean
    
    
  • Load a MGR module: e.g. prometheus (or disable and enable dashboard module) to make MGR daemon respawn.
    bin/ceph mgr module enable prometheus
    
  • On Dashboard's Cluster->Hosts page, we can see a row with empty hostname and it's service is mds.0
Some notes:
  • This happens when mgr daemon is respawned.
  • mds.0 is a task for mds scrubbing (seems introduced in this change)
  • In this test, we have 1 active mds daemon mds.a, two standby daemons mds.b and mds.c. They are not reported in list_servers_python(). Occasionally, I can see mds.b and mds.c are reported, but not mds.a
  • There is an issue about list_servers_python() reports mgr with empty hostname some time ago.
  • In mgr daemon's log, there are some unhandled messages (these messages can be seen without restarting MGR daemon)
    2019-08-28T07:43:47.620+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705430e00 mgrreport(mds.b +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6828/1605008353
    2019-08-28T07:43:47.620+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705441880 mgrreport(mds.c +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6830/2039538527
    2019-08-28T07:43:48.420+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v313: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
    2019-08-28T07:43:50.420+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v314: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
    2019-08-28T07:43:52.424+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v315: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
    2019-08-28T07:43:52.624+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705520000 mgrreport(mds.b +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6828/1605008353
    2019-08-28T07:43:52.624+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x55870534bc00 mgrreport(mds.c +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6830/2039538527
    2019-08-28T07:43:54.424+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v316: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
    

mds_empty_hostname_01.png View (63.9 KB) Kiefer Chang, 08/28/2019 07:35 AM


Related issues

Related to mgr - Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname Pending Backport 03/09/2018
Related to mgr - Bug #20887: Services reported with blank hostname by mgr Resolved 08/02/2017

History

#1 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #23286: mgr: ActivePyModules::list_servers_python() returns mgr with empty hostname added

#2 Updated by Sebastian Wagner about 1 year ago

https://github.com/ceph/ceph/pull/20875/files from Jan just fixes a missing host name for mgrs
I think this is something different

#3 Updated by Sebastian Wagner about 1 year ago

  • Related to Bug #20887: Services reported with blank hostname by mgr added

#4 Updated by Ernesto Puerta 7 months ago

  • Assignee set to Ernesto Puerta

Hard to reproduce, but I'll be watchful at this on as we conduct QA for the next Nautilus downstream release.

Also available in: Atom PDF