Bug #41549: mgr: ActivePyModules::list_servers_python() returns mds with empty hostname - mgr - Ceph

Actions

Copy link

Bug #41549

open

mgr: ActivePyModules::list_servers_python() returns mds with empty hostname

Added by Kiefer Chang over 4 years ago. Updated about 4 years ago.

Status:

New

Priority:

Normal

Assignee:

Ernesto Puerta

Category:

ceph-mgr

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v15.0.0

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Use vstart.sh to create a testing cluster (test on master e81ef76cae66d95af4725cdd81743f68f2e0593d), following is result of cluster status:

bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2019-08-28T07:31:23.492+0000 7fbf1a5b2700 -1 WARNING: all dangerous and experimental features are enabled.
2019-08-28T07:31:23.496+0000 7fbf1a5b2700 -1 WARNING: all dangerous and experimental features are enabled.
  cluster:
    id:     b7d4c8ca-9a64-4721-852e-587960f1a475
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum a,b,c (age 12m)
    mgr: x(active, since 12m)
    mds: a:1 {0=a=up:active} 2 up:standby
    osd: 3 osds: 3 up (since 11m), 3 in (since 11m)
    rgw: 1 daemon active (8000)

  task status:
    scrub status:
        mds.0: idle

  data:
    pools:   6 pools, 56 pgs
    objects: 225 objects, 6.4 KiB
    usage:   6.0 GiB used, 3.0 TiB / 3.0 TiB avail
    pgs:     56 active+clean

Load a MGR module: e.g. prometheus (or disable and enable dashboard module) to make MGR daemon respawn.
```
bin/ceph mgr module enable prometheus
```
On Dashboard's Cluster->Hosts page, we can see a row with empty hostname and it's service is mds.0

Some notes:

This happens when mgr daemon is respawned.
mds.0 is a task for mds scrubbing (seems introduced in this change)
In this test, we have 1 active mds daemon mds.a, two standby daemons mds.b and mds.c. They are not reported in list_servers_python(). Occasionally, I can see mds.b and mds.c are reported, but not mds.a
There is an issue about list_servers_python() reports mgr with empty hostname some time ago.

In mgr daemon's log, there are some unhandled messages (these messages can be seen without restarting MGR daemon)

2019-08-28T07:43:47.620+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705430e00 mgrreport(mds.b +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6828/1605008353
2019-08-28T07:43:47.620+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705441880 mgrreport(mds.c +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6830/2039538527
2019-08-28T07:43:48.420+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v313: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
2019-08-28T07:43:50.420+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v314: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
2019-08-28T07:43:52.424+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v315: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail
2019-08-28T07:43:52.624+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x558705520000 mgrreport(mds.b +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6828/1605008353
2019-08-28T07:43:52.624+0000 7f7d65da4700  0 ms_deliver_dispatch: unhandled message 0x55870534bc00 mgrreport(mds.c +0-0 packed 6) v8 from mds.? v2:192.168.15.191:6830/2039538527
2019-08-28T07:43:54.424+0000 7f7d64da2700  0 log_channel(cluster) log [DBG] : pgmap v316: 56 pgs: 56 active+clean; 7.5 KiB data, 3.0 GiB used, 3.0 TiB / 3.0 TiB avail

Files

mds_empty_hostname_01.png (63.9 KB) mds_empty_hostname_01.png

Kiefer Chang, 08/28/2019 07:35 AM

Related issues 2 (0 open — 2 closed)