Ceph » mgr

https://bugzilla.redhat.com/show_bug.cgi?id=1705464

It looks to me like `ceph status` is getting this state not from the ceph-mgr but from the MgrStatMonitor PaxosService. The difference between lspool and pool count being that the former is coming from osdmap while the later from the pgmap.

I started investigating this last week. I was only able to reproduce it once so far.

Still checking if we have an environment where this could be reproducible at will since I was only able to see the behavior once.

@Kefu Chai,

Just trying to understand it, but did you assign it to you to check if 'it is a dup' with what? Would you like to me still check on it?

Thanks,
-Daniel

not reproducible on master (261fab6465877862f777c9e6a7225863472cd53a), nautilus v14.2.0, nautilus v14.2.2, or nautilus HEAD (v14.2.4-27-g462e659cea).

@Kefu Chai,

No problem at all! I just wanted to make sure I was on the same page!
Also, your comment https://tracker.ceph.com/issues/40011#note-18 explains why I wasn't able to reproduce it and ended up helping me to validate it.

Thanks!

I wonder if the messenger is involved here? If it happens more often in downstream products, that might be because msgr version 1 is in use there, while Kefu's and Daniel's attempts might have been using msgr2?

(Just thinking out loud after reading Greg's comment #40011-6)

Nathan, that's plausible. i didn't adjust "ms_bind_msgr2". and i think "ms_bind_msgr2=true" has been around since v14.1.0:

$ git tag --contains 40a7dfbb1f25cae7cea68de18af981cb3a1b980f
v14.1.0
v14.1.1
v14.2.0
v14.2.1
v14.2.2
v14.2.3
v15.0.0

This still reproduces for me on current master

jan@ws ~/code/ceph/ceph/build (git)-[master] % bin/ceph osd pool rm foo foo --yes-i-really-really-mean-it
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2019-09-19T09:02:55.409+0200 7fcf49ab9700 -1 WARNING: all dangerous and experimental features are enabled.
2019-09-19T09:02:55.449+0200 7fcf49ab9700 -1 WARNING: all dangerous and experimental features are enabled.
pool 'foo' removed
jan@ws ~/code/ceph/ceph/build (git)-[master] % bin/ceph -s
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2019-09-19T09:03:01.793+0200 7fb389d32700 -1 WARNING: all dangerous and experimental features are enabled.
2019-09-19T09:03:01.817+0200 7fb389d32700 -1 WARNING: all dangerous and experimental features are enabled.
  cluster:
    id:     151dbb4b-8bf7-452f-9f81-0c3968859117
    health: HEALTH_WARN
            3 pools have too many placement groups

  services:
    mon: 3 daemons, quorum a,b,c (age 2m)
    mgr: x(active, since 2m)
    mds: a:1 {0=a=up:active} 2 up:standby
    osd: 3 osds: 3 up (since 102s), 3 in (since 102s)

  task status:
    scrub status:
        mds.0: idle

  data:
    pools:   5 pools, 48 pgs
    objects: 22 objects, 2.2 KiB
    usage:   6.0 GiB used, 3.0 TiB / 3.0 TiB avail
    pgs:     48 active+clean

jan@ws ~/code/ceph/ceph/build (git)-[master] % bin/ceph osd lspools
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
2019-09-19T09:03:08.125+0200 7fbb616de700 -1 WARNING: all dangerous and experimental features are enabled.
2019-09-19T09:03:08.157+0200 7fbb616de700 -1 WARNING: all dangerous and experimental features are enabled.
1 cephfs.a.meta
2 cephfs.a.data
4 bar
5 foobar
jan@ws ~/code/ceph/ceph/build (git)-[master] % bin/ceph --version
*** DEVELOPER MODE: setting PATH, PYTHONPATH and LD_LIBRARY_PATH ***
ceph version v15.0.0-5218-g3d7e5b0e3f (3d7e5b0e3fcf0dda9c664175ce6a0c0f3673a662) octopus (dev)

@Jan,

Thanks for the update! I will redeploy my test environment and recheck it.

This bug is probably somewhere in PGMap.cc--that's where the pool count is coming from. And that structure is updated in awkward ways by examining new OSDMap updates. It also happens on teh mgr, and is reported periodically to the mon, so it's normal for this mismatch to be there for 1-2 seconds (but not longer than that).