Project

General

Profile

Bug #53923

[Upgrade] mgr FAILED to decode MSG_PGSTATS

Added by Vikhyat Umrao 8 months ago. Updated 6 months ago.

Status:
Resolved
Priority:
Normal
Category:
-
Target version:
-
% Done:

100%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

# ceph -s
  cluster:
    id:     182eef00-53b5-11ec-84d3-3cecef3d8fb8
    health: HEALTH_WARN
            Reduced data availability: 65553 pgs inactive
            1 mgr modules have recently crashed

  services:
    mon: 5 daemons, quorum gibba001,gibba002,gibba004,gibba005,gibba006 (age 30m)
    mgr: gibba001.zptzqf(active, since 39m), standbys: gibba002.veobjs
    mds: 1/1 daemons up, 1 standby
    osd: 975 osds: 975 up (since 86m), 975 in (since 54m)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 65553 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             65553 unknown

  progress:
    Global Recovery Event (0s)
      [............................] 
    Upgrade to 17.0.0-10229-g7e035110 (6m)
      [............................] (remaining: 3h)

- MGR logs:

2022-01-18T19:54:52.484+0000 7f8914f04000  0 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev), process ceph-mgr, pid 7
2022-01-18T19:54:52.484+0000 7f8914f04000  0 pidfile_write: ignore empty --pid-file
2022-01-18T19:54:52.510+0000 7f8914f04000  1 mgr[py] Loading python module 'mirroring'

2022-01-18T19:56:14.017+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.018+0000 7f890d29e700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.018+0000 7f890ca9d700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.019+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.019+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input

[ubuntu@gibba001 ~]$ sudo ceph versions
{
    "mon": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 3,
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 2
    },
    "mgr": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 2
    },
    "osd": {
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 975
    },
    "mds": {
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 2
    },
    "overall": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 5,
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 979
    }
}

Related issues

Related to RADOS - Bug #55140: quincy OSD won't start: what(): void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input Duplicate
Copied to RADOS - Backport #55137: quincy: [Upgrade] mgr FAILED to decode MSG_PGSTATS Resolved

History

#1 Updated by Neha Ojha 8 months ago

  • Subject changed from PG Stats unknown after upgrade to 17.0.0-10229-g7e035110 to FAILED to decode MSG_PGSTATS

#2 Updated by Neha Ojha 8 months ago

  • Subject changed from FAILED to decode MSG_PGSTATS to mgr FAILED to decode MSG_PGSTATS

#3 Updated by Neha Ojha 8 months ago

  • Description updated (diff)

#4 Updated by Vikhyat Umrao 8 months ago

  • Subject changed from mgr FAILED to decode MSG_PGSTATS to [Upgrade] mgr FAILED to decode MSG_PGSTATS

#5 Updated by Neha Ojha 8 months ago

  • Status changed from New to In Progress
  • Assignee set to Ronen Friedman
  • Priority changed from Normal to Immediate

#6 Updated by Neha Ojha 6 months ago

  • Status changed from In Progress to Can't reproduce

#7 Updated by Aishwarya Mathuria 6 months ago

  • Status changed from Can't reproduce to In Progress
  • Assignee changed from Ronen Friedman to Aishwarya Mathuria

#8 Updated by Laura Flores 6 months ago

  • Pull request ID set to 45694

#10 Updated by Neha Ojha 6 months ago

  • Status changed from In Progress to Pending Backport
  • Backport set to quincy

#11 Updated by Backport Bot 6 months ago

  • Copied to Backport #55137: quincy: [Upgrade] mgr FAILED to decode MSG_PGSTATS added

#12 Updated by Neha Ojha 6 months ago

  • Status changed from Pending Backport to Resolved

#13 Updated by Aishwarya Mathuria 6 months ago

  • Related to Bug #55140: quincy OSD won't start: what(): void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input added

Also available in: Atom PDF