Project

General

Profile

Actions
Copy link

Bug #53923

closed

[Upgrade] mgr FAILED to decode MSG_PGSTATS

Added by Vikhyat Umrao over 2 years ago. Updated about 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Aishwarya Mathuria
Category:
-
Target version:
-
% Done:

100%

Source:
Tags:
Backport:
quincy
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
45694
Crash signature (v1):
Crash signature (v2):

Description

# ceph -s
  cluster:
    id:     182eef00-53b5-11ec-84d3-3cecef3d8fb8
    health: HEALTH_WARN
            Reduced data availability: 65553 pgs inactive
            1 mgr modules have recently crashed

  services:
    mon: 5 daemons, quorum gibba001,gibba002,gibba004,gibba005,gibba006 (age 30m)
    mgr: gibba001.zptzqf(active, since 39m), standbys: gibba002.veobjs
    mds: 1/1 daemons up, 1 standby
    osd: 975 osds: 975 up (since 86m), 975 in (since 54m)

  data:
    volumes: 1/1 healthy
    pools:   4 pools, 65553 pgs
    objects: 0 objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:     100.000% pgs unknown
             65553 unknown

  progress:
    Global Recovery Event (0s)
      [............................] 
    Upgrade to 17.0.0-10229-g7e035110 (6m)
      [............................] (remaining: 3h)

- MGR logs:

2022-01-18T19:54:52.484+0000 7f8914f04000  0 ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev), process ceph-mgr, pid 7
2022-01-18T19:54:52.484+0000 7f8914f04000  0 pidfile_write: ignore empty --pid-file
2022-01-18T19:54:52.510+0000 7f8914f04000  1 mgr[py] Loading python module 'mirroring'

2022-01-18T19:56:14.017+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.018+0000 7f890d29e700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.018+0000 7f890ca9d700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.019+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input
2022-01-18T19:56:14.019+0000 7f890da9f700 -1 failed to decode message of type 87 v2: void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input


[ubuntu@gibba001 ~]$ sudo ceph versions
{
    "mon": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 3,
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 2
    },
    "mgr": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 2
    },
    "osd": {
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 975
    },
    "mds": {
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 2
    },
    "overall": {
        "ceph version 17.0.0-10229-g7e035110 (7e035110784fba02ba81944e444be9a36932c6a3) quincy (dev)": 5,
        "ceph version 17.0.0-9964-gf2313edc (f2313edc67106699e6ab91f50fa91928e579f7ac) quincy (dev)": 979
    }
}

Related issues 2 (0 open — 2 closed)

Related to RADOS - Bug #55140: quincy OSD won't start: what(): void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed inputDuplicate

Actions
Copied to RADOS - Backport #55137: quincy: [Upgrade] mgr FAILED to decode MSG_PGSTATSResolvedAishwarya MathuriaActions
Actions
Copy link
 #1

Updated by Neha Ojha over 2 years ago

  • Subject changed from PG Stats unknown after upgrade to 17.0.0-10229-g7e035110 to FAILED to decode MSG_PGSTATS
Actions
Copy link
 #2

Updated by Neha Ojha over 2 years ago

  • Subject changed from FAILED to decode MSG_PGSTATS to mgr FAILED to decode MSG_PGSTATS
Actions
Copy link
 #3

Updated by Neha Ojha over 2 years ago

  • Description updated (diff)
Actions
Copy link
 #4

Updated by Vikhyat Umrao over 2 years ago

  • Subject changed from mgr FAILED to decode MSG_PGSTATS to [Upgrade] mgr FAILED to decode MSG_PGSTATS
Actions
Copy link
 #5

Updated by Neha Ojha over 2 years ago

  • Status changed from New to In Progress
  • Assignee set to Ronen Friedman
  • Priority changed from Normal to Immediate
Actions
Copy link
 #6

Updated by Neha Ojha about 2 years ago

  • Status changed from In Progress to Can't reproduce
Actions
Copy link
 #7

Updated by Aishwarya Mathuria about 2 years ago

  • Status changed from Can't reproduce to In Progress
  • Assignee changed from Ronen Friedman to Aishwarya Mathuria
Actions
Copy link
 #8

Updated by Laura Flores about 2 years ago

  • Pull request ID set to 45694
Actions
Copy link
 #10

Updated by Neha Ojha about 2 years ago

  • Status changed from In Progress to Pending Backport
  • Backport set to quincy
Actions
Copy link
 #11

Updated by Backport Bot about 2 years ago

  • Copied to Backport #55137: quincy: [Upgrade] mgr FAILED to decode MSG_PGSTATS added
Actions
Copy link
 #12

Updated by Neha Ojha about 2 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #13

Updated by Aishwarya Mathuria about 2 years ago

  • Related to Bug #55140: quincy OSD won't start: what(): void pg_stat_t::decode(ceph::buffer::v15_2_0::list::const_iterator&) decode past end of struct encoding: Malformed input added
Actions
Copy link

Also available in: Atom PDF