Project

General

Profile

Bug #50591

mgr/progress: progress can be negative

Added by Dan van der Ster 17 days ago. Updated 4 days ago.

Status:
Pending Backport
Priority:
High
Category:
progress module
Target version:
% Done:

40%

Source:
Community (user)
Tags:
Backport:
nautilus,octopus,pacific
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Seen in #50587: progress can be negative. This causes unresponsive mons in nautilus <= v14.2.20 when responding to `ceph status`.

2021-04-29 17:39:42.340 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577965..3577966
2021-04-29 17:40:42.423 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577966..3577967
2021-04-29 17:40:42.589 7fbfe2b47700  1 mgr[progress] osd.321 marked out
2021-04-29 17:40:43.072 7fbfe2b47700  1 mgr[progress] 0 PGs affected by osd.321 being marked out
2021-04-29 17:40:43.521 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577967..3577968
2021-04-29 17:40:44.669 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577968..3577969
2021-04-29 17:40:45.769 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577969..3577970
2021-04-29 17:40:46.879 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577970..3577971
2021-04-29 17:40:48.010 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577971..3577972
2021-04-29 17:40:56.083 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577972..3577973
2021-04-29 17:41:12.081 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577973..3577974
2021-04-29 17:41:12.489 7fbfe2b47700  1 mgr[progress] osd.321 marked in
2021-04-29 17:41:12.974 7fbfe2b47700  1 mgr[progress] 3 PGs affected by osd.321 being marked in
2021-04-29 17:41:18.097 7fbfe2b47700  4 mgr[progress] Updated progress to 0.0 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:18.684 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577974..3577976
2021-04-29 17:41:24.592 7fbfe2b47700  4 mgr[progress] Updated progress to 0.0 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:30.448 7fbfe2b47700  4 mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:30.722 7fbfe2b47700  4 mgr[progress] Processing OSDMap change 3577976..3577977
2021-04-29 17:41:31.222 7fbfe2b47700  1 mgr[progress] osd.538 marked in
2021-04-29 17:41:31.721 7fbfe2b47700  1 mgr[progress] 0 PGs affected by osd.538 being marked in
2021-04-29 17:41:36.768 7fbfe2b47700  4 mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:42.117 7fbfe2b47700  4 mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:47.358 7fbfe2b47700  4 mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd.321 marked in)
2021-04-29 17:41:52.587 7fbfe2b47700  4 mgr[progress] Updated progress to -0.333333333333 (Rebalancing after osd.321 marked in)


Related issues

Copied from RADOS - Bug #50587: mon election storm following osd recreation: huge tcmalloc and ceph::msgr::v2::FrameAssembler backtraces New
Copied to mgr - Backport #50777: nautilus: mgr/progress: progress can be negative In Progress
Copied to mgr - Backport #50778: octopus: mgr/progress: progress can be negative In Progress
Copied to mgr - Backport #50779: pacific: mgr/progress: progress can be negative In Progress

History

#1 Updated by Dan van der Ster 17 days ago

  • Copied from Bug #50587: mon election storm following osd recreation: huge tcmalloc and ceph::msgr::v2::FrameAssembler backtraces added

#2 Updated by Dan van der Ster 17 days ago

  • Status changed from New to Fix Under Review
  • Assignee set to Dan van der Ster
  • Backport set to nautilus,octopus,pacific

#3 Updated by Dan van der Ster 17 days ago

  • Pull request ID set to 41094

#4 Updated by Konstantin Shalygin 17 days ago

  • Category set to progress module
  • % Done changed from 0 to 40
  • Source set to Community (user)

#5 Updated by Kefu Chai 4 days ago

  • Status changed from Fix Under Review to Pending Backport

#6 Updated by Backport Bot 4 days ago

  • Copied to Backport #50777: nautilus: mgr/progress: progress can be negative added

#7 Updated by Backport Bot 4 days ago

  • Copied to Backport #50778: octopus: mgr/progress: progress can be negative added

#8 Updated by Backport Bot 4 days ago

  • Copied to Backport #50779: pacific: mgr/progress: progress can be negative added

Also available in: Atom PDF