Project

General

Profile

Actions

Bug #51961

closed

Stuck progress indicators in ceph status output

Added by Cory Snyder almost 3 years ago. Updated almost 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
cephadm
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Yes
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If an exception is thrown while cephadm is attempting to apply a service spec in the serve loop, the progress indicator for the failed deployment gets stuck. For example, we had stuck progress indicators after cephadm encountered exceptions while attempting to remove a couple of crash daemons. The following was our output from the ceph status command:

root@ceph01:~# ceph -s
  cluster:
    id:     <redacted>
    health: HEALTH_OK
  services:
    mon: 5 daemons, quorum ceph01,ceph02,ceph03,ceph04,ceph05 (age 58m)
    mgr: ceph01.sqycox(active, since 25m), standbys: ceph02.zmwnqy, ceph03.rtpqra, ceph05.jsdbre, ceph04.qkxmtx
    osd: 504 osds: 504 up (since 13m), 504 in (since 5h)
  data:
    pools:   3 pools, 5121 pgs
    objects: 6.87k objects, 23 GiB
    usage:   31 TiB used, 6.3 PiB / 6.3 PiB avail
    pgs:     5121 active+clean
  io:
    client:   2.8 KiB/s wr, 0 op/s rd, 0 op/s wr
  progress:
    Updating crash deployment (-2 -> 19) (0s)
      [............................]
    Updating crash deployment (-2 -> 19) (0s)
      [............................]

Since progress data is stored in memory on the active manager, the stuck progress indicators can be cleared by restarting the active manager.

Relates to the issue described in: https://tracker.ceph.com/issues/51616


Related issues 2 (0 open2 closed)

Related to Orchestrator - Bug #51616: Updating node-exporter deployment progress stuckResolved

Actions
Has duplicate Orchestrator - Bug #51621: Multiple "Updating node-exporter deployment"Duplicate

Actions
Actions #1

Updated by Cory Snyder almost 3 years ago

  • Related to Bug #51616: Updating node-exporter deployment progress stuck added
Actions #2

Updated by Kefu Chai almost 3 years ago

  • Status changed from New to Resolved
  • Pull request ID set to 42553
Actions #3

Updated by Sebastian Wagner almost 3 years ago

  • Has duplicate Bug #51621: Multiple "Updating node-exporter deployment" added
Actions

Also available in: Atom PDF