Project

General

Profile

Actions

Bug #62701

open

mgr: health_checks report slow op from already down osd

Added by yite gu 9 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

# ceph -s
  cluster:
    id:     12e67144-ead1-48c1-94e3-ad416b130a0d
    health: HEALTH_WARN
            1 osds down
            1 host (8 osds) down
            Degraded data redundancy: 1765808/5297424 objects degraded (33.333%), 129 pgs degraded, 129 pgs undersized
            129 pgs not deep-scrubbed in time
            129 pgs not scrubbed in time
            3 slow ops, oldest one blocked for 75 sec, daemons [osd.16,osd.22,osd.6] have slow ops.

  services:
    mon: 3 daemons, quorum b,c,e (age 3M)
    mgr: a(active, since 13M)
    osd: 24 osds: 16 up (since 17h), 17 in (since 17h)

  data:
    pools:   2 pools, 129 pgs
    objects: 1.77M objects, 6.7 TiB
    usage:   14 TiB used, 104 TiB / 119 TiB avail
    pgs:     1765808/5297424 objects degraded (33.333%)
             129 active+undersized+degraded

  io:
    client:   78 KiB/s rd, 19 MiB/s wr, 133 op/s rd, 201 op/s wr

blocked for 75 sec value no longer change.
osd.16, osd.22, osd.6 already down:
# ceph osd tree
ID  CLASS  WEIGHT     TYPE NAME               STATUS  REWEIGHT  PRI-AFF
-1         167.67114  root default
-3          55.89038      host node08-shcm03
 0    ssd    6.98630          osd.0               up   1.00000  1.00000
 1    ssd    6.98630          osd.1               up   1.00000  1.00000
 2    ssd    6.98630          osd.2               up   1.00000  1.00000
 5    ssd    6.98630          osd.5               up   1.00000  1.00000
 8    ssd    6.98630          osd.8               up   1.00000  1.00000
15    ssd    6.98630          osd.15              up   1.00000  1.00000
18    ssd    6.98630          osd.18              up   1.00000  1.00000
21    ssd    6.98630          osd.21              up   1.00000  1.00000
-5          55.89038      host node09-shcm03
 3    ssd    6.98630          osd.3             down         0  1.00000
 6    ssd    6.98630          osd.6             down         0  1.00000
 9    ssd    6.98630          osd.9             down         0  1.00000
11    ssd    6.98630          osd.11            down         0  1.00000
13    ssd    6.98630          osd.13            down         0  1.00000
16    ssd    6.98630          osd.16            down         0  1.00000
19    ssd    6.98630          osd.19            down         0  1.00000
22    ssd    6.98630          osd.22            down   1.00000  1.00000
-7          55.89038      host node10-shcm03
 4    ssd    6.98630          osd.4               up   1.00000  1.00000
 7    ssd    6.98630          osd.7               up   1.00000  1.00000
10    ssd    6.98630          osd.10              up   1.00000  1.00000
12    ssd    6.98630          osd.12              up   1.00000  1.00000
14    ssd    6.98630          osd.14              up   1.00000  1.00000
17    ssd    6.98630          osd.17              up   1.00000  1.00000
20    ssd    6.98630          osd.20              up   1.00000  1.00000
23    ssd    6.98630          osd.23              up   1.00000  1.00000

version is 15.2.7

No data to display

Actions

Also available in: Atom PDF