Project

General

Profile

Actions
Copy link

Bug #37783

closed

scrub health warning

Added by Chris Webb over 5 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
v13.2.3
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
v13.2.3
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

This message popped up this afternoon: HEALTH_WARN 1280 pgs not scrubbed for 1.61061e+10

It seems rather unlikely, further details:

[root@ceph1 ceph]# ceph status
cluster:
id: e474ffde-40fd-44c2-aa4d-753dd4dd0761
health: HEALTH_WARN
1280 pgs not scrubbed for 1.61061e+10

services:
    mon: 3 daemons, quorum ceph1,ceph2,ceph3
    mgr: ceph1(active), standbys: ceph2, ceph3
    mds: cephfs-nan-1/1/1 up  {0=ceph3=up:active}, 2 up:standby
    osd: 36 osds: 36 up, 36 in
data:
    pools:   2 pools, 1280 pgs
    objects: 43.00 M objects, 28 TiB
    usage:   49 TiB used, 279 TiB / 327 TiB avail
    pgs:     1280 active+clean
io:
    client:   234 MiB/s rd, 113 MiB/s wr, 915 op/s rd, 1.14 kop/s wr

[root@ceph1 ceph]#

[root@ceph1 ~]# date
Thu 3 Jan 15:08:14 GMT 2019

[root@ceph1 ~]# ceph health detail
HEALTH_WARN 1280 pgs not scrubbed for 1.61061e+10
PG_NOT_SCRUBBED 1280 pgs not scrubbed for 1.61061e+10
pg 5.3ff not scrubbed since 2019-01-02 17:09:41.041659
pg 5.3fe not scrubbed since 2019-01-03 13:55:12.059461
pg 5.3fd not scrubbed since 2019-01-02 14:55:46.924332
pg 5.3fc not scrubbed since 2019-01-03 02:17:17.074052
pg 5.3fb not scrubbed since 2019-01-02 10:47:21.080051
pg 5.3fa not scrubbed since 2019-01-03 03:26:22.262786
pg 5.3f9 not scrubbed since 2019-01-03 06:47:43.489866
pg 5.3f8 not scrubbed since 2019-01-02 06:23:15.001033
...

As you can see from the above messages scrubs are in fact ongoing!

Our cluster includes 3 servers, all running :

[root@ceph1 ~]# uname -a
Linux ceph1.iop.kcl.ac.uk 3.10.0-862.14.4.el7.x86_64 #1 SMP Tue Sep 25 14:32:52 CDT 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@ceph1 ~]# cat /etc/redhat-release
Scientific Linux release 7.6 (Nitrogen)
[root@ceph1 ~]# ceph --version
ceph version 13.2.3 (9bf3c8b1a04b0aa4a3cc78456a508f1c48e70279) mimic (stable)
[root@ceph1 ~]#

Rather annoyingly the yum-cron process updated ceph this morning, this feature has now been disabled. I noticed that all processes were restarted by the update, expect for ceph-mon which was still running. I have restarted these processes, but the error has not cleared.

Chris

Actions
Copy link

Also available in: Atom PDF