Project

General

Profile

Actions

Bug #41526

closed

Choosing the next PG for a deep scrubs wrong.

Added by Fyodor Ustinov over 4 years ago. Updated over 4 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
David Zafman
Category:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
OSD
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have ceph cluster in this state:

# ceph health detail
HEALTH_WARN 27 pgs not deep-scrubbed in time;
PG_NOT_DEEP_SCRUBBED 27 pgs not deep-scrubbed in time
    pg 15.e4 not deep-scrubbed since 2019-08-14 19:15:30.699834
    pg 2.db not deep-scrubbed since 2019-08-14 14:14:25.173851
    pg 15.d8 not deep-scrubbed since 2019-08-14 20:19:03.937229
    pg 9.d9 not deep-scrubbed since 2019-08-14 19:40:16.157361
    pg 9.db not deep-scrubbed since 2019-08-15 04:24:14.865325
    pg 2.5c not deep-scrubbed since 2019-08-14 20:22:04.605225
    pg 17.45 not deep-scrubbed since 2019-08-15 01:43:06.099446
    pg 6.51 not deep-scrubbed since 2019-08-14 13:22:51.959783
    pg 6.4a not deep-scrubbed since 2019-08-15 03:39:44.701350
    pg 2.4d not deep-scrubbed since 2019-08-14 23:54:40.245206
    pg 7.33 not deep-scrubbed since 2019-08-15 03:32:24.927287
    pg 17.27 not deep-scrubbed since 2019-08-14 14:15:42.543503
    pg 17.17 not deep-scrubbed since 2019-08-14 23:09:42.728755
    pg 6.3 not deep-scrubbed since 2019-08-14 14:10:45.757717
    pg 10.12 not deep-scrubbed since 2019-08-15 03:11:14.487778
    pg 17.c not deep-scrubbed since 2019-08-14 22:43:25.739869
    pg 9.2f not deep-scrubbed since 2019-08-15 01:20:00.682925
    pg 10.22 not deep-scrubbed since 2019-08-14 19:50:51.555694
    pg 17.67 not deep-scrubbed since 2019-08-14 06:07:33.732999
    pg 6.79 not deep-scrubbed since 2019-08-15 03:28:21.388018
    pg 10.86 not deep-scrubbed since 2019-08-14 23:56:32.100535
    pg 2.92 not deep-scrubbed since 2019-08-15 04:20:55.337923
    pg 9.93 not deep-scrubbed since 2019-08-15 01:06:08.958664
    pg 2.9b not deep-scrubbed since 2019-08-15 04:28:54.398524
    pg 6.a7 not deep-scrubbed since 2019-08-15 04:05:45.886923
    pg 10.a7 not deep-scrubbed since 2019-08-14 12:34:06.675859
    pg 17.a4 not deep-scrubbed since 2019-08-14 22:21:07.467654

But ceph select strange (random?) pg for deep scrubing.

I do next command:

ceph pg dump | awk '$1 ~ /[0-9a-f]+\.[0-9a-f]+/ {print $25, $26, $1}' | sort > b1

After time again:

ceph pg dump | awk '$1 ~ /[0-9a-f]+\.[0-9a-f]+/ {print $25, $26, $1}' | sort > b2

And now - diff between:

diff b1 b2
558d557
< 2019-08-17 03:24:30.058471 17.18
2821a2821
> 2019-08-27 11:21:32.744345 17.18

As you can see - for deep scrubing will be selected not so old pg 17.18 although there are older pg.

Actions

Also available in: Atom PDF