Project

General

Profile

Actions
Copy link

Bug #13007

closed

unexpected scrub operations on hammer 0.94.3

Added by huang jun over 8 years ago. Updated over 8 years ago.

Status:
Won't Fix
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I build a cluster on hammer 0.94.3, and set
osd_scrub_max_interval = 31536000
osd_deep_scrub_interval = 31536000
osd_scrub_min_interval = 31536000
to disable scrub operation.

After i increase pg_num of data pool, the "ceph -w" shows there are many
PG doing scrub.

I add some debug code to PG::reg_next_scrub function, here the debug log:
2015-09-09 14:56:06.240370 7f47bb1db700 0 osd.0 pg_epoch: 21 pg[0.17( empty local-les=9 n=0 ec=1 les/c 9/9 8/21/4) [0,1] r=0 lpr=21 pi=8-20/1 crt=0'0 mlcod 0'0 peering] reg_next_scrub must_scrub 0 stats_invalid 1 osd_scrub_invalid_stats 1
2015-09-09 14:56:06.240460 7f47bb1db700 0 osd.0 pg_epoch: 21 pg[0.17( empty local-les=9 n=0 ec=1 les/c 9/9 8/21/4) [0,1] r=0 lpr=21 pi=8-20/1 crt=0'0 mlcod 0'0 peering] reg_next_scrub pg 0.17, scrub_reg_stamp 0.000000

In the code:
void PG::reg_next_scrub() {
if (scrubber.must_scrub ||
(info.stats.stats_invalid && g_conf->osd_scrub_invalid_stats)) {
scrubber.scrub_reg_stamp = utime_t();
} else {
scrubber.scrub_reg_stamp = info.history.last_scrub_stamp;
}
if (is_primary())
osd->reg_last_pg_scrub(info.pgid, scrubber.scrub_reg_stamp);
}

we should use ceph_clock_now(cct) to replace utime_t()

Actions
Copy link
 #2

Updated by Kefu Chai over 8 years ago

and set
osd_scrub_max_interval = 31536000
osd_deep_scrub_interval = 31536000
osd_scrub_min_interval = 31536000
to disable scrub operation.

huangjun, i am not sure this is the expected way to disable scrub. if a "must_scrub" scrub is scheduled, it is put at the beginning of the scrub job queue (last_scrub_pg). so assigning utime_t(0,0) to this job is the expected way to do this.

to disable scrub temporarily, you can use

ceph osd set noscrub
ceph osd set nodeep-scrub

Actions
Copy link
 #3

Updated by Greg Farnum over 8 years ago

In particular I believe this initialization is deliberate. The OSD makes some guesses about the size of the child PGs but it doesn't actually know how big each one is until after a scrub has processed. That's not great for some scheduling and quota stuff so it specifies the PG as needing a scrub.

Actions
Copy link
 #4

Updated by Kefu Chai over 8 years ago

  • Status changed from New to Won't Fix

huangjun, i am closing it as "won't fix", please reopen it if you think otherwise.

Actions
Copy link
 #5

Updated by huang jun over 8 years ago

@Kefu Chai chai,
but if you check the master branch code, it using ceph_clock_now(cct)

Actions
Copy link
 #6

Updated by Kefu Chai over 8 years ago

huangjun, as a part of fix for #10973, it's using the the timestamp as the wall-time to schedule a scrub. but we have no plan to backport #10973 to hammer. since it's a feature but not a bug fix.

Actions
Copy link
 #7

Updated by huang jun over 8 years ago

kefu, you mean if the pg stats_invalid is true, it needs to scrub?
is there any problems if i set osd_scrub_invalid_stats to false?
or set noscrub and nodeep-scrub flag as you saied.

Actions
Copy link
 #8

Updated by Kefu Chai over 8 years ago

you mean if the pg stats_invalid is true, it needs to scrub?

right.

is there any problems if i set osd_scrub_invalid_stats to false?

it helps to fix stats of the pg which gets split. see #8147. anyway, it's on your own.

or set noscrub and nodeep-scrub flag as you saied.

it's a way to temporary disable the scrub. i am afraid that you data/disk can survive a long time without scrub. again, we do everything to ensure the correctness of data. the longer you disable the deep scrub, the longer your data are at risk.

Actions
Copy link

Also available in: Atom PDF