Bug #13007
closed
unexpected scrub operations on hammer 0.94.3
Added by huang jun over 8 years ago.
Updated over 8 years ago.
Description
I build a cluster on hammer 0.94.3, and set
osd_scrub_max_interval = 31536000
osd_deep_scrub_interval = 31536000
osd_scrub_min_interval = 31536000
to disable scrub operation.
After i increase pg_num of data pool, the "ceph -w" shows there are many
PG doing scrub.
I add some debug code to PG::reg_next_scrub function, here the debug log:
2015-09-09 14:56:06.240370 7f47bb1db700 0 osd.0 pg_epoch: 21 pg[0.17( empty local-les=9 n=0 ec=1 les/c 9/9 8/21/4) [0,1] r=0 lpr=21 pi=8-20/1 crt=0'0 mlcod 0'0 peering] reg_next_scrub must_scrub 0 stats_invalid 1 osd_scrub_invalid_stats 1
2015-09-09 14:56:06.240460 7f47bb1db700 0 osd.0 pg_epoch: 21 pg[0.17( empty local-les=9 n=0 ec=1 les/c 9/9 8/21/4) [0,1] r=0 lpr=21 pi=8-20/1 crt=0'0 mlcod 0'0 peering] reg_next_scrub pg 0.17, scrub_reg_stamp 0.000000
In the code:
void PG::reg_next_scrub()
{
if (scrubber.must_scrub ||
(info.stats.stats_invalid && g_conf->osd_scrub_invalid_stats)) {
scrubber.scrub_reg_stamp = utime_t();
} else {
scrubber.scrub_reg_stamp = info.history.last_scrub_stamp;
}
if (is_primary())
osd->reg_last_pg_scrub(info.pgid, scrubber.scrub_reg_stamp);
}
we should use ceph_clock_now(cct) to replace utime_t()
and set
osd_scrub_max_interval = 31536000
osd_deep_scrub_interval = 31536000
osd_scrub_min_interval = 31536000
to disable scrub operation.
huangjun, i am not sure this is the expected way to disable scrub. if a "must_scrub" scrub is scheduled, it is put at the beginning of the scrub job queue (last_scrub_pg). so assigning utime_t(0,0) to this job is the expected way to do this.
to disable scrub temporarily, you can use
ceph osd set noscrub
ceph osd set nodeep-scrub
In particular I believe this initialization is deliberate. The OSD makes some guesses about the size of the child PGs but it doesn't actually know how big each one is until after a scrub has processed. That's not great for some scheduling and quota stuff so it specifies the PG as needing a scrub.
- Status changed from New to Won't Fix
huangjun, i am closing it as "won't fix", please reopen it if you think otherwise.
@Kefu Chai chai,
but if you check the master branch code, it using ceph_clock_now(cct)
huangjun, as a part of fix for #10973, it's using the the timestamp as the wall-time to schedule a scrub. but we have no plan to backport #10973 to hammer. since it's a feature but not a bug fix.
kefu, you mean if the pg stats_invalid is true, it needs to scrub?
is there any problems if i set osd_scrub_invalid_stats to false?
or set noscrub and nodeep-scrub flag as you saied.
you mean if the pg stats_invalid is true, it needs to scrub?
right.
is there any problems if i set osd_scrub_invalid_stats to false?
it helps to fix stats of the pg which gets split. see #8147. anyway, it's on your own.
or set noscrub and nodeep-scrub flag as you saied.
it's a way to temporary disable the scrub. i am afraid that you data/disk can survive a long time without scrub. again, we do everything to ensure the correctness of data. the longer you disable the deep scrub, the longer your data are at risk.
Also available in: Atom
PDF