Project

General

Profile

Actions

Bug #1340

closed

OSD: Strange scrub behavior

Added by Greg Farnum over 12 years ago. Updated over 12 years ago.

Status:
Rejected
Priority:
Normal
Assignee:
Category:
OSD
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):


Files

osd.5.log.5.gz (211 KB) osd.5.log.5.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.6.gz (160 KB) osd.5.log.6.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.7.gz (168 KB) osd.5.log.7.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log (1.36 MB) osd.5.log Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.1.gz (182 KB) osd.5.log.1.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.2.gz (136 KB) osd.5.log.2.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.3.gz (184 KB) osd.5.log.3.gz Fyodor Ustinov, 07/26/2011 12:19 PM
osd.5.log.4.gz (178 KB) osd.5.log.4.gz Fyodor Ustinov, 07/26/2011 12:19 PM
Actions #2

Updated by Greg Farnum over 12 years ago

Okay, this is just bizarre. There are actually 4 different PGs that report a stat mismatch error. 3 of them (including 0.f8) are off by the exact same amount -- 34055KB more in the on-disk state than the in-memory info -- and the other two somehow fix themselves. 0.f8 seems to eventually fix itself, since after doing the repair (which detects the same problem as the initial scrub 3.5 hours later, so the problem can't be JUST a race) it later says "hey, my recorded stats are too high compared to my on-disk state!" and then reduces it by the same amount it initially incremented them by.
But, somehow, between the first repair and the next scrub, the on-disk quantity goes down despite the PG adding 8 objects and apparently only doing write operations (since the repgather commit messages are set to way too high a debug level, this is the one thing I can see).

Actions #3

Updated by Greg Farnum over 12 years ago

  • Status changed from In Progress to Rejected

Not A Bug!

Fyodor wrote:

"Hmmm" I said to myself.

stop osd5
fsck -fy /dev/sdb1

And got:

Inode 230558734, i_blocks is 8, should be 16. Fix? yes

Inode 230559279, i_blocks is 8208, should be 8200. Fix? yes

Inode 230559410, i_blocks is 40, should be 32. Fix? yes

Inode 230559527, i_blocks is 24, should be 32. Fix? yes

Inode 231342174, i_blocks is 40, should be 32. Fix? yes

Inode 231342209, i_blocks is 24, should be 32. Fix? yes

Actions

Also available in: Atom PDF