Project

General

Profile

Bug #33420

Forced deep-scrub doesn't start

Added by Maks Kowalik about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
Start date:
08/28/2018
Due date:
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature:

Description

Version: 12.2.7
Issue timeline:

1. Cyclic deep-scrub discovered inconsistency:

2018-08-23 17:21:07.933458 osd.62 osd.62 10.122.0.140:6805/77767 6 : cluster [ERR] 9.3cd shard 113: soid 9:b3cd8d89:::.dir.default.153398310.112:head omap_digest 0xea4ba012 != omap_digest 0xc5acebfd from shard 62, omap_digest 0xea4ba012 != omap_digest 0xc5acebfd from auth oi 9:b3cd8d89:::.dir.default.153398310.112:head(138609'2009129 osd.250.0:64658209 dirty|omap|data_digest|omap_digest s 0 uv 1995230 dd ffffffff od c5acebfd alloc_hint [0 0 0])
2018-08-23 18:53:30.585944 osd.62 osd.62 10.122.0.140:6805/77767 7 : cluster [ERR] 9.3cd deep-scrub 0 missing, 1 inconsistent objects
2018-08-23 18:53:30.585948 osd.62 osd.62 10.122.0.140:6805/77767 8 : cluster [ERR] 9.3cd deep-scrub 2 errors

2. A temporary key was set on the object:
rados -p .rgw.buckets setomapval .dir.default.153398310.112 temp-key temp-value

3. Ran deep-scrub on pg 9.3cd:
2018-08-27 14:36:22.703848 7faa7e860700 20 osd.62 713813 OSD::ms_dispatch: scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703869 7faa7e860700 20 osd.62 713813 _dispatch 0x55725b76d180 scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703871 7faa7e860700 10 osd.62 713813 handle_scrub scrub([9.3cd] deep) v2
2018-08-27 14:36:22.703878 7faa7e860700 10 osd.62 713813 marking pg[9.3cd( v 713813'2359292 (713107'2357731,713813'2359292] local-lis/les=711049/711050 n=41419 ec=178/178 lis/c 711049/711049 les/c/f 711050/711149/222921 711049/711049/710352) [62,53,163,113] r=0 lpr=711049 crt=713813'2359292 lcod 713813'2359291 mlcod 713813'2359291 active+clean+inconsistent MUST_DEEP_SCRUB MUST_SCRUB] for scrub

OSD got the scrub dispatch request, but the scrub doesn't start.

History

#1 Updated by John Spray about 1 year ago

  • Project changed from Ceph to RADOS
  • Category deleted (common)

Also available in: Atom PDF