Project

General

Profile

Bug #25084

Attempt to read object that can't be repaired loops forever

Added by David Zafman about 1 year ago. Updated 12 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
07/24/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
See comment
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:

Description

If all replicas are of an object are bad causes a loop of continuous recovery and calls to rep_repair_primary_object(). I've reproduced this by making the object data_digest mismatch the object_info_t data_digest.

I saw this in qa/standalone/scrub/osd-scrub-repair.sh TEST_corrupt_scrub_replicated() ROBJ17 if you try to read it BEFORE the repair. The current code verifies the repair by reading it afterwards.


Related issues

Related to RADOS - Bug #24875: OSD: still returning EIO instead of recovering objects on checksum errors Resolved 07/11/2018
Related to RADOS - Bug #12615: Repair of Erasure Coded pool with an unrepairable object causes pg state to lose clean state Verified 08/04/2015

History

#1 Updated by David Zafman about 1 year ago

  • Related to Bug #24875: OSD: still returning EIO instead of recovering objects on checksum errors added

#2 Updated by David Zafman about 1 year ago

  • Status changed from New to In Progress

What I actually ran into is that when do_read() fails because of the CRC mismatch, the recovery repair can pull from an OSD that also has a mismatched CRC. The recovery doesn't appear to have failed, because the pushing peer and the primary don't check the CRC.

One way to fix this, is to check the CRC when possible in build_push_op(). If it doesn't match, then return an EIO.

Another fix would be to check the CRC at the primary when receiving the read data.

My concern is in the situation what the object info data_digest is inconsistent on all replicas, we would be unable to copy such an object off of the current set of OSDs. This inconsistency happened due to a bug in Luminous 10.2.5.

#4 Updated by David Zafman about 1 year ago

  • Backport set to See comment

I don't think we should backport this change. In Luminous and possibly upgraded to Mimic there is a possibility that object info data_digest mismatches in a mixed filestore/bluestore cluster. In that case this change will cause a PG to go to recovery_unfound state and not be clean after a read CRC error is found.

#5 Updated by David Zafman about 1 year ago

  • Related to Bug #12615: Repair of Erasure Coded pool with an unrepairable object causes pg state to lose clean state added

#6 Updated by David Zafman about 1 year ago

  • Status changed from In Progress to Need Review

#7 Updated by David Zafman 12 months ago

  • Status changed from Need Review to Resolved

Also available in: Atom PDF