Ceph » RADOS

The do_sparse_read() path doesn't attempt to repair a checksum error. Could that be the real issue?

The do_read() path looks fine since it calls rep_repair_primary_object() whether the EIO came from the disk or the crc check.

      if (oi.data_digest != crc) {
        osd->clog->error() << info.pgid << std::hex
                           << " full-object read crc 0x" << crc
                           << " != expected 0x" << oi.data_digest
                           << std::dec << " on " << soid;
        r = -EIO; // try repair later
      }
    }
    if (r == -EIO) {
      r = rep_repair_primary_object(soid, ctx->op);
    }

Ah, the error was reported on luminous, which doesn't do the repair, and I guess I missed it on master. Sorry for the mis-diagnosis.

(Looks like the MDS doesn't use sparse-read, but we should definitely still fix that path too!)

Is this the relevant fix? https://github.com/ceph/ceph/commit/4667280f8afe6cd68dfffea61d7530581f3dd0eb

Alessandro's OSDs are bluestore, and he doesn't get any bluestore block checksum errors. So the crc can be wrong when the bluestore data is correct?

Will the above patch correct this type of crc error?

Also, we deep-scrubbed the PG and it didn't find any inconsistent objects.

FTR, this crc issue is probably due to an incomplete backport to 12.2.6 of the skip_digest changes for bluestore:

[12:56:59]  <dvanders> regarding the 12.2.6 cephfs crc errors, could it be `b519a0b1c1 osd/PrimaryLogPG: do not generate data digest for BlueStore by default`  or one of the other omap/data_digest changes that landed in 12.2.6 ?
[12:59:29]  <dvanders> Seems similar to https://tracker.ceph.com/issues/23871 ... which was fixed in mimic but not luminous: `fe5038c7f9 osd/PrimaryLogPG: clear data digest on WRITEFULL if skip_data_digest`
[14:10:16]  <sage>    dvanders: yeah does seem similar, but i'm not sure why it would manifest during a .5 to .6 upgrade.  looking...
[14:10:37]  <sage>    dvanders: it's definitely bluestore-only?
[14:11:20]  <dvanders>    i didn't try filestore, but all the clusters i've seen were bluestore
[14:14:56]  <sage>    ah, it's becaues teh other skip_digest handlng code was just backporting/changed
[14:14:59]  <sage>    backported/changed

This issue is related to https://tracker.ceph.com/issues/23871

master PR: https://github.com/ceph/ceph/pull/23377