Ceph

(Copied from conversation in slack ceph-devel)

I'm following up on some apparent data corruption in an EC pool (I'll thread a link to my ceph-users message with more background), which appears to be presenting in an unexpected way: from what I can tell, individual shards were corrupted before being saved, so each per-OSD checksum matches the expected value, and deep scrubbing doesn't identify an issue, but requests for the objects from the OSD trigger a checksum mismatch warning. Manual testing by disabling a targeted OSD makes some objects readable, so I suspect enough of the data is stored to recover, but in this case the automated checks aren't catching (or fixing) it.
I'd like to pull the individual shards of a relevant object from all of the acting OSDs directly (rather than just k of them), so I can pass them to ceph-erasure-code-tool and hopefully recover the data. This is a bit tedious, and I'll eventually want to write more tooling around it and dig into the root cause of the issue, but for now I'm just trying to get the data out of the pool so I feel more comfortable I'm not going to lose it. (Of course, other ways of recovering the data are probably also welcome, but this seems the most expedient off the top of my head.)
All of that is to ask: what's a relatively quick way of retrieving the EC shards for an object from the OSDs? It doesn't seem there's a way to get rados (the CLI tool), or even librados to do it. I'm about to start trying to modify Objecter to change an Op for an OSD read into a shard read (ECSubRead? I haven't gotten that far.), but I figured I'd ask here and see if anyone else has thoughts. (Or can point me towards another place to ask.) I'd sort of rather avoid modifying the osd daemon itself to do this since the pool is in use, but if there's a better path that way - or by other direct access to the backing devices - that would be good to know too. (edited)

First, identify the name of the object, the pg, and the acting set osds. Stop all of the acting set OSDs. ceph-objectstore-tool can then be used to extract the shard for the object from each osd store. If you can attach those pieces to a bug, it would be helpful in identifying what's going on.

Also, can you expand on what you mean by "Oddly, the CephFS kernel module (at least in Ubuntu 22.04 appears to ignore the OSD error and return incorrect data for the parts of the file composed of affected objects, while both the rados CLI tool and the cephfs FUSE helper treat it as an I/O error."? Normal reads in cephfs simply return the invalid data but the rados cli tool actually returns an error? What error, is there also an error log? Can you enable debug_osd=20 debug_bluestore=20 debug_ms=1 on the OSDs in the acting set, reproduce that read error, and attach the logs from that (hopefully short) period of time?

By using effectively the same "take one OSD down and read again" mechanism, I identified that all of the obvious corruption in the set of data I was examining appears to have taken place on three OSDs: roughly 99.9% split about evenly between two OSDs on one server, and a tiny portion on an OSD on a different server. I'm not sure if that means anything to anyone, but I'll mention it in case it does.

Additionally, the vast majority of data stored was fine, the affected objects were maybe 5.5 GB out of ~38 TB for this project, and there were plenty of working objects on each PG: it's not as if a whole PG or OSD was affected. (The pool has 256 PGs.)

I was also able to recover all of the data I was examining via taking those individual drives offline and reading the "bad" objects, then reassembling the affected files, so I don't think there has been any data loss in the end. (However, this pool is used for other things, and I have yet to examine them: I would not be surprised to find further corruption with other objects.)

For timeline matters, a few notes:

• I copied in all of the files and then a few days later generated SHA-256 hashes of all of them, from the copies on this pool. At that time, the hashes matched the data I had copied. I am very confident on this.
• Some time later - approximately a month or two later - I shared copies of the files with another person. Their copies appear to be corrupt in the same exact way my current reads are when reading via the CephFS kernel module, from comparing hashes of several files. (Their copies were served from a mount using the kernel module, so this is not a surprise, but it appears the corruption has not increased since it happened in the first place.) I am moderately confident on this.
• The cluster has had drives added at least once, and possibly twice since the corrupt copies above were taken. I am very confident on this, and it would have caused data to move between drives. I also suspect I upgraded at least from Pacific to Quincy during that time, but I am not confident on when that occurred. (I know I am currently on Quincy: 17.2.6.)
• The cluster is set to perform deep scrubs approximately every two weeks, and has not reported any health issues relating to them. I am confident on this.

I'm not actually sure how the blaum_roth EC mechanism works, but I suppose it is possible that data got written incorrectly in the first place, given the successful deep scrubs (assuming they actually check the hash of the shards on disk, as they appear to not reassemble objects themselves). I'm also not sure how the data got corrupted in the first place: RAM or CPU malfunctions are a possible culprit, perhaps. It is also possible that adding drives may have moved some PGs between physical servers, so it is possible that only one server corrupted data.

I have a couple overlapping questions:

1. How can I detect corruption like this? It appears I could loop over every object on the pool and attempt to read it, which will tell me if the first four active shards are accurate at least (or I've run into a CRC collision), but it's not clear to me that there's a way to force reading or verification of the parity shards.
2. How can I force repair of this kind of corruption? I can manually reassemble files and write them back out (which is what I've done so far, though I haven't yet removed the originals so they can be debugged), but is there a way to trigger a repair in Ceph? It appears even a forced deep scrub of a PG doesn't detect the issue.