Ceph » RADOS

There is an object with peer missing:

peer osd.20(0) missing {2:d0cc2acb:::10000080236.0000007d:head=0'0}

An EIO from the 46(6) shard:

2016-12-06 14:33:35.757133 7f335c282700 10 osd.3 pg_epoch: 110014 pg[2.bs2( v 94305'214276 (78823'204254,94305'214276] local-les=110014 n=137180 ec=136 les/c/f 110014/79592/109085 110010/110010/110010) [20,2147483647,3,39,60,7,53,48,55,10,26,43,59,37]/[2147483647,2147483647,3,32,37,19,46,33,40,11,28,21,44,15] r=2 lpr=110010 pi=63136-110009/3155 rops=1 bft=7(5),10(9),20(0),26(10),37(13),39(3),43(11),48(7),53(6),55(8),59(12),60(4) crt=91692'214272 lcod 0'0 mlcod 0'0 active+undersized+degraded+remapped+backfilling NIBBLEWISE] _failed_push: Read error 2:d0cc2acb:::10000080236.0000007d:head r=0 errors={46(6)=-5}

The _failed_push() removed the object from recovering, but it is after last_backfill, so it is expected to be there.

Another commit has been added to PR12088 to fix a problem. The scenario in this bug might not be impacted, but I thought I'd mention it.

David Zafman wrote:

Another commit has been added to PR12088 to fix a problem. The scenario in this bug might not be impacted, but I thought I'd mention it.

I got the impression this additional commit fixes an unrelated issue, but I will compile again and test soon.

David Zafman wrote:

Another commit has been added to PR12088 to fix a problem. The scenario in this bug might not be impacted, but I thought I'd mention it.

No joy. I tried both building the PR itself (based on 10.2.3) and manually merging the PR changeset onto the tag 10.2.4. In both cases, the behavior is the same as this bug report. I think this issue is distinct from what PR12088 is addressing.

Is there anything I can try to mitigate the problem without a code change? You mentioned the order of operations is leading to an expectation being violated - can I reset the expectation somehow?

My cluster has been operationally down for over a month now due to these two issues :)

Find all the "_failed_push: Read error" message in the logs of crashing OSDs and note the errors={osd#(shard#)=-5}. See log above which indicates and EIO from the disk for osd 46. shard# doesn't matter. Hopefully, there is a limited number of bad disks.

So for osd.46, for example, set noout and take osd.46 down. Replace the drive and bring osd.46 back. This should repopulate the new osd.46 disk. There are probably detailed instructions on how to replace a disk in a ceph how-to somewhere. Don't forget to unset noout once you've replaced all the bad drives.

David Zafman wrote:

Find all the "_failed_push: Read error" message in the logs of crashing OSDs and note the errors={osd#(shard#)=-5}. See log above which indicates and EIO from the disk for osd 46. shard# doesn't matter. Hopefully, there is a limited number of bad disks.

So for osd.46, for example, set noout and take osd.46 down. Replace the drive and bring osd.46 back. This should repopulate the new osd.46 disk. There are probably detailed instructions on how to replace a disk in a ceph how-to somewhere. Don't forget to unset noout once you've replaced all the bad drives.

I'm not entirely sure I follow - are you suggesting replacing the OSD's backing store without removing the OSD from the CRUSH map? Or the "normal" process of killing the process, removing the OSD from the CRUSH map, then creating a new OSD, adding to CRUSH map, and starting the daemon?

I am entirely certain (by experimentation) that too many OSDs are having the read errors to successfully bring all PGs up if I replace all the OSDs having this problem. I don't know how my cluster got to this state, but there are a few missing objects here and there across 20+ OSDs in a 65 OSD cluster, with k=10 m=4 erasure coding on the pool in question. Based on the log message above, it seems only one of 10 OSDs is missing the object, so I don't understand what's stopping the erasure code from recovering the chunk.

Any advice or pointers to in-depth documentation would be appreciated. I'll happily go through and manually resurrect objects as necessary, if need be, but I have no idea how to manually recover a missing EC chunk.

Thanks!

I'm running an experiment against the current kraken branch, preventing the ceph_abort() call which produces this error. Will update when I have results.

https://work.techtonium.com/bitbucket/users/aaron/repos/ceph/commits/aa6226d22e2c5aa1265bd2d15c2488a8f49c5c41

Aaron Ten Clay wrote:

I'm running an experiment against the current kraken branch, preventing the ceph_abort() call which produces this error. Will update when I have results.

https://work.techtonium.com/bitbucket/users/aaron/repos/ceph/commits/aa6226d22e2c5aa1265bd2d15c2488a8f49c5c41

Not unexpectedly, this hack did not resolve the issue. It led to a crash when the map of recovering PGs is not empty in PrimaryLogPG::_clear_recovery_state(). I'm not familiar enough with Ceph internals to go any further. Would greatly appreciate pointers. Not afraid to compile and run proposed fixes :)

Thanks,
-Aaron

Nevermind the patch; clearing backfills_in_flight seems to be wrong and dangerous; I think it might even make the object seem to disappear, or even actually disappear.

I am now trying to leave it alone, and have recover_replicas behave in a more useful way if it finds the object already in backfills_in_flight. I'll let you know how that goes once I've implemented it and given it at least some testing.

Alexandre Oliva wrote:

Here's another incremental patch, that fixes a problem in which the presence of multiple read errors in the same PG might cause the backfill interval to advance past some failures, discarding the information required to retry them, so that recovery will reach the end of the backfill and claim to complete recovery with missing replicas, but actually keeping a backfill slot busy. Fixing it required removing the assumption that, when recover_backfill is called and there are pending retries, the recovery set is empty. While at that, I arranged for the test that guards the verbose logging of retries to be skipped if we do not have pending retries for sure.

Now, is there anyone watching this space, that could integrate these patches, or should I post them elsewhere?

I'd love to see these patches make their way into 11.x, which I upgraded to hoping to get resolution and found none. Of course now my OSD's cant be downgraded :D I hope some maintainers are watching this.

Now, is there anyone watching this space, that could integrate these patches, or should I post them elsewhere?

You should post your patches as a GitHub pull request against the master branch of ceph/ceph.git. Make sure the commit description includes the following two lines at the end:

Fixes: http://tracker.ceph.com/issues/18162
Signed-off-by: Your Name <me@example.com>

For detailed instructions on submitting patches, see https://github.com/ceph/ceph/blob/master/SubmittingPatches.rst

Sorry, github is off-limits for me (it tries to run non-Free Software on my browser, and it refuses to work if I don't let it). I'll look into posting the patches to the ceph-devel list.

That said, I'm afraid there's still something wrong even after with all of them installed, at least with non-default settings allowing starting multiple reads in a single operation. If we do so, and we get errors from different OSDs, the retry for both will insist on one of the failing OSDs, and fail again and again. I haven't been able to fix that yet. Should I submit the patchset to the list despite this caveat?

Hi

My institute has a large cluster running Kraken 11.2.1-0 and using EC 8+3 and believe we have run into this bug. I was wondering if any progress had been made on this?

I should also point out that I think the severity is set incorrectly (It should be much higher). The only solution to this problem we have found is to delete and recreate PGs which obviously causes data loss. You cannot run a production quality service using EC pools with this bug.

I am afraid I am not in a position to write/commit code but if there are other ways I could help with the resolution of the problem, I would certainly be willing to try.

Alastair

Is there a reason this hasn't been marked for back porting to Luminous yet? (Apologies if this is just because you haven't had time).