Bug #18178
closedUnfound objects lost after OSD daemons restarted
0%
Description
Steps to reproduce in both Hammer and Jewel:
1. Create an EC pool, in my case, named ecpool-01, in k3+m2;
2. Fill ecpool-01 with some data;
3. Go into any OSD xxx/current/ data folder and pick any one object id, in the ec pool ID, such as object 1717 from
/var/lib/ceph/osd/osd0/current/7.35s3_head/object1717__head_20CC4775__7_ffffffffffffffff_3
where 7 is ecpool-01's pool ID, 7.35 is pg ID;
4. Look up the same object in all OSDs, such as:
#osd0 /var/lib/ceph/osd/osd0/current/7.35s3_head/object1717__head_20CC4775__7_ffffffffffffffff_3 # osd1 /var/lib/ceph/osd/osd1/current/7.35s1_head/object1717__head_20CC4775__7_ffffffffffffffff_1 # osd2 /var/lib/ceph/osd/osd2/current/7.35s4_head/object1717__head_20CC4775__7_ffffffffffffffff_4 # osd3 /var/lib/ceph/osd/osd3/current/7.35s0_head/object1717__head_20CC4775__7_ffffffffffffffff_0 # osd5 /var/lib/ceph/osd/osd5/current/7.35s2_head/object1717__head_20CC4775__7_ffffffffffffffff_2
5. Mess up at least 3 objects, for example, empty object shard #0, 111 bytes in shard #1, 222 bytes in shard #2;
6. Run deep scrub on the pg, 7.35;
7. Observe "pgs inconsistent" is correctly marked in "ceph status";
8. Run repair on the pg 7.35;
9. Observe 1 "unfound" object is correctly marked in "ceph status", and all 3 messed shards are in the modified size, without any fix or touch;
10. Reboot OSDs, as some suggested as a fix, such as in http://tracker.ceph.com/issues/15006;
11. Observe no error message in "ceph status", the cluster shows healthy, but all 3 messed shards are in the modified size, without any fix or touch;
12. Repeat deep scrub and repair again, and see the same errors coming back.
The real issue here is that a daemon restart should not wipe out error states and messages in Step 10.
And a minor complaint is that "unfound object" message is misleading. Unfound here actually means "Ceph cannot find CORRECT shards to repair the broken object". It confused some ones, and they claimed that they could find the (broken) objects.
Updated by Samuel Just over 7 years ago
- Assignee set to David Zafman
- Priority changed from Normal to High
David: I know why it's losing the unfound status, but it should be losing the inconsistent pg state, right?
Updated by David Zafman over 7 years ago
The issue is that repair uses recovery to fix a PG and in this scenario the recovery can't complete because the object can't be fixed. Probably not worth fixing this at least in master and Kraken because we are moving repair into user mode.
Master
After repair attempt:
cluster 65e1a997-e2b8-47d0-9183-cd99043c617c
health HEALTH_ERR
1 pgs inconsistent
1 pgs recovering
recovery 202/300 objects degraded (67.333%)
recovery 1/100 unfound (1.000%)
6 near full osd(s)
3 scrub errors
monmap e2: 1 mons at {a=127.0.0.1:40000/0}
election epoch 4, quorum 0 a
mgr no daemons active
osdmap e21: 6 osds: 6 up, 6 in
flags nearfull,sortbitwise,require_jewel_osds,require_kraken_osds
pgmap v134: 9 pgs, 2 pools, 155 kB data, 100 objects
532 GB used, 68973 MB / 599 GB avail
202/300 objects degraded (67.333%)
1/100 unfound (1.000%)
8 active+clean
1 active+recovering+inconsistentAfter cluster restart:
cluster 65e1a997-e2b8-47d0-9183-cd99043c617c
health HEALTH_ERR
1 pgs degraded
1 pgs inconsistent
1 pgs recovering
recovery 1/300 objects degraded (0.333%)
6 near full osd(s)
3 scrub errors
monmap e2: 1 mons at {a=127.0.0.1:40000/0}
election epoch 5, quorum 0 a
mgr no daemons active
osdmap e24: 6 osds: 6 up, 6 in
flags nearfull,sortbitwise,require_jewel_osds,require_kraken_osds
pgmap v193: 9 pgs, 2 pools, 155 kB data, 100 objects
532 GB used, 69174 MB / 599 GB avail
1/300 objects degraded (0.333%)
8 active+clean
1 active+recovering+degraded+inconsistentUpdated by Greg Farnum almost 7 years ago
- Project changed from Ceph to RADOS
- Category set to Scrub/Repair
- Component(RADOS) OSD added
Updated by David Zafman about 6 years ago
- Related to Bug #18162: osd/ReplicatedPG.cc: recover_replicas: object added to missing set for backfill, but is not in recovering, error! added
Updated by David Zafman about 6 years ago
- Status changed from New to Won't Fix
Reasons this is being close
1. PG repair is moving to user mode so on the fly object repair probably won't use recovery anymore
2. The scrub "inconsistent" state is maintained across a reboot
3. New recovery_unfound and backfill_unfound indicate that the recovery is stuck (bug #18162)