Project

General

Profile

Bug #40454

snap_mapper error, scrub gets r -2..repaired

Added by Sage Weil almost 5 years ago. Updated over 3 years ago.

Status:
Can't reproduce
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper.remove_oid 2:c54cd850:::smithi15416397-271:68
2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper._remove_oid 2:c54cd850:::smithi15416397-271:68
2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper.get_snaps 2:c54cd850:::smithi15416397-271:68 got.empty()
2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper.add_oid 2:c54cd850:::smithi15416397-271:68 65,67
2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper.get_snaps 2:c54cd850:::smithi15416397-271:68 got.empty()
2019-06-19T18:35:23.157+0000 7f1767ab7700 20 snap_mapper.set_snaps 2:c54cd850:::smithi15416397-271:68 65,67
...
2019-06-19T18:37:00.923+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c54cd850:::smithi15416397-271:68 snaps missing in mapper, should be: 65,67 was  r -2...repaired
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.add_oid 2:c54cd850:::smithi15416397-271:68 65,67
2019-06-19T18:37:00.923+0000 7f0a594e4700 15 bluestore(/var/lib/ceph/osd/ceph-3) omap_get_values meta oid #-1:c0371625:::snapmapper:0#
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 bluestore.onode(0x557150ea2480).flush flush done
2019-06-19T18:37:00.923+0000 7f0a594e4700 10 bluestore(/var/lib/ceph/osd/ceph-3) omap_get_values meta oid #-1:c0371625:::snapmapper:0# = 0
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.get_snaps 2:c54cd850:::smithi15416397-271:68 got.empty()
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.set_snaps 2:c54cd850:::smithi15416397-271:68 65,67
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.set_snaps set OBJ_0000000000000002.3A23B1A0.68.smithi15416397-271..
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.add_oid set MAP_0000000000000065_0000000000000002.3A23B1A0.68.smithi15416397-271..
2019-06-19T18:37:00.923+0000 7f0a594e4700 20 snap_mapper.add_oid set MAP_0000000000000067_0000000000000002.3A23B1A0.68.smithi15416397-271..

/a/sage-2019-06-19_13:01:10-rados:thrash-wip-sloppy-snaps-distro-basic-smithi/4048465

The earlier variations of this bug appear to have been closed (#38840, #23909), but I don't think there is any racing operation here.


Related issues

Related to RADOS - Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired Resolved 04/27/2018
Related to RADOS - Bug #38840: snaps missing in mapper, should be: ca was r -2...repaired Resolved
Related to bluestore - Bug #40520: snap_mapper record resurrected: trim_object: Can not trim 3:205afc9d:::smithi13915600-489:124 repair needed (no obc) Can't reproduce

History

#1 Updated by Sage Weil almost 5 years ago

  • Related to Bug #23909: snaps missing in mapper, should be: 188,18f,191,195,197,198,199,19d,19e,1a1,1a3 was r -2...repaired added

#2 Updated by Sage Weil almost 5 years ago

  • Related to Bug #38840: snaps missing in mapper, should be: ca was r -2...repaired added

#3 Updated by Sage Weil almost 5 years ago

Also note there are a lot of objects in this PG that are similarly affected:

$ grep ERR remote/*/log/*osd.3*
2019-06-19T18:37:00.923+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c54cd850:::smithi15416397-271:68 snaps missing in mapper, should be: 65,67 was  r -2...repaired
2019-06-19T18:37:00.923+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c54cd850:::smithi15416397-271:46 snaps missing in mapper, should be: 3b,3c,3e,3f,41,43,45,46 was  r -2...repaired
2019-06-19T18:37:00.924+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c54cd850:::smithi15416397-271:d snaps missing in mapper, should be: 8,b,c,d was  r -2...repaired
2019-06-19T18:37:00.924+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c281eb34:::smithi15416397-389:62 snaps missing in mapper, should be: 5e,62 was  r -2...repaired
2019-06-19T18:37:00.924+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c262d77f:::smithi15416397-345:35 snaps missing in mapper, should be: 23,2e,30,33,34,35 was  r -2...repaired
2019-06-19T18:37:00.928+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c5d5d72d:::smithi15416397-343:5b snaps missing in mapper, should be: 4e,51,55,59,5a,5b was  r -2...repaired
2019-06-19T18:37:00.928+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c5d5d72d:::smithi15416397-343:26 snaps missing in mapper, should be: 25 was  r -2...repaired
2019-06-19T18:37:00.929+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:c5d5d72d:::smithi15416397-343:23 snaps missing in mapper, should be: 1e,20,21,22,23 was  r -2...repaired
2019-06-19T18:37:00.934+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d3da0cf0:::smithi15416397-130:67 snaps missing in mapper, should be: 65,66,67 was  r -2...repaired
2019-06-19T18:37:00.934+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d27dc6ae:::smithi15416397-36:5b snaps missing in mapper, should be: 4e,51,55,59,5b was  r -2...repaired
2019-06-19T18:37:00.934+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d2759381:::smithi15416397-410:36 snaps missing in mapper, should be: 23,2e,30,33,34,35,36 was  r -2...repaired
2019-06-19T18:37:00.934+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d2759381:::smithi15416397-410:19 snaps missing in mapper, should be: f,12,15,17,18,19 was  r -2...repaired
2019-06-19T18:37:00.939+0000 7f0a5d4ec700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d540feb8:::smithi15416397-480:27 snaps missing in mapper, should be: 20,21,23,25,27 was  r -2...repaired
2019-06-19T18:37:00.942+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:df97c111:::smithi15416397-177:6f snaps missing in mapper, should be: 6f was  r -2...repaired
2019-06-19T18:37:00.942+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:df958ce8:::smithi15416397-173:64 snaps missing in mapper, should be: 63,64 was  r -2...repaired
2019-06-19T18:37:00.942+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:df958ce8:::smithi15416397-173:37 snaps missing in mapper, should be: 23,2e,30,33,34,35,36,37 was  r -2...repaired
2019-06-19T18:37:00.943+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:d9fe434e:::smithi15416397-351:6e snaps missing in mapper, should be: 6c,6e was  r -2...repaired
2019-06-19T18:37:00.950+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:e919df42:::smithi15416397-493:47 snaps missing in mapper, should be: 3f,43,45,46,47 was  r -2...repaired
2019-06-19T18:37:00.950+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:e919df42:::smithi15416397-493:39 snaps missing in mapper, should be: 2e,33,36,38,39 was  r -2...repaired
2019-06-19T18:37:00.950+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:e919df42:::smithi15416397-493:2a snaps missing in mapper, should be: 23,28,29,2a was  r -2...repaired
2019-06-19T18:37:00.950+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:e8966df2:::smithi15416397-279:43 snaps missing in mapper, should be: 38,3b,3c,3e,3f,41,43 was  r -2...repaired
2019-06-19T18:37:00.954+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:e984d006:::smithi15416397-112:61 snaps missing in mapper, should be: 5e,5f,60,61 was  r -2...repaired
2019-06-19T18:37:00.962+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:f739dbfb:::smithi15416397-443:4b snaps missing in mapper, should be: 45,47,48,49,4a,4b was  r -2...repaired
2019-06-19T18:37:00.963+0000 7f0a594e4700 -1 log_channel(cluster) log [ERR] : osd.3 found snap mapper error on pg 2.3 oid 2:f629205d:::smithi15416397-5:58 snaps missing in mapper, should be: 4a,4b,4c,4e,51,52,55,57,58 was  r -2...repaired

#4 Updated by Greg Farnum almost 5 years ago

Neha reports this hasn't been seen in master in a while; is there a reason you think it's not a new-in-branch bug?

#5 Updated by Sage Weil almost 5 years ago

  • Related to Bug #40520: snap_mapper record resurrected: trim_object: Can not trim 3:205afc9d:::smithi13915600-489:124 repair needed (no obc) added

#6 Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New

#7 Updated by Neha Ojha over 3 years ago

  • Status changed from New to Can't reproduce

Also available in: Atom PDF