Project

General

Profile

Actions

Bug #14666

closed

memstore: fails to set cid on collection

Added by Greg Farnum about 8 years ago. Updated about 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
-
Category:
OSD
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

One example here http://pulpito.ceph.com/gregf-2016-02-04_07:29:11-fs-master---basic-mira/6616/ but I've been getting this failure in this yaml on every run for a day or three.

2016-02-04 23:54:42.655911 7ff3e6f78700  7 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] sub_op_scrub_map
2016-02-04 23:54:42.655933 7ff3e6f78700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep]  got 1 scrub map
2016-02-04 23:54:42.655954 7ff3e6f78700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] map version is 120'5
2016-02-04 23:54:42.655969 7ff3e6f78700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] requeue_scrub: queueing
2016-02-04 23:54:42.655989 7ff3e6f78700 10 osd.2 120 dequeue_op 0x7ff40a1b0f00 finish
2016-02-04 23:54:42.656023 7ff3e4773700 20 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub state WAIT_REPLICAS
2016-02-04 23:54:42.656058 7ff3e4773700 20 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub state COMPARE_MAPS
2016-02-04 23:54:42.656081 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub_compare_maps has maps, analyzing
2016-02-04 23:54:42.656095 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub_compare_maps  comparing replica scru
b maps
2016-02-04 23:54:42.656108 7ff3e4773700  2 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub_compare_maps   osd.2 has 0 items
2016-02-04 23:54:42.656123 7ff3e4773700  2 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub_compare_maps replica 1 has 0 items
2016-02-04 23:54:42.656141 7ff3e4773700  2 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] 
2016-02-04 23:54:42.656157 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] _scrub
2016-02-04 23:54:42.656175 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] _scrub (deep-scrub) finish
2016-02-04 23:54:42.656189 7ff3e4773700 15 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep]  requeue_ops 
2016-02-04 23:54:42.656204 7ff3e4773700 20 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub state FINISH
2016-02-04 23:54:42.656217 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] deep-scrub got 0/3 objects, 0/0 clones, 0/
3 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/12582912 bytes, 0/0 hit_set_archive bytes.
2016-02-04 23:54:42.656244 7ff3e4773700 -1 log_channel(cluster) log [ERR] : 22.5 deep-scrub stat mismatch, got 0/3 objects, 0/0 clones, 0/3 dirty, 0/0 omap, 0/0 pinned, 0/0 hit_set_archive, 0/0 whiteouts, 0/12582912 bytes, 0/0 hit_set_archive bytes.
2016-02-04 23:54:42.656253 7ff3e4773700 10 osd.2 pg_epoch: 120 pg[22.5( v 120'5 (0'0,120'5] local-les=115 n=3 ec=114 les/c/f 115/115/119 114/114/114) [2,1] r=0 lpr=114 crt=120'3 lcod 120'4 mlcod 120'4 active+clean+scrubbing+deep] scrub_process_inconsistent: checking authoritative
2016-02-04 23:54:42.656270 7ff3e4773700 -1 log_channel(cluster) log [ERR] : 22.5 deep-scrub 1 errors
Actions #1

Updated by Sage Weil about 8 years ago

  • Status changed from New to Fix Under Review
Actions #2

Updated by Sage Weil about 8 years ago

  • Subject changed from nightlies are failing scrub on stat mismatch to memstore: fails to set cid on collection
Actions #3

Updated by Sage Weil about 8 years ago

  • Status changed from Fix Under Review to Resolved
Actions

Also available in: Atom PDF