Project

General

Profile

Actions
Copy link

Bug #12785

closed

shard 4 missing -2/00000014/temp_recovering_0.14_25'135_118_head/head

Added by Josh Durgin over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

http://pulpito.ceph.com/teuthology-2015-08-22_23:06:01-krbd-master-testing-basic-multi/1027470/

 "2015-08-24 15:41:34.292354 osd.1 10.214.131.33:6808/8614 6 : cluster [ERR] 0.14 shard 4 missing -2/00000014/temp_recovering_0.14_25'135_118_head/head" in cluster log 
<pre>

Looking at osd.1 (sdc on plana07) I don't see any signs of disk issues.
Actions
Copy link
 #1

Updated by Sage Weil over 8 years ago

  • Status changed from New to 7
  • Assignee set to Sage Weil

off by one error in hobject_t::is_temp() breaks the temp cleanup:

2015-08-24 15:25:26.371369 7f09b9f64980 20 osd.1 119  clearing temps in 0.14_head pgid 0.14
2015-08-24 15:25:26.371380 7f09b9f64980 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list pool is 0 shard is 255 pgid 0.14
2015-08-24 15:25:26.371385 7f09b9f64980 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list first checking temp pool
2015-08-24 15:25:26.371394 7f09b9f64980 20 filestore(/var/lib/ceph/osd/ceph-1) collection_list pool is -2 shard is 255 pgid 0.14
2015-08-24 15:25:26.371447 7f09b9f64980 20 _collection_list_partial start:GHMIN end:GHMAX-64 ls.size 0
2015-08-24 15:25:26.371577 7f09b9f64980 20 list_by_hash_bitwise prefix 41000000
2015-08-24 15:25:26.371581 7f09b9f64980 20 list_by_hash_bitwise prefix 41000000 ob -2/00000014/temp_recovering_0.14_25'135_118_head/head
2015-08-24 15:25:26.371586 7f09b9f64980 20 filestore(/var/lib/ceph/osd/ceph-1) objects: 0x7ffc6e5fa800
2015-08-24 15:25:26.371590 7f09b9f64980 10 filestore(/var/lib/ceph/osd/ceph-1) collection_list fall through to non-temp collection, start -1/00000000//0
2015-08-24 15:25:26.371597 7f09b9f64980 20 _collection_list_partial start:-1/00000000//0 end:GHMAX-64 ls.size 1
2015-08-24 15:25:26.371827 7f09b9f64980 20 list_by_hash_bitwise prefix 41000000
2015-08-24 15:25:26.371831 7f09b9f64980 20 list_by_hash_bitwise prefix 41000000 ob 0/00000014//head
2015-08-24 15:25:26.371834 7f09b9f64980 20 list_by_hash_bitwise prefix 410CB9F8
2015-08-24 15:25:26.371836 7f09b9f64980 20 list_by_hash_bitwise prefix 410CB9F8 ob 0/8f9bc014/rbd_data.101b2ae8944a.000000000000020a/head
2015-08-24 15:25:26.371838 7f09b9f64980 20 list_by_hash_bitwise prefix 4106C0BE
2015-08-24 15:25:26.371839 7f09b9f64980 20 list_by_hash_bitwise prefix 4106C0BE ob 0/eb0c6014/rbd_data.101b2ae8944a.000000000000009d/head
2015-08-24 15:25:26.371841 7f09b9f64980 20 list_by_hash_bitwise prefix 41094C09
2015-08-24 15:25:26.371842 7f09b9f64980 20 list_by_hash_bitwise prefix 41094C09 ob 0/90c49014/rbd_data.101b2ae8944a.00000000000000d9/head
2015-08-24 15:25:26.371844 7f09b9f64980 20 list_by_hash_bitwise prefix 4122BD36
2015-08-24 15:25:26.371845 7f09b9f64980 20 list_by_hash_bitwise prefix 4122BD36 ob 0/63db2214/rbd_data.101b2ae8944a.00000000000001fc/head
2015-08-24 15:25:26.371847 7f09b9f64980 20 list_by_hash_bitwise prefix 416B6D8A
2015-08-24 15:25:26.371848 7f09b9f64980 20 list_by_hash_bitwise prefix 416B6D8A ob 0/a8d6b614/rbd_data.101b2ae8944a.000000000000045f/head
2015-08-24 15:25:26.371849 7f09b9f64980 20 list_by_hash_bitwise prefix 41E00403
2015-08-24 15:25:26.371850 7f09b9f64980 20 list_by_hash_bitwise prefix 41E00403 ob 0/30400e14/rbd_data.101b2ae8944a.00000000000000b3/head
2015-08-24 15:25:26.371852 7f09b9f64980 20 list_by_hash_bitwise prefix 41EB318C
2015-08-24 15:25:26.371853 7f09b9f64980 20 list_by_hash_bitwise prefix 41EB318C ob 0/c813be14/rbd_data.101b2ae8944a.00000000000007f4/head
2015-08-24 15:25:26.371855 7f09b9f64980 20 list_by_hash_bitwise prefix 41DBD890
2015-08-24 15:25:26.371856 7f09b9f64980 20 list_by_hash_bitwise prefix 41DBD890 ob 0/098dbd14/rbd_data.101b2ae8944a.0000000000000558/head
2015-08-24 15:25:26.371858 7f09b9f64980 20 list_by_hash_bitwise prefix 41D72792
2015-08-24 15:25:26.371859 7f09b9f64980 20 list_by_hash_bitwise prefix 41D72792 ob 0/29727d14/rbd_data.101b2ae8944a.0000000000000a82/head
2015-08-24 15:25:26.371864 7f09b9f64980 20 filestore(/var/lib/ceph/osd/ceph-1) objects: 0x7ffc6e5fa800

... doesn't delete it! ...

2015-08-24 15:25:26.371867 7f09b9f64980 20 osd.1 119  clearing temps in 0.7_head pgid 0.7
Actions
Copy link
 #2

Updated by Sage Weil over 8 years ago

  • Status changed from 7 to Resolved
Actions
Copy link

Also available in: Atom PDF