Actions
Bug #38637
closedBlueStore::ExtentMap::fault_range() assert
Status:
Won't Fix
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Hi,
I have rook with ceph ceph-12.2.4
3 Mon, 5 OSD.
For a last few hours one of my OSD is in crashing loop.
2019-03-08 09:02:19.245758 7feec44ed700 20 bluestore(/var/lib/rook/osd2) _txc_finish 0x5610c8f81300 onodes
2019-03-08 09:02:19.385084 I | osd2: -64> 2019-03-08 09:02:19.245758 7feec44ed700 20 bluestore(/var/lib/rook/osd2) _txc_finish txc 0x5610c8f81300 done
2019-03-08 09:02:19.385089 I | osd2: -63> 2019-03-08 09:02:19.245759 7feec44ed700 20 bluestore(/var/lib/rook/osd2) _txc_finish osr 0x5610c2d3cc40 q now empty
2019-03-08 09:02:19.385094 I | osd2: -62> 2019-03-08 09:02:19.245761 7feec44ed700 10 bluestore(/var/lib/rook/osd2) _txc_release_alloc 0x5610c8f81300 []
2019-03-08 09:02:19.385110 I | osd2: -61> 2019-03-08 09:02:19.245767 7feec44ed700 10 bluestore(/var/lib/rook/osd2) _reap_collections all reaped
2019-03-08 09:02:19.385123 I | osd2: -60> 2019-03-08 09:02:19.245775 7feec44ed700 20 bluestore(/var/lib/rook/osd2) _kv_finalize_thread sleep
2019-03-08 09:02:19.385129 I | osd2: -59> 2019-03-08 09:02:19.245803 7feeb4cce700 20 bluestore(/var/lib/rook/osd2).collection(1.3e_head 0x5610c2651200) r 0 v.len 429
2019-03-08 09:02:19.385133 I | osd2: -58> 2019-03-08 09:02:19.245828 7feeb4cce700 30 bluestore.OnodeSpace(0x5610c2651368 in 0x5610c2194220) add #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# 0x5610c7f3a840
2019-03-08 09:02:19.385138 I | osd2: -57> 2019-03-08 09:02:19.245835 7feeb4cce700 10 bluestore(/var/lib/rook/osd2) getattr 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# _ = 0
2019-03-08 09:02:19.385144 I | osd2: -56> 2019-03-08 09:02:19.245854 7feeb4cce700 15 bluestore(/var/lib/rook/osd2) getattr 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# snapset
2019-03-08 09:02:19.385148 I | osd2: -55> 2019-03-08 09:02:19.245871 7feeb4cce700 30 bluestore.OnodeSpace(0x5610c2651368 in 0x5610c2194220) lookup
2019-03-08 09:02:19.385152 I | osd2: -54> 2019-03-08 09:02:19.245874 7feeb4cce700 30 bluestore.OnodeSpace(0x5610c2651368 in 0x5610c2194220) lookup #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# hit 0x5610c7f3a840
2019-03-08 09:02:19.385157 I | osd2: -53> 2019-03-08 09:02:19.245877 7feeb4cce700 10 bluestore(/var/lib/rook/osd2) getattr 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# snapset = 0
2019-03-08 09:02:19.385163 I | osd2: -52> 2019-03-08 09:02:19.245894 7feeb5cd0700 20 bluestore(/var/lib/rook/osd2).collection(1.1e_head 0x5610c264fa00) r 0 v.len 429
2019-03-08 09:02:19.385168 I | osd2: -51> 2019-03-08 09:02:19.245909 7feeb5cd0700 30 bluestore.OnodeSpace(0x5610c264fb68 in 0x5610c2192e00) add #1:7975ea66:::rbd_data.138142643c9869.0000000000000100:head# 0x5610c8dabb80
2019-03-08 09:02:19.385172 I | osd2: -50> 2019-03-08 09:02:19.245914 7feeb5cd0700 10 bluestore(/var/lib/rook/osd2) getattr 1.1e_head #1:7975ea66:::rbd_data.138142643c9869.0000000000000100:head# _ = 0
2019-03-08 09:02:19.385177 I | osd2: -49> 2019-03-08 09:02:19.245927 7feeb5cd0700 15 bluestore(/var/lib/rook/osd2) getattr 1.1e_head #1:7975ea66:::rbd_data.138142643c9869.0000000000000100:head# snapset
2019-03-08 09:02:19.385182 I | osd2: -48> 2019-03-08 09:02:19.245930 7feeb5cd0700 30 bluestore.OnodeSpace(0x5610c264fb68 in 0x5610c2192e00) lookup
2019-03-08 09:02:19.385187 I | osd2: -47> 2019-03-08 09:02:19.245932 7feeb5cd0700 30 bluestore.OnodeSpace(0x5610c264fb68 in 0x5610c2192e00) lookup #1:7975ea66:::rbd_data.138142643c9869.0000000000000100:head# hit 0x5610c8dabb80
2019-03-08 09:02:19.385191 I | osd2: -46> 2019-03-08 09:02:19.245935 7feeb5cd0700 10 bluestore(/var/lib/rook/osd2) getattr 1.1e_head #1:7975ea66:::rbd_data.138142643c9869.0000000000000100:head# snapset = 0
2019-03-08 09:02:19.385195 I | osd2: -45> 2019-03-08 09:02:19.245958 7feeb4cce700 10 bluestore(/var/lib/rook/osd2) queue_transactions existing 0x5610c3681dc0 osr(1.3e 0x5610c3107ac0)
2019-03-08 09:02:19.385201 I | osd2: -44> 2019-03-08 09:02:19.245974 7feeb4cce700 20 bluestore(/var/lib/rook/osd2) _txc_create osr 0x5610c3681dc0 = 0x5610c86c8a00 seq 6
2019-03-08 09:02:19.385206 I | osd2: -43> 2019-03-08 09:02:19.245979 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) transaction dump:
2019-03-08 09:02:19.385211 I | osd2: {
2019-03-08 09:02:19.385215 I | osd2: "ops": [
2019-03-08 09:02:19.385219 I | osd2: {
2019-03-08 09:02:19.385224 I | osd2: "op_num": 0,
2019-03-08 09:02:19.385228 I | osd2: "op_name": "setattrs",
2019-03-08 09:02:19.385232 I | osd2: "collection": "1.3e_head",
2019-03-08 09:02:19.385237 I | osd2: "oid": "#1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head#",
2019-03-08 09:02:19.385242 I | osd2: "attr_lens": {
2019-03-08 09:02:19.385247 I | osd2: "_": 295,
2019-03-08 09:02:19.385251 I | osd2: "snapset": 35
2019-03-08 09:02:19.385255 I | osd2: }
2019-03-08 09:02:19.385259 I | osd2: },
2019-03-08 09:02:19.385263 I | osd2: {
2019-03-08 09:02:19.385268 I | osd2: "op_num": 1,
2019-03-08 09:02:19.385284 I | osd2: "op_name": "op_setallochint",
2019-03-08 09:02:19.385297 I | osd2: "collection": "1.3e_head",
2019-03-08 09:02:19.385302 I | osd2: "oid": "#1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head#",
2019-03-08 09:02:19.385307 I | osd2: "expected_object_size": "4194304",
2019-03-08 09:02:19.385311 I | osd2: "expected_write_size": "4194304"
2019-03-08 09:02:19.385315 I | osd2: },
2019-03-08 09:02:19.385320 I | osd2: {
2019-03-08 09:02:19.385324 I | osd2: "op_num": 2,
2019-03-08 09:02:19.385328 I | osd2: "op_name": "write",
2019-03-08 09:02:19.385332 I | osd2: "collection": "1.3e_head",
2019-03-08 09:02:19.385336 I | osd2: "oid": "#1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head#",
2019-03-08 09:02:19.385341 I | osd2: "length": 4096,
2019-03-08 09:02:19.385345 I | osd2: "offset": 2740224,
2019-03-08 09:02:19.385350 I | osd2: "bufferlist length": 4096
2019-03-08 09:02:19.385354 I | osd2: },
2019-03-08 09:02:19.385358 I | osd2: {
2019-03-08 09:02:19.385362 I | osd2: "op_num": 3,
2019-03-08 09:02:19.385366 I | osd2: "op_name": "omap_setkeys",
2019-03-08 09:02:19.385371 I | osd2: "collection": "1.3e_head",
2019-03-08 09:02:19.385375 I | osd2: "oid": "#1:7c000000::::head#",
2019-03-08 09:02:19.385380 I | osd2: "attr_lens": {
2019-03-08 09:02:19.385384 I | osd2: "0000004013.00000000000004644411": 184,
2019-03-08 09:02:19.385388 I | osd2: "_fastinfo": 186
2019-03-08 09:02:19.385392 I | osd2: }
2019-03-08 09:02:19.385396 I | osd2: }
2019-03-08 09:02:19.385401 I | osd2: ]
2019-03-08 09:02:19.385405 I | osd2: }
2019-03-08 09:02:19.385409 I | osd2:
2019-03-08 09:02:19.385414 I | osd2: -42> 2019-03-08 09:02:19.245846 7feecbce5700 5 -- 10.233.115.2:6801/988 >> 10.233.66.39:6801/1000069 conn(0x5610c8764800 :6801 s=STATE_OPEN_MESSAGE_READ_FOOTER_AND_DISPATCH pgs=1738 cs=1 l=0). rx osd.1 seq 94 0x5610c918a800 pg_info((query:4013 sent:4013 3.4es1( empty local-lis/les=4012/4013 n=0 ec=33/33 lis/c 4012/3972 les/c/f 4013/3973/0 4012/4012/4012) 1->0)=(empty) epoch 4013) v5
2019-03-08 09:02:19.385418 I | osd2: -41> 2019-03-08 09:02:19.246026 7feeb4cce700 30 bluestore.OnodeSpace(0x5610c2651368 in 0x5610c2194220) lookup
2019-03-08 09:02:19.385423 I | osd2: -40> 2019-03-08 09:02:19.246028 7feeb4cce700 30 bluestore.OnodeSpace(0x5610c2651368 in 0x5610c2194220) lookup #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# hit 0x5610c7f3a840
2019-03-08 09:02:19.385428 I | osd2: -39> 2019-03-08 09:02:19.246034 7feeb4cce700 15 bluestore(/var/lib/rook/osd2) _setattrs 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# 2 keys
2019-03-08 09:02:19.385433 I | osd2: -38> 2019-03-08 09:02:19.246040 7feeb4cce700 10 bluestore(/var/lib/rook/osd2) _setattrs 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# 2 keys = 0
2019-03-08 09:02:19.385438 I | osd2: -37> 2019-03-08 09:02:19.246043 7feeb4cce700 15 bluestore(/var/lib/rook/osd2) _set_alloc_hint 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# object_size 4194304 write_size 4194304 flags -
2019-03-08 09:02:19.385443 I | osd2: -36> 2019-03-08 09:02:19.246046 7feeb4cce700 10 bluestore(/var/lib/rook/osd2) _set_alloc_hint 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# object_size 4194304 write_size 4194304 flags - = 0
2019-03-08 09:02:19.385447 I | osd2: -35> 2019-03-08 09:02:19.246051 7feeb4cce700 15 bluestore(/var/lib/rook/osd2) _write 1.3e_head #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# 0x29d000~1000
2019-03-08 09:02:19.385452 I | osd2: -34> 2019-03-08 09:02:19.246053 7feeb4cce700 20 bluestore(/var/lib/rook/osd2) _do_write #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# 0x29d000~1000 - have 0x400000 (4194304) bytes fadvise_flags 0x0
2019-03-08 09:02:19.385458 I | osd2: -33> 2019-03-08 09:02:19.246056 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_onode 0x5610c7f3a840 #1:7f320877:::rbd_data.13131c74b0dc51.000000000000010a:head# nid 19315 size 0x400000 (4194304) expected_object_size 4194304 expected_write_size 4194304 in 8 shards, 0 spanning blobs
2019-03-08 09:02:19.385466 I | osd2: -32> 2019-03-08 09:02:19.246059 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_onode attr _ len 295
2019-03-08 09:02:19.385471 I | osd2: -31> 2019-03-08 09:02:19.246060 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_onode attr snapset len 35
2019-03-08 09:02:19.385475 I | osd2: -30> 2019-03-08 09:02:19.246063 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x0(0x211 bytes)
2019-03-08 09:02:19.385480 I | osd2: -29> 2019-03-08 09:02:19.246065 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x80000(0x213 bytes)
2019-03-08 09:02:19.385484 I | osd2: -28> 2019-03-08 09:02:19.246066 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x100000(0x213 bytes)
2019-03-08 09:02:19.385490 I | osd2: -27> 2019-03-08 09:02:19.246067 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x180000(0x213 bytes)
2019-03-08 09:02:19.385494 I | osd2: -26> 2019-03-08 09:02:19.246068 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x200000(0x213 bytes)
2019-03-08 09:02:19.385526 I | osd2: -25> 2019-03-08 09:02:19.246068 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x280000(0x213 bytes)
2019-03-08 09:02:19.385545 I | osd2: -24> 2019-03-08 09:02:19.246069 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x300000(0x213 bytes)
2019-03-08 09:02:19.385558 I | osd2: -23> 2019-03-08 09:02:19.246070 7feeb4cce700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map shard 0x380000(0x213 bytes)
2019-03-08 09:02:19.385564 I | osd2: -22> 2019-03-08 09:02:19.246072 7feeb4cce700 20 bluestore(/var/lib/rook/osd2) _choose_write_options prefer csum_order 12 target_blob_size 0x80000
2019-03-08 09:02:19.385568 I | osd2: -21> 2019-03-08 09:02:19.246073 7feeb4cce700 30 bluestore.extentmap(0x5610c7f3a990) fault_range 0x29d000~1000
2019-03-08 09:02:19.385572 I | osd2: -20> 2019-03-08 09:02:19.246075 7feeb4cce700 30 bluestore.extentmap(0x5610c7f3a990) fault_range opening shard 0x280000
2019-03-08 09:02:19.385577 I | osd2: -19> 2019-03-08 09:02:19.246101 7feec055f700 1 -- 10.233.115.2:6801/988 <== osd.1 10.233.66.39:6801/1000069 94 ==== pg_info((query:4013 sent:4013 3.4es1( empty local-lis/les=4012/4013 n=0 ec=33/33 lis/c 4012/3972 les/c/f 4013/3973/0 4012/4012/4012) 1->0)=(empty) epoch 4013) v5 ==== 922+0+0 (3832126841 0 0) 0x5610c918a800 con 0x5610c8764800
2019-03-08 09:02:19.385585 I | osd2: -18> 2019-03-08 09:02:19.245762 7feeb44cd700 30 bluestore(/var/lib/rook/osd2) _dump_extent_map csum: [3ee420a9,7cbd9ffe,881ba617,99a2be55,c0f1c0df,5683fd68,3185c45a,7f23f6cf,4d26ffe4,9e5f326b,6a42042c,2cb59657,12941546,dc2138a5,266c728e,37a8f3fe,8b077ef4,7b8accde,a979ddf1,6d1ffe81,e0064102,d052eeeb,befbc3df,cedeb23,b0c1d28b,22995d7,23a46c21,992f3cce,e6e084f,6a85f552,c38ee289,e6053f0e,6d34d3b8,35e10f4e,d75b342d,a44884a1,cc694cff,d44c5f25,9433414b,e09d7b0b,0,0,0,0,0,0,0,0]
2019-03-08 09:02:19.385590 I | osd2: -17> 2019-03-08 09:02:19.246133 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _choose_write_options prefer csum_order 12 target_blob_size 0x80000
2019-03-08 09:02:19.385594 I | osd2: -16> 2019-03-08 09:02:19.246136 7feeb44cd700 30 bluestore.extentmap(0x5610c86be510) fault_range 0x21000~1000
2019-03-08 09:02:19.385598 I | osd2: -15> 2019-03-08 09:02:19.246139 7feeb44cd700 10 bluestore(/var/lib/rook/osd2) _do_write_small 0x21000~1000
2019-03-08 09:02:19.385604 I | osd2: -14> 2019-03-08 09:02:19.246143 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _do_write_small considering Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,10000]) SharedBlob(0x5610c8fad790 sbid 0x0)) bstart 0x0
2019-03-08 09:02:19.385608 I | osd2: -13> 2019-03-08 09:02:19.246149 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _do_write_small reading head 0x0 and tail 0x0
2019-03-08 09:02:19.385613 I | osd2: -12> 2019-03-08 09:02:19.246152 7feeb44cd700 20 bluestore.BufferSpace(0x5610c8fad7a8 in 0x5610c2194300) _discard 0x21000~1000
2019-03-08 09:02:19.385618 I | osd2: -11> 2019-03-08 09:02:19.246158 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _do_write_small deferred write 0x21000~1000 of mutable Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,10000]) SharedBlob(0x5610c8fad790 sbid 0x0)) at [0xdd7431000~1000]
2019-03-08 09:02:19.385622 I | osd2: -10> 2019-03-08 09:02:19.246162 7feeb44cd700 20 bluestore.blob(0x5610c86e4060) get_ref 0x21000~1000 Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,10000]) SharedBlob(0x5610c8fad790 sbid 0x0))
2019-03-08 09:02:19.385627 I | osd2: -9> 2019-03-08 09:02:19.246166 7feeb44cd700 20 bluestore.blob(0x5610c86e4060) put_ref 0x21000~1000 Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,11000]) SharedBlob(0x5610c8fad790 sbid 0x0))
2019-03-08 09:02:19.385634 I | osd2: -8> 2019-03-08 09:02:19.246170 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _do_write_small lex 0x21000~1000: 0x21000~1000 Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,10000]) SharedBlob(0x5610c8fad790 sbid 0x0))
2019-03-08 09:02:19.385639 I | osd2: -7> 2019-03-08 09:02:19.246175 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _do_alloc_write txc 0x5610c3a32100 0 blobs
2019-03-08 09:02:19.385643 I | osd2: -6> 2019-03-08 09:02:19.246177 7feeb44cd700 30 estimate gc range(hex): [21000, 22000)
2019-03-08 09:02:19.385659 I | osd2: -5> 2019-03-08 09:02:19.246178 7feeb44cd700 20 bluestore(/var/lib/rook/osd2) _wctx_finish lex_old 0x21000~1000: 0x21000~1000 Blob(0x5610c86e4060 blob([!~10000,0x29e3b0000~10000,0xdd7430000~10000] csum+has_unused crc32c/0x1000 unused=0x3df) use_tracker(0x3*0x10000 0x[0,1000,10000]) SharedBlob(0x5610c8fad790 sbid 0x0))
2019-03-08 09:02:19.385671 I | osd2: -4> 2019-03-08 09:02:19.246183 7feeb44cd700 20 bluestore.blob(0x5610c86e4060) discard_unallocated 0x0~10000
2019-03-08 09:02:19.385677 I | osd2: -3> 2019-03-08 09:02:19.246182 7feebe4e1700 20 bluestore(/var/lib/rook/osd2) _kv_sync_thread committed 3 cleaned 0 in 0.001028 (0.000006 flush + 0.001022 kv commit)
2019-03-08 09:02:19.385683 I | osd2: -2> 2019-03-08 09:02:19.246185 7feeb44cd700 20 bluestore.BufferSpace(0x5610c8fad7a8 in 0x5610c2194300) _discard 0x0~10000
2019-03-08 09:02:19.385688 I | osd2: -1> 2019-03-08 09:02:19.246194 7feebe4e1700 20 bluestore(/var/lib/rook/osd2) _kv_sync_thread committing 4 submitting 4 deferred done 0 stable 0
2019-03-08 09:02:19.385695 I | osd2: 0> 2019-03-08 09:02:19.245530 7feeb54cf700 -1 /build/ceph-12.2.4/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7feeb54cf700 time 2019-03-08 09:02:19.237489
2019-03-08 09:02:19.385700 I | osd2: /build/ceph-12.2.4/src/os/bluestore/BlueStore.cc: 2671: FAILED assert(r >= 0)
2019-03-08 09:02:19.385704 I | osd2:
2019-03-08 09:02:19.385708 I | osd2: ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
2019-03-08 09:02:19.385713 I | osd2: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5610b79c5872]
2019-03-08 09:02:19.385718 I | osd2: 2: (()+0x8f0601) [0x5610b77fe601]
2019-03-08 09:02:19.385722 I | osd2: 3: (()+0x8effa5) [0x5610b77fdfa5]
2019-03-08 09:02:19.385726 I | osd2: 4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5bd) [0x5610b782f82d]
2019-03-08 09:02:19.385731 I | osd2: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x255) [0x5610b7875235]
2019-03-08 09:02:19.385736 I | osd2: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0xfc) [0x5610b7875f9c]
2019-03-08 09:02:19.385740 I | osd2: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1b34) [0x5610b787cea4]
2019-03-08 09:02:19.385745 I | osd2: 8: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x5610b787e02e]
2019-03-08 09:02:19.385749 I | osd2: 9: (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x66) [0x5610b75a5256]
2019-03-08 09:02:19.385755 I | osd2: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, Context*, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0xcb8) [0x5610b76d5ee8]
2019-03-08 09:02:19.385760 I | osd2: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x9fa) [0x5610b75421fa]
2019-03-08 09:02:19.385764 I | osd2: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x139b) [0x5610b758a2eb]
2019-03-08 09:02:19.385769 I | osd2: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x308e) [0x5610b758dd0e]
2019-03-08 09:02:19.385774 I | osd2: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5610b7548339]
2019-03-08 09:02:19.385778 I | osd2: 15: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5610b73c13b9]
2019-03-08 09:02:19.385783 I | osd2: 16: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5610b7664047]
2019-03-08 09:02:19.385788 I | osd2: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x5610b73e99ae]
2019-03-08 09:02:19.385792 I | osd2: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5610b79ca664]
2019-03-08 09:02:19.385796 I | osd2: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5610b79cd6a0]
2019-03-08 09:02:19.385801 I | osd2: 20: (()+0x76ba) [0x7feecf01b6ba]
2019-03-08 09:02:19.385806 I | osd2: 21: (clone()+0x6d) [0x7feece09241d]
2019-03-08 09:02:19.385811 I | osd2: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-03-08 09:02:19.385815 I | osd2:
2019-03-08 09:02:19.385819 I | osd2: --- logging levels ---
2019-03-08 09:02:19.385823 I | osd2: 0/ 5 none
2019-03-08 09:02:19.385828 I | osd2: 0/ 1 lockdep
2019-03-08 09:02:19.385832 I | osd2: 0/ 1 context
2019-03-08 09:02:19.385848 I | osd2: 1/ 1 crush
2019-03-08 09:02:19.385860 I | osd2: 1/ 5 mds
2019-03-08 09:02:19.385865 I | osd2: 1/ 5 mds_balancer
2019-03-08 09:02:19.385869 I | osd2: 1/ 5 mds_locker
2019-03-08 09:02:19.385873 I | osd2: 1/ 5 mds_log
2019-03-08 09:02:19.385877 I | osd2: 1/ 5 mds_log_expire
2019-03-08 09:02:19.385881 I | osd2: 1/ 5 mds_migrator
2019-03-08 09:02:19.385886 I | osd2: 0/ 1 buffer
2019-03-08 09:02:19.385890 I | osd2: 0/ 1 timer
2019-03-08 09:02:19.385894 I | osd2: 0/ 1 filer
2019-03-08 09:02:19.385899 I | osd2: 0/ 1 striper
2019-03-08 09:02:19.385903 I | osd2: 0/ 1 objecter
2019-03-08 09:02:19.385907 I | osd2: 0/ 0 rados
2019-03-08 09:02:19.385912 I | osd2: 0/ 5 rbd
2019-03-08 09:02:19.385916 I | osd2: 0/ 5 rbd_mirror
2019-03-08 09:02:19.385921 I | osd2: 0/ 5 rbd_replay
2019-03-08 09:02:19.385925 I | osd2: 0/ 5 journaler
2019-03-08 09:02:19.385929 I | osd2: 0/ 5 objectcacher
2019-03-08 09:02:19.385933 I | osd2: 0/ 5 client
2019-03-08 09:02:19.385937 I | osd2: 0/ 0 osd
2019-03-08 09:02:19.385941 I | osd2: 0/ 5 optracker
2019-03-08 09:02:19.385946 I | osd2: 0/ 5 objclass
2019-03-08 09:02:19.385950 I | osd2: 0/ 0 filestore
2019-03-08 09:02:19.385955 I | osd2: 0/ 0 journal
2019-03-08 09:02:19.385959 I | osd2: 0/ 5 ms
2019-03-08 09:02:19.385963 I | osd2: 0/ 0 mon
2019-03-08 09:02:19.385968 I | osd2: 0/10 monc
2019-03-08 09:02:19.385971 I | osd2: 1/ 5 paxos
2019-03-08 09:02:19.385977 I | osd2: 0/ 5 tp
2019-03-08 09:02:19.385981 I | osd2: 1/ 5 auth
2019-03-08 09:02:19.385985 I | osd2: 1/ 5 crypto
2019-03-08 09:02:19.385989 I | osd2: 1/ 1 finisher
2019-03-08 09:02:19.385993 I | osd2: 1/ 1 reserver
2019-03-08 09:02:19.385998 I | osd2: 1/ 5 heartbeatmap
2019-03-08 09:02:19.386002 I | osd2: 1/ 5 perfcounter
2019-03-08 09:02:19.386006 I | osd2: 1/ 5 rgw
2019-03-08 09:02:19.386012 I | osd2: 1/10 civetweb
2019-03-08 09:02:19.386024 I | osd2: 1/ 5 javaclient
2019-03-08 09:02:19.386033 I | osd2: 1/ 5 asok
2019-03-08 09:02:19.386040 I | osd2: 1/ 1 throttle
2019-03-08 09:02:19.386049 I | osd2: 0/ 0 refs
2019-03-08 09:02:19.386056 I | osd2: 1/ 5 xio
2019-03-08 09:02:19.386062 I | osd2: 1/ 5 compressor
2019-03-08 09:02:19.386081 I | osd2: 1/30 bluestore
2019-03-08 09:02:19.386090 I | osd2: 5/ 5 bluefs
2019-03-08 09:02:19.386098 I | osd2: 1/ 3 bdev
2019-03-08 09:02:19.386118 I | osd2: 1/ 5 kstore
2019-03-08 09:02:19.386126 I | osd2: 4/ 5 rocksdb
2019-03-08 09:02:19.386134 I | osd2: 0/ 0 leveldb
2019-03-08 09:02:19.386148 I | osd2: 4/ 5 memdb
2019-03-08 09:02:19.386154 I | osd2: 1/ 5 kinetic
2019-03-08 09:02:19.386158 I | osd2: 1/ 5 fuse
2019-03-08 09:02:19.386162 I | osd2: 1/ 5 mgr
2019-03-08 09:02:19.386166 I | osd2: 1/ 5 mgrc
2019-03-08 09:02:19.386172 I | osd2: 1/ 5 dpdk
2019-03-08 09:02:19.386176 I | osd2: 1/ 5 eventtrace
2019-03-08 09:02:19.386181 I | osd2: -2/-2 (syslog threshold)
2019-03-08 09:02:19.386185 I | osd2: -1/-1 (stderr threshold)
2019-03-08 09:02:19.386190 I | osd2: max_recent 10000
2019-03-08 09:02:19.386194 I | osd2: max_new 1000
2019-03-08 09:02:19.386199 I | osd2: log_file /dev/stdout
2019-03-08 09:02:19.386204 I | osd2: --- end dump of recent events ---
2019-03-08 09:02:19.506628 I | osd2: 2019-03-08 09:02:17.265523 7feed0b45e00 -1 osd.2 4009 log_to_monitors {default=true}
2019-03-08 09:02:19.507485 I | osd2: 2019-03-08 09:02:19.237437 7feeb54cf700 -1 bluestore.extentmap(0x5610c86be7d0) operator() missing shard 0x100000 for #1:cd9f14f5:::rbd_data.10fffa643c9869.00000000000004d5:head#
2019-03-08 09:02:19.507547 I | osd2: /build/ceph-12.2.4/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7feeb54cf700 time 2019-03-08 09:02:19.237489
2019-03-08 09:02:19.507557 I | osd2: /build/ceph-12.2.4/src/os/bluestore/BlueStore.cc: 2671: FAILED assert(r >= 0)
2019-03-08 09:02:19.507563 I | osd2: ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous (stable)
2019-03-08 09:02:19.507568 I | osd2: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x5610b79c5872]
2019-03-08 09:02:19.507573 I | osd2: 2: (()+0x8f0601) [0x5610b77fe601]
2019-03-08 09:02:19.507577 I | osd2: 3: (()+0x8effa5) [0x5610b77fdfa5]
2019-03-08 09:02:19.507581 I | osd2: 4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5bd) [0x5610b782f82d]
2019-03-08 09:02:19.507586 I | osd2: 5: (BlueStore::_do_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0x255) [0x5610b7875235]
2019-03-08 09:02:19.507591 I | osd2: 6: (BlueStore::_write(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&, unsigned long, unsigned long, ceph::buffer::list&, unsigned int)+0xfc) [0x5610b7875f9c]
2019-03-08 09:02:19.507606 I | osd2: 7: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1b34) [0x5610b787cea4]
2019-03-08 09:02:19.507612 I | osd2: 8: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x52e) [0x5610b787e02e]
2019-03-08 09:02:19.507617 I | osd2: 9: (PrimaryLogPG::queue_transactions(std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<OpRequest>)+0x66) [0x5610b75a5256]
2019-03-08 09:02:19.507623 I | osd2: 10: (ReplicatedBackend::submit_transaction(hobject_t const&, object_stat_sum_t const&, eversion_t const&, std::unique_ptr<PGTransaction, std::default_delete<PGTransaction> >&&, eversion_t const&, eversion_t const&, std::vector<pg_log_entry_t, std::allocator<pg_log_entry_t> > const&, boost::optional<pg_hit_set_history_t>&, Context*, Context*, Context*, unsigned long, osd_reqid_t, boost::intrusive_ptr<OpRequest>)+0xcb8) [0x5610b76d5ee8]
2019-03-08 09:02:19.507628 I | osd2: 11: (PrimaryLogPG::issue_repop(PrimaryLogPG::RepGather*, PrimaryLogPG::OpContext*)+0x9fa) [0x5610b75421fa]
2019-03-08 09:02:19.507632 I | osd2: 12: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x139b) [0x5610b758a2eb]
2019-03-08 09:02:19.507637 I | osd2: 13: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x308e) [0x5610b758dd0e]
2019-03-08 09:02:19.507641 I | osd2: 14: (PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0xea9) [0x5610b7548339]
2019-03-08 09:02:19.507646 I | osd2: 15: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x3a9) [0x5610b73c13b9]
2019-03-08 09:02:19.507650 I | osd2: 16: (PGQueueable::RunVis::operator()(boost::intrusive_ptr<OpRequest> const&)+0x57) [0x5610b7664047]
2019-03-08 09:02:19.507654 I | osd2: 17: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x130e) [0x5610b73e99ae]
2019-03-08 09:02:19.507671 I | osd2: 18: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x884) [0x5610b79ca664]
2019-03-08 09:02:19.507676 I | osd2: 19: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x5610b79cd6a0]
2019-03-08 09:02:19.507680 I | osd2: 20: (()+0x76ba) [0x7feecf01b6ba]
2019-03-08 09:02:19.507687 I | osd2: 21: (clone()+0x6d) [0x7feece09241d]
2019-03-08 09:02:19.507691 I | osd2: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-03-08 09:02:19.507697 I | osd2: 2019-03-08 09:02:19.245530 7feeb54cf700 -1 /build/ceph-12.2.4/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7feeb54cf700 time 2019-03-08 09:02:19.237489
The cluster was working smoothly for a few months without any significant issue.
Updated by Brad Hubbard about 5 years ago
- Project changed from RADOS to bluestore
Can you make sure the underlying device is OK as a first step? This error might indicate corruption. It may be also be worth running an fsck via the ceph-bluestore-tool and reporting the results or, alternatively and if feasible, just redeploy the osd?
2019-03-08 09:02:19.237437 7feeb54cf700 -1 bluestore.extentmap(0x5610c86be7d0) operator() missing shard 0x100000 for #1:cd9f14f5:::rbd_data.10fffa643c9869.00000000000004d5:head#
Updated by Neha Ojha about 5 years ago
- Status changed from New to Need More Info
Updated by Josh Durgin almost 5 years ago
- Status changed from Need More Info to Closed
Please re-open if you can gather more information
Updated by David Zafman over 4 years ago
- Subject changed from OSD goes in crashing loop to BlueStore::ExtentMap::fault_range() assert
- Status changed from Closed to 12
Something similar happened in 12.2.12 with ceph-objectstore-tool trying to remove an object and fsck and fsck-deep. The OSD didn't crash but scrub found an extra clone which is also having a problem with the extentmap:
-1> 2019-10-01 13:25:59.308531 7f9e81f0bf80 -1 bluestore.extentmap(0x55e398f692d0) operator() missing shard 0x0 for #1:5ec95ae6:::rbd_data.81733c6b8b4567.0000000000000e8b:fbfffffffffffffe#"
remove:
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-14/ --pgid 1.37a '{"oid":"rbd_data.81733c6b8b4567.0000000000000e8b","key":"","snapid":-288230376151711746,"hash":1733989242,"max":0,"pool":1,"namespace":"","max":0}' remove
remove #1:5ec95ae6:::rbd_data.81733c6b8b4567.0000000000000e8b:fbfffffffffffffe#
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7fafb54ec580 time 2019-10-01 12:43:05.350834
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2636: FAILED assert(r >= 0)
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fafab907302]
2: (()+0x7cd81e) [0x565479f8381e]
3: (()+0x7ccf65) [0x565479f82f65]
4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x565479fc79ac]
5: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::allocator<BlueStore::SharedBlob*> >*)+0x284) [0x565479ff5ef4]
6: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc5) [0x565479ff67a5]
7: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7b) [0x565479ff840b]
8: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1f55) [0x56547a006275]
9: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x54e) [0x56547a006f8e]
10: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x1ad) [0x565479e9bfad]
11: (ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*)+0x5e) [0x565479ad50de]
12: (do_remove_object(ObjectStore*, coll_t, ghobject_t&, bool, bool, ObjectStore::Sequencer&)+0x8d1) [0x565479ab0f11]
13: (main()+0x65ed) [0x565479a238ed]
14: (__libc_start_main()+0xf1) [0x7fafa8d842e1]
15: (_start()+0x2a) [0x565479aaa55a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7fafb54ec580 thread_name:ceph-objectstor
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (()+0x94a114) [0x56547a100114]
2: (()+0x110e0) [0x7fafaa1fd0e0]
3: (gsignal()+0xcf) [0x7fafa8d96fff]
4: (abort()+0x16a) [0x7fafa8d9842a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fafab90748e]
6: (()+0x7cd81e) [0x565479f8381e]
7: (()+0x7ccf65) [0x565479f82f65]
8: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x565479fc79ac]
9: (BlueStore::_do_truncate(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>, unsigned long, std::set<BlueStore::SharedBlob*, std::less<BlueStore::SharedBlob*>, std::allocator<BlueStore::SharedBlob*> >*)+0x284) [0x565479ff5ef4]
10: (BlueStore::_do_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>)+0xc5) [0x565479ff67a5]
11: (BlueStore::_remove(BlueStore::TransContext*, boost::intrusive_ptr<BlueStore::Collection>&, boost::intrusive_ptr<BlueStore::Onode>&)+0x7b) [0x565479ff840b]
12: (BlueStore::_txc_add_transaction(BlueStore::TransContext*, ObjectStore::Transaction*)+0x1f55) [0x56547a006275]
13: (BlueStore::queue_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*)+0x54e) [0x56547a006f8e]
14: (ObjectStore::apply_transactions(ObjectStore::Sequencer*, std::vector<ObjectStore::Transaction, std::allocator<ObjectStore::Transaction> >&, Context*)+0x1ad) [0x565479e9bfad]
15: (ObjectStore::apply_transaction(ObjectStore::Sequencer*, ObjectStore::Transaction&&, Context*)+0x5e) [0x565479ad50de]
16: (do_remove_object(ObjectStore*, coll_t, ghobject_t&, bool, bool, ObjectStore::Sequencer&)+0x8d1) [0x565479ab0f11]
17: (main()+0x65ed) [0x565479a238ed]
18: (__libc_start_main()+0xf1) [0x7fafa8d842e1]
19: (_start()+0x2a) [0x565479aaa55a]
Aborted
fsck-deep
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-14/ --deep yes --command fsck
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7fc57c383f80 time 2019-10-01 13:22:00.796957
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2636: FAILED assert(r >= 0)
2019-10-01 13:22:00.796921 7fc57c383f80 -1 bluestore.extentmap(0x56162fc08a90) operator() missing shard 0x0 for #1:5ec95ae6:::rbd_data.81733c6b8b4567.0000000000000e8b:fbfffffffffffffe#
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc5729a3302]
2: (()+0x1e739e) [0x561610f4839e]
3: (()+0x1e6ae5) [0x561610f47ae5]
4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
5: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
6: (main()+0xfa1) [0x561610e8c611]
7: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
8: (_start()+0x2a) [0x561610f1042a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2019-10-01 13:22:00.798063 7fc57c383f80 -1 /mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7fc57c383f80 time 2019-10-01 13:22:00.796957
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2636: FAILED assert(r >= 0)
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc5729a3302]
2: (()+0x1e739e) [0x561610f4839e]
3: (()+0x1e6ae5) [0x561610f47ae5]
4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
5: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
6: (main()+0xfa1) [0x561610e8c611]
7: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
8: (_start()+0x2a) [0x561610f1042a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
-1> 2019-10-01 13:22:00.796921 7fc57c383f80 -1 bluestore.extentmap(0x56162fc08a90) operator() missing shard 0x0 for #1:5ec95ae6:::rbd_data.81733c6b8b4567.0000000000000e8b:fbfffffffffffffe#
0> 2019-10-01 13:22:00.798063 7fc57c383f80 -1 /mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: In function 'BlueStore::ExtentMap::fault_range(KeyValueDB*, uint32_t, uint32_t)::<lambda(const string&)>' thread 7fc57c383f80 time 2019-10-01 13:22:00.796957
/mnt/pve/store/tlamprecht/sources/ceph/ceph-12.2.12/src/os/bluestore/BlueStore.cc: 2636: FAILED assert(r >= 0)
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x102) [0x7fc5729a3302]
2: (()+0x1e739e) [0x561610f4839e]
3: (()+0x1e6ae5) [0x561610f47ae5]
4: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
5: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
6: (main()+0xfa1) [0x561610e8c611]
7: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
8: (_start()+0x2a) [0x561610f1042a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
*** Caught signal (Aborted) **
in thread 7fc57c383f80 thread_name:ceph-bluestore-
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (()+0x424ff4) [0x561611185ff4]
2: (()+0x110e0) [0x7fc5712990e0]
3: (gsignal()+0xcf) [0x7fc56fc2efff]
4: (abort()+0x16a) [0x7fc56fc3042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc5729a348e]
6: (()+0x1e739e) [0x561610f4839e]
7: (()+0x1e6ae5) [0x561610f47ae5]
8: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
9: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
10: (main()+0xfa1) [0x561610e8c611]
11: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
12: (_start()+0x2a) [0x561610f1042a]
2019-10-01 13:22:00.799287 7fc57c383f80 -1 *** Caught signal (Aborted) **
in thread 7fc57c383f80 thread_name:ceph-bluestore-
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (()+0x424ff4) [0x561611185ff4]
2: (()+0x110e0) [0x7fc5712990e0]
3: (gsignal()+0xcf) [0x7fc56fc2efff]
4: (abort()+0x16a) [0x7fc56fc3042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc5729a348e]
6: (()+0x1e739e) [0x561610f4839e]
7: (()+0x1e6ae5) [0x561610f47ae5]
8: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
9: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
10: (main()+0xfa1) [0x561610e8c611]
11: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
12: (_start()+0x2a) [0x561610f1042a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
0> 2019-10-01 13:22:00.799287 7fc57c383f80 -1 *** Caught signal (Aborted) **
in thread 7fc57c383f80 thread_name:ceph-bluestore-
ceph version 12.2.12 (39cfebf25a7011204a9876d2950e4b28aba66d11) luminous (stable)
1: (()+0x424ff4) [0x561611185ff4]
2: (()+0x110e0) [0x7fc5712990e0]
3: (gsignal()+0xcf) [0x7fc56fc2efff]
4: (abort()+0x16a) [0x7fc56fc3042a]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x28e) [0x7fc5729a348e]
6: (()+0x1e739e) [0x561610f4839e]
7: (()+0x1e6ae5) [0x561610f47ae5]
8: (BlueStore::ExtentMap::fault_range(KeyValueDB*, unsigned int, unsigned int)+0x5cc) [0x561610f8c52c]
9: (BlueStore::_fsck(bool, bool)+0x2343) [0x561610fcfee3]
10: (main()+0xfa1) [0x561610e8c611]
11: (__libc_start_main()+0xf1) [0x7fc56fc1c2e1]
12: (_start()+0x2a) [0x561610f1042a]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
Aborted
Actions