Ceph » crimson

DEBUG 2024-02-23 11:21:17,031 [shard 0:main] osd -  pg_epoch 731 pg[2.0( v 10'97 (0'0,10'97] local-lis/les=9/10 n=2 ec=9/9 lis/c=9/9 les/c/f=10/10/0 sis=9) [2,1] r=0 lpr=9 luod=10'98 lua=0'0 crt=10'98 mlcod 10'97 active+clean+scrubbing+deep  ScrubScan::deep_scan_object: op: scrub_scan(id=1451078, detail=(deep=1, local=1, begin=MIN, end=MAX)), obj: #2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head#, progress: obj_scrub_progress_t(offset: <null>, header_done: false, next_key: <null>, keys_done: false) scanning omap header
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore - SeaStore::get_attr: 2.0_head #2:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head#
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore_t - 0x3000030dde00 trans.70714 Cache::create_transaction: created name=get_attr, source=READ, is_weak=false
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: get root ...
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore_cache - 0x3000030dde00 trans.70714 Cache::get_root: root not on t -- CachedExtent(addr=0x300001fe4200, type=ROOT, version=648, dirty_from_or_retired_at=jseq(sseq(6), paddr<Seg[Dev(0),30],27127808>), modify_time=tp(2024-02-23 11:21:08), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=4, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000c3c7a00, backref_root_node=0x30000b693200))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::read_onode_root: 18446539312207167488
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: got root 0xffff45c53f630000
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Node::load_root: loading root_addr=ffff45c53f630000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: reading at 0xffff45c53f630000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::read_extent: 18446539312207167488
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::get_pin: 18446539312207167488
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::get_mapping: 18446539312207167488
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18446539312207167488
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_cache - 0x3000030dde00 trans.70714 Cache::get_root: root already on t -- CachedExtent(addr=0x300001fe4200, type=ROOT, version=648, dirty_from_or_retired_at=jseq(sseq(6), paddr<Seg[Dev(0),30],27127808>), modify_time=tp(2024-02-23 11:21:08), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000c3c7a00, backref_root_node=0x30000b693200))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18446539312207167488 got 18446539312207167488, lba_map_val_t(paddr<Seg[Dev(0),28],37081088>~8192, refcount=1, checksum=0)
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: read 8192B at 0xffff45c53f630000 -- CachedExtent(addr=0x30000893a300, type=ONODE_BLOCK_STAGED, version=0, dirty_from_or_retired_at=JOURNAL_SEQ_NULL, modify_time=tp(NULL), paddr=paddr<Seg[Dev(0),28],37081088>, prior_paddr=nullopt, length=8192, state=CLEAN, last_committed_crc=2507329540, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x30000a1f0ad0, parent=0x30000be04c00, laddr=18446539312207167488, fltree_header=headerI0(is_level_tail=1, level=2))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Node::load_root: loaded NodeI0@0xffff45c53f630000+2000Lv2$
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::InternalNode::get_or_track_child: loading child at pos(0, 0, 0) addr=ffff30f0e9260000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: reading at 0xffff30f0e9260000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::read_extent: 18446516409994706944
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::get_pin: 18446516409994706944
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::get_mapping: 18446516409994706944
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18446516409994706944
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_cache - 0x3000030dde00 trans.70714 Cache::get_root: root already on t -- CachedExtent(addr=0x300001fe4200, type=ROOT, version=648, dirty_from_or_retired_at=jseq(sseq(6), paddr<Seg[Dev(0),30],27127808>), modify_time=tp(2024-02-23 11:21:08), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000c3c7a00, backref_root_node=0x30000b693200))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18446516409994706944 got 18446516409994706944, lba_map_val_t(paddr<Seg[Dev(0),29],1667072>~8192, refcount=1, checksum=0)
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: read 8192B at 0xffff30f0e9260000 -- CachedExtent(addr=0x300008c8fe00, type=ONODE_BLOCK_STAGED, version=14, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],2424832>), modify_time=tp(2024-02-23 11:18:28), paddr=paddr<Seg[Dev(0),29],1667072>, prior_paddr=nullopt, length=8192, state=DIRTY, last_committed_crc=1043794758, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x30000be71080, parent=0x30000cbe2c00, laddr=18446516409994706944, fltree_header=headerI0(is_level_tail=0, level=1))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::InternalNode::get_or_track_child: loaded child untracked NodeI0@0xffff30f0e9260000+2000Lv1
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::InternalNode::get_or_track_child: loading child at pos(26, 0, 0) addr=ff03980000004000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: reading at 0xff03980000004000 ...
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::read_extent: 18375698030369193984
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::get_pin: 18375698030369193984
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::get_mapping: 18375698030369193984
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18375698030369193984
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_cache - 0x3000030dde00 trans.70714 Cache::get_root: root already on t -- CachedExtent(addr=0x300001fe4200, type=ROOT, version=648, dirty_from_or_retired_at=jseq(sseq(6), paddr<Seg[Dev(0),30],27127808>), modify_time=tp(2024-02-23 11:21:08), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000c3c7a00, backref_root_node=0x30000b693200))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18375698030369193984 got 18375698030369193984, lba_map_val_t(paddr<Seg[Dev(0),29],32833536>~16384, refcount=1, checksum=0)
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::Seastore: read 16384B at 0xff03980000004000 -- CachedExtent(addr=0x30000683e600, type=ONODE_BLOCK_STAGED, version=0, dirty_from_or_retired_at=JOURNAL_SEQ_NULL, modify_time=tp(NULL), paddr=paddr<Seg[Dev(0),29],32833536>, prior_paddr=nullopt, length=16384, state=CLEAN, last_committed_crc=1513623895, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x300002dd7970, parent=0x30000a8cac00, laddr=18375698030369193984, fltree_header=headerL0(is_level_tail=0, level=0))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_onode - 0x3000030dde00 trans.70714 OTree::InternalNode::get_or_track_child: loaded child untracked NodeL0@0xff03980000004000+4000Lv0
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore_omap - 0x3000030dde00 trans.70714 BtreeOMapManager::omap_get_value: key=omap_header
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::read_extent: 18375279179119321088~8192
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_tm - 0x3000030dde00 trans.70714 TransactionManager::get_pin: 18375279179119321088
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::get_mapping: 18375279179119321088
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18375279179119321088
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_cache - 0x3000030dde00 trans.70714 Cache::get_root: root already on t -- CachedExtent(addr=0x300001fe4200, type=ROOT, version=648, dirty_from_or_retired_at=jseq(sseq(6), paddr<Seg[Dev(0),30],27127808>), modify_time=tp(2024-02-23 11:21:08), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000c3c7a00, backref_root_node=0x30000b693200))
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_lba - 0x3000030dde00 trans.70714 BtreeLBAManager::_get_mapping: 18375279179119321088 got 18375279179119321088, lba_map_val_t(paddr<Seg[Dev(0),29],233472>~8192, refcount=1, checksum=0)
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore_omap - 0x3000030dde00 trans.70714 OMapLeafNode::get_value: key = omap_header, this: CachedExtent(addr=0x300003bd1680, type=OMAP_LEAF, version=0, dirty_from_or_retired_at=JOURNAL_SEQ_NULL, modify_time=tp(NULL), paddr=paddr<Seg[Dev(0),29],233472>, prior_paddr=nullopt, length=8192, state=CLEAN, last_committed_crc=2781592532, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x300002dd7970, parent=0x30000a8cac00, laddr=18375279179119321088, size=8, depth=1)
TRACE 2024-02-23 11:21:17,031 [shard 0:main] seastore_t - 0x3000030dde00 trans.70714 Cache::on_transaction_destruct: 
DEBUG 2024-02-23 11:21:17,031 [shard 0:main] seastore_t - 0x3000030dde00 trans.70714 Cache::on_transaction_destruct: done 9(57344B) read
Segmentation fault on shard 0.
Backtrace:
Reactor stalled for 95 ms on shard 0. Backtrace: 0x2a47bfe 0x2a02de0 0x2a03346 0x2a03527 0x12cdf 0xfa877 0x194b4dd 0x194b6f1 0x194bb98 0x194c3df 0x194a1a0 0x194a293 0x194b029 0x12cdf 0x140b699 0x140bdc9 0x29fed47 0x29ff0b0 0x2a38a8d 0x2a395ec 0x299a55b 0x299ac9b 0xff6556 0x3aca2 0x104e09d
kernel callstack: 0xffffffffffffff80 0xffffffff9600c9a7 0xffffffff958e0c59 0xffffffff958e182c 0xffffffff958e4abd 0xffffffff958e8aa2 0xffffffff958e8cea 0xffffffff956828e0 0xffffffff95682b67 0xffffffff962011ae
 0# 0x000055EC2F6F569A in ceph-osd
 1# 0x000055EC2F6F5DCA in ceph-osd
 2# 0x000055EC30CE8D48 in ceph-osd
 3# 0x000055EC30CE90B1 in ceph-osd
 4# 0x000055EC30D22A8E in ceph-osd
 5# 0x000055EC30D235ED in ceph-osd
 6# 0x000055EC30C8455C in ceph-osd
 7# 0x000055EC30C84C9C in ceph-osd
 8# main in ceph-osd
 9# __libc_start_main in /lib64/libc.so.6
10# _start in ceph-osd
Dump of siginfo:
  si_signo: 11
  si_errno: 0
  si_code: 1
  si_pid: 8
  si_uid: 0
  si_status: 0
  si_utime: 0
  si_stime: 0
  si_int: 0
  si_ptr: 0
  si_overrun: 0
  si_timerid: 8
  si_addr: 0x8
  si_band: 8
  si_fd: 0
  si_addr_lsb: 0