Actions
Bug #64535
closedcrimson osd crashes during crimson-rados-experimental teuthology tests
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Description
DEBUG 2024-02-22 08:39:40,036 [shard 0:main] seastore - 0x300003146400 trans.40333 SeaStore::list_objects: pstart #2:40000000::::0#0, pend #2:60000000::::0#0, limit 18446744073709551615 DEBUG 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: start #2:40000000::::0#0, end #2:60000000::::0#0, limit 18446744073709551615 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: get root ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200)) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_onode_root: 18446520859580825600 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: got root 0xffff34fce9260000 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Node::load_root: loading root_addr=ffff34fce9260000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xffff34fce9260000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18446520859580825600 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18446520859580825600 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18446520859580825600 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446520859580825600 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200)) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446520859580825600 got 18446520859580825600, lba_map_val_t(paddr<Seg[Dev(0),28],23965696>~8192, refcount=1, checksum=0) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 8192B at 0xffff34fce9260000 -- CachedExtent(addr=0x300003d22780, type=ONODE_BLOCK_STAGED, version=0, dirty_from_or_retired_at=JOURNAL_SEQ_NULL, modify_time=tp(NULL), paddr=paddr<Seg[Dev(0),28],23965696>, prior_paddr=nullopt, length=8192, state=CLEAN, last_committed_crc=3163152008, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x3000098c41c0, parent=0x300009e00400, laddr=18446520859580825600, fltree_header=headerI0(is_level_tail=1, level=2)) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Node::load_root: loaded NodeI0@0xffff34fce9260000+2000Lv2$ TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loading child at pos(0, 0, 0) addr=ffff30f0e9260000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xffff30f0e9260000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18446516409994706944 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18446516409994706944 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18446516409994706944 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446516409994706944 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200)) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446516409994706944 got 18446516409994706944, lba_map_val_t(paddr<Seg[Dev(0),29],978944>~8192, refcount=1, checksum=0) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 8192B at 0xffff30f0e9260000 -- CachedExtent(addr=0x300005c63680, type=ONODE_BLOCK_STAGED, version=3, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],1961984>), modify_time=tp(2024-02-22 08:39:08), paddr=paddr<Seg[Dev(0),29],978944>, prior_paddr=nullopt, length=8192, state=DIRTY, last_committed_crc=67784944, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x3000098c41c0, parent=0x300009e00400, laddr=18446516409994706944, fltree_header=headerI0(is_level_tail=0, level=1)) TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loaded child untracked NodeI0@0xffff30f0e9260000+2000Lv1 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loading child at pos(11, 0, 0) addr=ff02e00000004000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xff02e00000004000 ... TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18375495720229683200 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18375495720229683200 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18375495720229683200 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18375495720229683200 TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200)) TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18375495720229683200 got 18375495720229683200, lba_map_val_t(paddr<Seg[Dev(0),21],44466176>~16384, refcount=1, checksum=0) TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 16384B at 0xff02e00000004000 -- CachedExtent(addr=0x300000256300, type=ONODE_BLOCK_STAGED, version=2, dirty_from_or_retired_at=jseq(sseq(3), paddr<Seg[Dev(0),21],62865408>), modify_time=tp(2024-02-22 08:10:02), paddr=paddr<Seg[Dev(0),21],44466176>, prior_paddr=nullopt, length=16384, state=DIRTY, last_committed_crc=2728444179, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x300007d8a3e0, parent=0x30000a190e00, laddr=18375495720229683200, fltree_header=headerL0(is_level_tail=0, level=0)) TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loaded child untracked NodeL0@0xff02e00000004000+4000Lv0 DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: found onode for #2:40000000::::head# DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: found onode for #2:40000000:::snapmapper:0# DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: reached the end #2:60000000::::head# > #2:60000000::::0#0 DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore - 0x300003146400 trans.40333 SeaStore::list_objects: got 2 objects, left limit 18446744073709551613 ceph-osd: /da1/xxh/rpmbuild/BUILD/ceph-19.0.0-1586-g7df0cc68304/src/crimson/os/seastore/seastore.cc:731: crimson::os::seastore::SeaStore::Shard::list_objects(crimson::os::FuturizedStore::Shard::CollectionRef, const ghobject_t&, const ghobject_t&, uint64_t) const::<lambda(auto:208&, auto:209&)>::<lambda()>::<lambda(auto:210&)>::<lambda(auto:211)>::<lambda()> mutable::<lambda(auto:212&&)> mutable [with auto:212 = std::tuple<std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t>]: Assertion `limit == 0 || std::get<1>(ret) == pend || std::get<1>(ret) == ghobject_t::get_max()' failed.
The crash is caused by scrubs.
Actions