Project

General

Profile

Actions

Bug #64535

closed

crimson osd crashes during crimson-rados-experimental teuthology tests

Added by Xuehan Xu about 2 months ago. Updated about 13 hours ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

DEBUG 2024-02-22 08:39:40,036 [shard 0:main] seastore - 0x300003146400 trans.40333 SeaStore::list_objects: pstart #2:40000000::::0#0, pend #2:60000000::::0#0, limit 18446744073709551615
DEBUG 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: start #2:40000000::::0#0, end #2:60000000::::0#0, limit 18446744073709551615
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: get root ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200))
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_onode_root: 18446520859580825600
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: got root 0xffff34fce9260000
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Node::load_root: loading root_addr=ffff34fce9260000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xffff34fce9260000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18446520859580825600
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18446520859580825600
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18446520859580825600
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446520859580825600
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200))
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446520859580825600 got 18446520859580825600, lba_map_val_t(paddr<Seg[Dev(0),28],23965696>~8192, refcount=1, checksum=0)
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 8192B at 0xffff34fce9260000 -- CachedExtent(addr=0x300003d22780, type=ONODE_BLOCK_STAGED, version=0, dirty_from_or_retired_at=JOURNAL_SEQ_NULL, modify_time=tp(NULL), paddr=paddr<Seg[Dev(0),28],23965696>, prior_paddr=nullopt, length=8192, state=CLEAN, last_committed_crc=3163152008, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x3000098c41c0, parent=0x300009e00400, laddr=18446520859580825600, fltree_header=headerI0(is_level_tail=1, level=2))
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Node::load_root: loaded NodeI0@0xffff34fce9260000+2000Lv2$
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loading child at pos(0, 0, 0) addr=ffff30f0e9260000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xffff30f0e9260000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18446516409994706944
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18446516409994706944
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18446516409994706944
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446516409994706944
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200))
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18446516409994706944 got 18446516409994706944, lba_map_val_t(paddr<Seg[Dev(0),29],978944>~8192, refcount=1, checksum=0)
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 8192B at 0xffff30f0e9260000 -- CachedExtent(addr=0x300005c63680, type=ONODE_BLOCK_STAGED, version=3, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],1961984>), modify_time=tp(2024-02-22 08:39:08), paddr=paddr<Seg[Dev(0),29],978944>, prior_paddr=nullopt, length=8192, state=DIRTY, last_committed_crc=67784944, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x3000098c41c0, parent=0x300009e00400, laddr=18446516409994706944, fltree_header=headerI0(is_level_tail=0, level=1))
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loaded child untracked NodeI0@0xffff30f0e9260000+2000Lv1
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loading child at pos(11, 0, 0) addr=ff02e00000004000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: reading at 0xff02e00000004000 ...
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::read_extent: 18375495720229683200
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_tm - 0x300003146400 trans.40333 TransactionManager::get_pin: 18375495720229683200
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::get_mapping: 18375495720229683200
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18375495720229683200
TRACE 2024-02-22 08:39:40,036 [shard 0:main] seastore_cache - 0x300003146400 trans.40333 Cache::get_root: root already on t -- CachedExtent(addr=0x3000034f5a00, type=ROOT, version=231, dirty_from_or_retired_at=jseq(sseq(5), paddr<Seg[Dev(0),29],10010624>), modify_time=tp(2024-02-22 08:39:35), paddr=paddr<Dev(ROOT),0>, prior_paddr=nullopt, length=0, state=DIRTY, last_committed_crc=0, refcount=6, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULL, root_block(lba_root_node=0x30000910c200, backref_root_node=0x30000a555200))
TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_lba - 0x300003146400 trans.40333 BtreeLBAManager::_get_mapping: 18375495720229683200 got 18375495720229683200, lba_map_val_t(paddr<Seg[Dev(0),21],44466176>~16384, refcount=1, checksum=0)
TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::Seastore: read 16384B at 0xff02e00000004000 -- CachedExtent(addr=0x300000256300, type=ONODE_BLOCK_STAGED, version=2, dirty_from_or_retired_at=jseq(sseq(3), paddr<Seg[Dev(0),21],62865408>), modify_time=tp(2024-02-22 08:10:02), paddr=paddr<Seg[Dev(0),21],44466176>, prior_paddr=nullopt, length=16384, state=DIRTY, last_committed_crc=2728444179, refcount=3, user_hint=Hint(NULL), fully_loaded=1, rewrite_gen=GEN_NULLparent_tracker=0x300007d8a3e0, parent=0x30000a190e00, laddr=18375495720229683200, fltree_header=headerL0(is_level_tail=0, level=0))
TRACE 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 OTree::InternalNode::get_or_track_child: loaded child untracked NodeL0@0xff02e00000004000+4000Lv0
DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: found onode for #2:40000000::::head#
DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: found onode for #2:40000000:::snapmapper:0#
DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore_onode - 0x300003146400 trans.40333 FLTreeOnodeManager::list_onodes: reached the end #2:60000000::::head# > #2:60000000::::0#0
DEBUG 2024-02-22 08:39:40,037 [shard 0:main] seastore - 0x300003146400 trans.40333 SeaStore::list_objects: got 2 objects, left limit 18446744073709551613
ceph-osd: /da1/xxh/rpmbuild/BUILD/ceph-19.0.0-1586-g7df0cc68304/src/crimson/os/seastore/seastore.cc:731: crimson::os::seastore::SeaStore::Shard::list_objects(crimson::os::FuturizedStore::Shard::CollectionRef, const ghobject_t&, const ghobject_t&, uint64_t) const::<lambda(auto:208&, auto:209&)>::<lambda()>::<lambda(auto:210&)>::<lambda(auto:211)>::<lambda()> mutable::<lambda(auto:212&&)> mutable [with auto:212 = std::tuple<std::vector<ghobject_t, std::allocator<ghobject_t> >, ghobject_t>]: Assertion `limit == 0 || std::get<1>(ret) == pend || std::get<1>(ret) == ghobject_t::get_max()' failed.

The crash is caused by scrubs.

Actions #1

Updated by Xuehan Xu about 2 months ago

  • Pull request ID set to 55705
Actions #2

Updated by Matan Breizman about 13 hours ago

  • Status changed from New to Resolved
Actions

Also available in: Atom PDF