Project

General

Profile

Bug #51639

crimson/store_nbd: crash after start

Added by Xuehan Xu over 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

crimson-store-nbd crashes right after startup completed

DEBUG 2021-07-13 12:13:06,034 [shard 0] seastore - ~btree_range_pin_t: removing btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x6e4bf80)
DEBUG 2021-07-13 12:13:06,034 [shard 0] seastore - remove_pin: btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x6e4bf80)
DEBUG 2021-07-13 12:13:06,034 [shard 0] seastore - SegmentCleaner::log_gc_state(GCProcess::maybe_wait_should_run): total 107374182400, available 107307069440, unavailable 67112960, used 4096, reclaimable 0, reclaim_ratio 0.0, available_ratio 0.9993749618530273, should_block_on_gc false, gc_should_reclaim_space false, journal_head journal_seq_t(segment_seq=1, offset=paddr_t<1, 4096>), journal_tail_target journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), dirty_tail journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), dirty_tail_limit journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), gc_should_trim_journal false, 
DEBUG 2021-07-13 12:13:06,034 [shard 0] test - Running nbd server...
DEBUG 2021-07-13 12:13:06,034 [shard 0] test - About to listen on /tmp/store_nbd_socket.sock
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - TransactionManager::close: enter
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - SegmentCleaner::log_gc_state(GCProcess::maybe_wait_should_run): total 107374182400, available 107307069440, unavailable 67112960, used 4096, reclaimable 0, reclaim_ratio 0.0, available_ratio 0.9993749618530273, should_block_on_gc false, gc_should_reclaim_space false, journal_head journal_seq_t(segment_seq=1, offset=paddr_t<1, 4096>), journal_tail_target journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), dirty_tail journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), dirty_tail_limit journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), gc_should_trim_journal false, 
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - SegmentCleaner::log_gc_state(GCProcess::run): total 107374182400, available 107307069440, unavailable 67112960, used 4096, reclaimable 0, reclaim_ratio 0.0, available_ratio 0.9993749618530273, should_block_on_gc false, gc_should_reclaim_space false, journal_head journal_seq_t(segment_seq=1, offset=paddr_t<1, 4096>), journal_tail_target journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), dirty_tail journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), dirty_tail_limit journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), gc_should_trim_journal false, 
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - Cache::dump_contents: enter
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - Cache::dump_contents: exit
DEBUG 2021-07-13 12:13:06,035 [shard 0] seastore - block: do_write offset 4096 len 4096
ERROR 2021-07-13 12:13:06,035 [shard 0] seastore - do_write: dma_write got error std::system_error (error system:9, Bad file descriptor)
ERROR 2021-07-13 12:13:06,035 [shard 0] none - ../src/crimson/common/errorator.h:1183 : In function 'void crimson::ct_error::assert_all::operator()(ErrorT&&) [with ErrorT = const crimson::unthrowable_wrapper<const std::error_code&, ((const std::error_code&)(& crimson::ec<std::errc::io_error>))>&]', abort(%s)
abort() called
Aborting on shard 0.
Backtrace:
  0x4780c83
  0x4778d9a
  0x47272da
  0x472739b
  0x473e2d3
  0x474cd78
  0x474cdd4
  /lib64/libpthread.so.0+0x12dcf
  /lib64/libc.so.6+0x3770e
  /lib64/libc.so.6+0x21b24
  0x44d8bad
  0x3115c1b
  0x310c3c3
  0x31025bb
  0x30f62f5
  0x30e97f7
  0x30df50f
  0x30df2c3
  0x3102608
  0x30f6350
  0x310c46f
  0x308eea9
  0x3087da4
  0x308eef9
  0x3087cda
  0x30b8102
  0x4737b0a
  0x47388ad
  0x47398e5
  0x46e2d4b
  0x46e246a
  0x30485db
  /lib64/libc.so.6+0x236a2
  0x304762d
Aborted (core dumped)

With ASAN:

DEBUG 2021-07-13 12:32:59,193 [shard 0] seastore - ~btree_range_pin_t: removing btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x6130000088c0)
DEBUG 2021-07-13 12:32:59,193 [shard 0] seastore - remove_pin: btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x6130000088c0)
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - Cache::get_root(0x612000017bc0): root already on transaction CachedExtent(addr=0x61a000003080, type=ROOT, version=0, dirty_from_or_retired_at=journal_seq_t(segment_seq=0, offset=paddr_t<0, 4096>), paddr=paddr_t<NULL_SEG, NULL_OFF>, state=DIRTY, last_committed_crc=0, refcount=3)
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - BtreeLBAManager::get_root: reading root at paddr_t<0, 8192> depth 1
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - get_lba_btree_extent: reading leaf at offset paddr_t<0, 8192>, depth 1
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - Cache::add_extent: extent CachedExtent(addr=0x613000008a80, type=LADDR_LEAF, version=0, dirty_from_or_retired_at=journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), paddr=paddr_t<0, 8192>, state=CLEAN, last_committed_crc=0, refcount=2, size=0, meta=btree_node_meta_t(begin=0, end=0, depth=0))
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - block: do_read offset 16384 len 4096
DEBUG 2021-07-13 12:32:59,194 [shard 0] seastore - get_lba_btree_extent: read leaf at offset paddr_t<0, 8192> CachedExtent(addr=0x613000008a80, type=LADDR_LEAF, version=0, dirty_from_or_retired_at=journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), paddr=paddr_t<0, 8192>, state=CLEAN, last_committed_crc=345887052, refcount=1, size=0, meta=btree_node_meta_t(begin=0, end=18446744073709551615, depth=1)), parent CachedExtent(addr=0x61a000003080, type=ROOT, version=0, dirty_from_or_retired_at=journal_seq_t(segment_seq=0, offset=paddr_t<0, 4096>), paddr=paddr_t<NULL_SEG, NULL_OFF>, state=DIRTY, last_committed_crc=0, refcount=4)
DEBUG 2021-07-13 12:32:59,195 [shard 0] seastore - ~btree_range_pin_t: removing btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x613000008a80)
DEBUG 2021-07-13 12:32:59,195 [shard 0] seastore - remove_pin: btree_range_pin_t(begin=0, end=18446744073709551615, depth=1, extent=0x613000008a80)
DEBUG 2021-07-13 12:32:59,195 [shard 0] seastore - SegmentCleaner::log_gc_state(GCProcess::maybe_wait_should_run): total 107374182400, available 107307069440, unavailable 67112960, used 4096, reclaimable 0, reclaim_ratio 0.0, available_ratio 0.9993749618530273, should_block_on_gc false, gc_should_reclaim_space false, journal_head journal_seq_t(segment_seq=1, offset=paddr_t<1, 4096>), journal_tail_target journal_seq_t(segment_seq=0, offset=paddr_t<NULL_SEG, NULL_OFF>), dirty_tail journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), dirty_tail_limit journal_seq_t(segment_seq=0, offset=paddr_t<1, 4096>), gc_should_trim_journal false, 
DEBUG 2021-07-13 12:32:59,195 [shard 0] test - Running nbd server...
DEBUG 2021-07-13 12:32:59,195 [shard 0] test - About to listen on /tmp/store_nbd_socket.sock
=================================================================
==225314==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x2b1f7afa0f08 at pc 0x000007d2b3a8 bp 0x2b1f7afa0eb0 sp 0x2b1f7afa0ea0
READ of size 8 at 0x2b1f7afa0f08 thread T0
Reactor stalled for 261 ms on shard 0. Backtrace: 0x45bad 0x7f9d29a 0x7f77363 0x7e2b4ae 0x7e424dd 0x7e3eeb6 0x7e3f076 0x7e41eaa 0x12dcf 0x2b1f684f8012 0x2b1f684f8253 0x2b1f684e914f 0x2b1f684f7930 0x2b1f684f7da7 0x2b1f684e9471 0x2b1f684e9585 0x2b1f684da5d0 0x2b1f684dc3ba 0x2b1f684d5bcf 0x325b3 0xb97fb 0xb90a9 0xb9eba 0x7d2b3a7 0x7d2a4f9 0x13184 0xf722 0x108ff 0x1133a 0x304b4ca 0x30405c8 0x306133d 0x305e5ec 0x3057cb7 0x3057d2d 0x3057e5b 0x305131e 0x3065b8c 0x7d94049 0x81fd310
    #0 0x7d2b3a7 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&) /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/stl_iterator.h:1091
    #1 0x7d2a4f9 in dl_iterate_phdr ../src/seastar/src/core/exception_hacks.cc:121
    #2 0x2b1f6d182184 in _Unwind_Find_FDE (/lib64/libgcc_s.so.1+0x13184)
    #3 0x2b1f6d17e722  (/lib64/libgcc_s.so.1+0xf722)
    #4 0x2b1f6d17f8ff  (/lib64/libgcc_s.so.1+0x108ff)
    #5 0x2b1f6d18033a in _Unwind_Resume (/lib64/libgcc_s.so.1+0x1133a)
    #6 0x304b4ca in NBDHandler::run() ../src/crimson/tools/store_nbd/store-nbd.cc:404
    #7 0x30405c8 in operator() ../src/crimson/tools/store_nbd/store-nbd.cc:309
    #8 0x306133d in __invoke_impl<int, main(int, char**)::<lambda()>::<lambda()> > /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:60
    #9 0x305e5ec in __invoke<main(int, char**)::<lambda()>::<lambda()> > /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:95
    #10 0x3057cb7 in __apply_impl<main(int, char**)::<lambda()>::<lambda()>, std::tuple<> > /opt/rh/gcc-toolset-10/root/usr/include/c++/10/tuple:1723
    #11 0x3057d2d in apply<main(int, char**)::<lambda()>::<lambda()>, std::tuple<> > /opt/rh/gcc-toolset-10/root/usr/include/c++/10/tuple:1734
    #12 0x3057e5b in apply<main(int, char**)::<lambda()>::<lambda()> > ../src/seastar/include/seastar/core/future.hh:2104
    #13 0x305131e in operator() ../src/seastar/include/seastar/core/thread.hh:258
    #14 0x3065b8c in call ../src/seastar/include/seastar/util/noncopyable_function.hh:124
    #15 0x7d94049 in seastar::noncopyable_function<void ()>::operator()() const ../src/seastar/include/seastar/util/noncopyable_function.hh:209
    #16 0x81fd310 in seastar::thread_context::main() ../src/seastar/src/core/thread.cc:299

0x2b1f7afa0f08 is located 257800 bytes inside of 262144-byte region [0x2b1f7af62000,0x2b1f7afa2000)
allocated by thread T0 here:
    #0 0x2b1f684b5177 in aligned_alloc (/lib64/libasan.so.6+0xb1177)
    #1 0x81fbd1b in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196
    #2 0x81fab9a in seastar::thread_context::thread_context(seastar::thread_attributes, seastar::noncopyable_function<void ()>) ../src/seastar/src/core/thread.cc:173
    #3 0x305e8b6 in make_unique<seastar::thread_context, seastar::thread_attributes, seastar::async<main(int, char**)::<lambda()>::<lambda()>, {}>::<lambda()> > /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/unique_ptr.h:962
    #4 0x305820f in thread<seastar::async<main(int, char**)::<lambda()>::<lambda()>, {}>::<lambda()> > ../src/seastar/include/seastar/core/thread.hh:201
    #5 0x30521e9 in async<main(int, char**)::<lambda()>::<lambda()> > ../src/seastar/include/seastar/core/thread.hh:257
    #6 0x304c62c in async<main(int, char**)::<lambda()>::<lambda()> > ../src/seastar/include/seastar/core/thread.hh:281
    #7 0x3040fdb in operator() ../src/crimson/tools/store_nbd/store-nbd.cc:315
    #8 0x305f258 in __invoke_impl<seastar::future<int>, main(int, char**)::<lambda()>&> /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:60
    #9 0x305947c in __invoke_r<seastar::future<int>, main(int, char**)::<lambda()>&> /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:115
    #10 0x3052f45 in _M_invoke /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/std_function.h:292
    #11 0x7d1b258 in std::function<seastar::future<int> ()>::operator()() const /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/std_function.h:622
    #12 0x7d172ec in seastar::future<int> seastar::futurize<seastar::future<int> >::invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) ../src/seastar/include/seastar/core/future.hh:2135
    #13 0x7d14d04 in auto seastar::futurize_invoke<std::function<seastar::future<int> ()>&>(std::function<seastar::future<int> ()>&) ../src/seastar/include/seastar/core/future.hh:2166
    #14 0x7d07fbf in operator() ../src/seastar/src/core/app-template.cc:129
    #15 0x7d11bbd in __invoke_impl<void, seastar::app_template::run(int, char**, std::function<seastar::future<int>()>&&)::<lambda()>&> /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:60
    #16 0x7d10160 in __invoke_r<void, seastar::app_template::run(int, char**, std::function<seastar::future<int>()>&&)::<lambda()>&> /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:110
    #17 0x7d0ddd1 in _M_invoke /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/std_function.h:291
    #18 0x42adae5 in std::function<void ()>::operator()() const /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/std_function.h:622
    #19 0x7d20dfb in seastar::future<void> seastar::futurize<void>::invoke<std::function<void ()>&>(std::function<void ()>&) ../src/seastar/include/seastar/core/future.hh:2132
    #20 0x7d1d7d4 in auto seastar::futurize_invoke<std::function<void ()>&>(std::function<void ()>&) ../src/seastar/include/seastar/core/future.hh:2166
    #21 0x7d1d70c in _ZZN7seastar6futureIvE4thenISt8functionIFvvEES1_EET0_OT_ENUlDpOT_E_clIJEEEDaSB_ ../src/seastar/include/seastar/core/future.hh:1527
    #22 0x7d21101 in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenISt8functionIFvvEES2_EET0_OT_EUlDpOT_E_E4callEPKS4_ ../src/seastar/include/seastar/util/noncopyable_function.hh:124
    #23 0x31a4d88 in seastar::noncopyable_function<seastar::future<void> ()>::operator()() const ../src/seastar/include/seastar/util/noncopyable_function.hh:209
    #24 0x318e230 in seastar::future<void> std::__invoke_impl<seastar::future<void>, seastar::noncopyable_function<seastar::future<void> ()>&>(std::__invoke_other, seastar::noncopyable_function<seastar::future<void> ()>&) /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:60
    #25 0x3170969 in std::__invoke_result<seastar::noncopyable_function<seastar::future<void> ()>&>::type std::__invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/invoke.h:96
    #26 0x3151bea in std::invoke_result<seastar::noncopyable_function<seastar::future<void> ()>&>::type std::invoke<seastar::noncopyable_function<seastar::future<void> ()>&>(seastar::noncopyable_function<seastar::future<void> ()>&) /opt/rh/gcc-toolset-10/root/usr/include/c++/10/functional:89
    #27 0x31325a9 in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) ../src/seastar/include/seastar/core/future.hh:1209
    #28 0x31324da in seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}::operator()() const ../src/seastar/include/seastar/core/future.hh:1582
    #29 0x3151cde in void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) (/da1/xxh/ceph/build/bin/crimson-store-nbd+0x3151cde)

SUMMARY: AddressSanitizer: stack-buffer-overflow /opt/rh/gcc-toolset-10/root/usr/include/c++/10/bits/stl_iterator.h:1091 in bool __gnu_cxx::operator!=<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > >(__gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&, __gnu_cxx::__normal_iterator<dl_phdr_info*, std::vector<dl_phdr_info, std::allocator<dl_phdr_info> > > const&)
Shadow bytes around the buggy address:
  0x05646f5ec190: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x05646f5ec1a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x05646f5ec1b0: 00 00 f1 f1 f1 f1 04 f2 04 f2 00 f2 f2 f2 00 f2
  0x05646f5ec1c0: f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 00 00 00 00 00
  0x05646f5ec1d0: 00 00 00 f2 f2 f2 00 f2 f2 f2 00 f2 f2 f2 00 00
=>0x05646f5ec1e0: 04[f2]f2 f2 f2 f2 00 00 00 f2 f2 f2 f2 f2 00 00
  0x05646f5ec1f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f2
  0x05646f5ec200: f2 f2 f2 f2 f2 f2 f2 f2 00 00 00 00 00 00 00 00
  0x05646f5ec210: 00 00 00 00 00 00 00 00 00 f3 f3 f3 f3 f3 f3 f3
  0x05646f5ec220: f3 f3 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x05646f5ec230: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
  Shadow gap:              cc
==225314==ABORTING

Also available in: Atom PDF