Ceph » crimson

WARNING: debug mode. Not for benchmarking or production
INFO  2024-02-21 00:28:51,270 seastar - Reactor backend: linux-aio
INFO  2024-02-21 00:28:51,271 seastar - Perf-based stall detector creation failed (EACCESS), try setting /proc/sys/kernel/perf_event_paranoid to 1 or less to enable kernel backtraces: falling back to posix timer.
==154147==WARNING: ASan doesn't fully support makecontext/swapcontext functions and may produce false positives in some cases!
Reactor stalled for 66 ms on shard 0. Backtrace: 0x4852f58 0xf999354 0xf99885c 0xf7cc3a5 0xf7c6bef 0xf7c6697 0xf7c6fe4 0xf7cd7fa 0x54daf 0x8143019 0x81422f8 0x813f8fa 0x7fdbccb 0x7fa1484 0x7fc4efa 0x7e0574d 0x7e0
33f9 0x969c57c 0x969ae84 0x969a99f 0x969a8ef 0x969a83b 0x969a74f 0x969a57c 0x9698fc3 0x9698963 0x96985a0 0x9697ffa 0x96978fa 0x9697517 0x969729f 0x96971ef 0x9697139 0x9697067 0x7b6f551 0x7b6f3bf 0x7b6f30f 0x7b6f2
5f 0x7b6f1af 0x7b6f105 0x7b6eedd 0x7b6ecd7 0x7b6e7de 0xf7eff78 0xf7fd1be 0xf80307d 0xf800ceb 0xf51185a 0xf50edfa 0x4c18071 0x3feaf 0x3ff5f 0x480c584
Reactor stalled for 124 ms on shard 0. Backtrace: 0x4852f58 0xf999354 0xf99885c 0xf7cc3a5 0xf7c6bef 0xf7c6697 0xf7c6fe4 0xf7cd7fa 0x54daf 0x4ba1d02 0x7e2e461 0x7e2e3b0 0x7e2e1e4 0x7e2e02d 0x7e2dedc 0x7e2b0f8 0x7e
2aae2 0x7e25fe2 0x7e25d01 0x7df9c2a 0x969c602 0x969ae84 0x969a99f 0x969a8ef 0x969a83b 0x969a74f 0x969a57c 0x9698fc3 0x9698963 0x96985a0 0x9697ffa 0x96978fa 0x9697517 0x969729f 0x96971ef 0x9697139 0x9697067 0x7b6f
551 0x7b6f3bf 0x7b6f30f 0x7b6f25f 0x7b6f1af 0x7b6f105 0x7b6eedd 0x7b6ecd7 0x7b6e7de 0xf7eff78 0xf7fd1be 0xf80307d 0xf800ceb 0xf51185a 0xf50edfa 0x4c18071 0x3feaf 0x3ff5f 0x480c584
=================================================================
==154147==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f2e82ff3620 at pc 0x000005daf254 bp 0x7ffee47cea30 sp 0x7ffee47cea28
READ of size 4 at 0x7f2e82ff3620 thread T0
    #0 0x5daf253 in auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>(unsigned int, std::uni
que_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const::'lambda'()::operator()() const /home/sam/git-checkouts/ceph-workspace/main/src/crimson/osd/shard_services.cc:837:27
    #1 0x5daec23 in seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::futurize<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>::invoke<auto crimso
n::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>(unsigned int, std::unique_ptr<MOSDMap, crimson::commo
n::UniquePtrDeleter>&) const::'lambda'()&>(std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:2034:20
    #2 0x5daebb3 in auto seastar::futurize_invoke<auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDe
leter>>(unsigned int, std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const::'lambda'()&>(std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&&) /home/sam/git-checkouts/ceph-workspace/main/s
rc/seastar/include/seastar/core/future.hh:2065:12
    #3 0x5daeb3d in auto seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::future<void>::then<auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsi
gned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>(unsigned int, std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const::'lambda'(), seastar::future<std::unique_p
tr<MOSDMap, crimson::common::UniquePtrDeleter>>>(std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&&)::'lambda'(auto&&...)::operator()<>(auto&&...) /home/sam/git-checkouts/ceph-workspace/main/src/seasta
r/include/seastar/core/future.hh:1427:24
    #4 0x5daeaab in seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>::direct_vtable_for<seastar::future<std::unique_ptr<MOSDMap, crimson::common::Uniq
uePtrDeleter>> seastar::future<void>::then<auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>
(unsigned int, std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const::'lambda'(), seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>(std::unique_ptr<MOSDMap, crimson::comm
on::UniquePtrDeleter>&&)::'lambda'(auto&&...)>::call(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()> const*) /home/sam/git-checkouts/ceph-workspace/m
ain/src/seastar/include/seastar/util/noncopyable_function.hh:129:20
    #5 0x5e84415 in seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>::operator()() const /home/sam/git-checkouts/ceph-workspace/main/src/seastar/inclu
de/seastar/util/noncopyable_function.hh:215:16
    #6 0x5e842c3 in seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> std::__invoke_impl<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>, seastar::noncopyab
le_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&>(std::__invoke_other, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePt
rDeleter>> ()>&) /opt/rh/gcc-toolset-13/root/usr/lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:61:14
    #7 0x5e84253 in std::__invoke_result<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&>::type std::__invoke<seastar::noncopyable_function<seastar:
:future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&) /opt/rh/gcc-toolset-13/r
oot/usr/lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/bits/invoke.h:96:14
    #8 0x5e841e3 in std::invoke_result<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&>::type std::invoke<seastar::noncopyable_function<seastar::fut
ure<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&) /opt/rh/gcc-toolset-13/root/
usr/lib/gcc/x86_64-redhat-linux/13/../../../../include/c++/13/functional:113:14
    #9 0x5e84167 in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::internal::monostate>(seastar::no
ncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::internal::monostate&&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/f
uture.hh:1176:16
    #10 0x5e840f0 in seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crims
on::common::UniquePtrDeleter>> ()>, seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrD
eleter>> ()>&&)::'lambda'(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()::operator()() const /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1479:28
    #11 0x5e83edd in void seastar::futurize<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>::satisfy_with_result_of<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>, seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&&)::'lambda'(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::'lambda'()>(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:2019:9
    #12 0x5e83cd7 in seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>, seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&&)::'lambda'(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&)::operator()(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&) const /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1475:17
    #13 0x5e837de in seastar::continuation<seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>, seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>, seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>>(seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&&)::'lambda'(seastar::internal::promise_base_with_type<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>&&, seastar::noncopyable_function<seastar::future<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>> ()>&, seastar::future_state<seastar::internal::monostate>&&), void>::run_and_dispose() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:748:13
    #14 0xf7eff78 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/reactor.cc:2678:14
    #15 0xf7fd1be in seastar::reactor::run_some_tasks() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/reactor.cc:3141:9
    #16 0xf80307d in seastar::reactor::do_run() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/reactor.cc:3317:9
    #17 0xf800ceb in seastar::reactor::run() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/reactor.cc:3200:16
    #18 0xf51185a in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/app-template.cc:276:31
    #19 0xf50edfa in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/app-template.cc:167:12
    #20 0x4c18071 in main /home/sam/git-checkouts/ceph-workspace/main/src/crimson/osd/main.cc:121:16
    #21 0x7f2e8537feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) (BuildId: c35bc1f89429b082ad2ec3426fe17eb1077f4627)
    #22 0x7f2e8537ff5f in __libc_start_main@GLIBC_2.2.5 (/lib64/libc.so.6+0x3ff5f) (BuildId: c35bc1f89429b082ad2ec3426fe17eb1077f4627)
    #23 0x480c584 in _start (/mnt/fast-scratch/git-checkouts/ceph-workspace/main/build/bin/crimson-osd+0x480c584) (BuildId: 6152921dacedbd3f5447dbce6d6c28c48df5e7de)

Address 0x7f2e82ff3620 is located in stack of thread T0 at offset 32 in frame
    #0 0x5dac5ff in auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>(unsigned int, std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const /home/sam/git-checkouts/ceph-workspace/main/src/crimson/osd/shard_services.cc:812

  This frame has 8 object(s):
    [32, 36) 'map_message_max.addr' <== Memory access at offset 32 is inside this variable
    [48, 64) 'maybe_handle_mapgap' (line 815)
    [80, 104) 'agg.tmp'
    [144, 152) 'agg.tmp29'
    [176, 192) 'ref.tmp' (line 822)
    [208, 256) 'ref.tmp42' (line 822)
    [288, 312) 'ref.tmp45' (line 822)
    [352, 384) 'ref.tmp51' (line 829)
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return /home/sam/git-checkouts/ceph-workspace/main/src/crimson/osd/shard_services.cc:837:27 in auto crimson::osd::OSDSingletonState::build_incremental_map_msg(unsigned int, unsigned int)::$_0::operator()<std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>>(unsigned int, std::unique_ptr<MOSDMap, crimson::common::UniquePtrDeleter>&) const::'lambda'()::operator()() const

Visible with a fix for https://tracker.ceph.com/issues/64512 on clang-17 (presumably newer gcc as well).