Ceph » crimson

On clang-17 (output below) and also gcc-12/13, address sanitizer seems to be throwing stack-use-after-return errors related to seastar::async:

INFO  2024-02-21 00:12:18,900 [shard 0:main] ms - [0x511000042800 client.?(temp_mon_client) 10.0.2.100:0/374772225@59765 >> mon.0 v2:10.0.2.100:40505/0] closing: reset no, replace no
INFO  2024-02-21 00:12:18,901 [shard 0:main] ms - [0x511000042800 client.?(temp_mon_client) 10.0.2.100:0/374772225@59765 >> mon.0 v2:10.0.2.100:40505/0] do_in_dispatch(): fault at drop, io_stat(io_state=drop, in_
seq=3, out_seq=3, out_pending_msgs_size=0, out_sent_msgs_size=0, need_ack=0, need_keepalive=0, need_keepalive_ack=0) -- read eof
=================================================================
==123962==ERROR: AddressSanitizer: stack-use-after-return on address 0x7f419f14dfa0 at pc 0x00000492b36e bp 0x7f419ef45c70 sp 0x7f419ef45c68
WRITE of size 8 at 0x7f419f14dfa0 thread T0
    #0 0x492b36d in seastar::internal::future_base::future_base() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1106:30
    #1 0x4962978 in seastar::future<void>::future<>(seastar::ready_future_marker) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1262:5
    #2 0x49628f3 in seastar::future<void> seastar::make_ready_future<void>() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1945:12
    #3 0x4d7bc65 in seastar::future<void> seastar::futurize<void>::apply<crimson::osd::populate_config_from_mon()::$_0>(crimson::osd::populate_config_from_mon()::$_0&&, std::tuple<>&&) /home/sam/git-checkouts/cep
h-workspace/main/src/seastar/include/seastar/core/future.hh:1999:20
    #4 0x4d7bb18 in seastar::futurize<std::invoke_result<crimson::osd::populate_config_from_mon()::$_0>::type>::type seastar::async<crimson::osd::populate_config_from_mon()::$_0>(seastar::thread_attributes, crimson::osd::populate_config_from_mon()::$_0&&)::'lambda'()::operator()() const /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/thread.hh:260:13
    #5 0x4d7b958 in seastar::noncopyable_function<void ()>::direct_vtable_for<seastar::futurize<std::invoke_result<crimson::osd::populate_config_from_mon()::$_0>::type>::type seastar::async<crimson::osd::populate_config_from_mon()::$_0>(seastar::thread_attributes, crimson::osd::populate_config_from_mon()::$_0&&)::'lambda'()>::call(seastar::noncopyable_function<void ()> const*) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/util/noncopyable_function.hh:129:20
    #6 0xf5ad4b2 in seastar::noncopyable_function<void ()>::operator()() const /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/util/noncopyable_function.hh:215:16
    #7 0xfc7ebf7 in seastar::thread_context::main() /home/sam/git-checkouts/ceph-workspace/main/src/seastar/src/core/thread.cc:311:9

Address 0x7f419f14dfa0 is located in stack of thread T0 at offset 32 in frame
    #0 0x49749cf in seastar::internal::log_buf::inserter_iterator::operator++(int) /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/util/log-impl.hh:89

  This frame has 1 object(s):
    [32, 40) 'retval' <== Memory access at offset 32 is inside this variable
HINT: this may be a false positive if your program uses some custom stack unwind mechanism, swapcontext or vfork
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-return /home/sam/git-checkouts/ceph-workspace/main/src/seastar/include/seastar/core/future.hh:1106:30 in seastar::internal::future_base::future_base()
Shadow bytes around the buggy address:

The actual bug appears to be that seastar is getting built without Sanitizers_FIBER_SUPPORT. seastar has a cmake module which checks and sets it (.src/seastar/cmake/FindSanitizers.cmake), but there appears to be no machinery that actually invokes it (no include() or find_package()). ceph also has a FindSanitizers.cmake with similar goals (./cmake/modules/FindSanitizers.cmake) but which doesn't include set Sanitizers_FIBER_SUPPORT. I'm preparing a fix to copy that part over.

See src/seastar/src/core/thread.cc for the SEASTAR_HAVE_ASAN_FIBER_SUPPORT usage and src/seastar/CMakeLists.txt for the Sanitizers_FIBER_SUPPORT check:

if (Sanitizers_FIBER_SUPPORT)
  list (APPEND Seastar_PRIVATE_COMPILE_DEFINITIONS SEASTAR_HAVE_ASAN_FIBER_SUPPORT)
endif ()

I'm not actually sure why this wasn't triggering on older clang and gcc versions -- it likely just wasn't on by default.