Ceph » crimson

Integration branch wip-sjust-crimson-testing-2023-12-15 ac5f0a00b675c1157dbca443e10f2f8ddb2e351a

https://pulpito.ceph.com/sjust-2023-12-16_03:12:47-crimson-rados-wip-sjust-crimson-testing-2023-12-15-distro-default-smithi/7494341/

Reactor stalled for 148 ms on shard 0. Backtrace: 0x45d5d 0x2cbad828 0x2cbaebd6 0x2cbb0124 0x2cbb04a8 0x2cbb05f2 0x2cbb0a48 0x54daf 0x295ada599d81 0x295ada59a9fb 0x295ada59d78a 0x295ada59e410 0x295ad803a2d3 0x2c805477 0x295ada59e684 0x295ada58dbe9 0x295ada58dcb5 0x295ada57f165 0x295ada580fea 0xd6280 0x32402 0xbd907 0xbd194 0xbdc7a 0x1ef852c5 0x214a57bf 0x214a5b12 0x1eb52bce 0x1eb77dbb 0x1eb957b9 0x2cb4fb43 0x2cbed950 0x2ce12479 0x2ce142c3 0x2c868920 0x2c86a829 0x1eef093a 0x3feaf 0x3ff5f 0x1e998a44
kernel callstack:
Reactor stalled for 305 ms on shard 0. Backtrace: 0x45d5d 0x2cbad828 0x2cbaebd6 0x2cbb0124 0x2cbb04a8 0x2cbb05f2 0x2cbb0a48 0x54daf 0x295ada599f6f 0x295ada59aa5e 0x295ada59d78a 0x295ada59e410 0x295ad803a2d3 0x2c805477 0x295ada59e684 0x295ada58dbe9 0x295ada58dcb5 0x295ada57f165 0x295ada580fea 0xd6280 0x32402 0xbd907 0xbd194 0xbdc7a 0x1ef852c5 0x214a57bf 0x214a5b12 0x1eb52bce 0x1eb77dbb 0x1eb957b9 0x2cb4fb43 0x2cbed950 0x2ce12479 0x2ce142c3 0x2c868920 0x2c86a829 0x1eef093a 0x3feaf 0x3ff5f 0x1e998a44
kernel callstack:
#0 0x5606244e12c5 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1ef852c5)
#1 0x560626a017bf in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x214a57bf)
#2 0x560626a01b12 in ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4 (/usr/bin/ceph-osd+0x214a5b12)
#3 0x5606240aebce in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) (/usr/bin/ceph-osd+0x1eb52bce)
#4 0x5606240d3dbb in void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) (/usr/bin/ceph-osd+0x1eb77dbb)
#5 0x5606240f17b9 in seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x1eb957b9)
#6 0x5606320abb43 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2cb4fb43)
#7 0x560632149950 in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2cbed950)
#8 0x56063236e479 in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2ce12479)
#9 0x5606323702c3 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2ce142c3)
#10 0x560631dc4920 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c868920)
#11 0x560631dc6829 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c86a829)
#12 0x56062444c93a in main (/usr/bin/ceph-osd+0x1eef093a)
#13 0x7f60dd43feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
#14 0x7f60dd43ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f)
#15 0x560623ef4a44 in _start (/usr/bin/ceph-osd+0x1e998a44)

0x61500009865c is located 92 bytes inside of 472-byte region [0x615000098600,0x6150000987d8)
freed by thread T0 here:
#0 0x7f60dfab73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x560626b3fdb9 in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x215e3db9)

previously allocated by thread T0 here:
#0 0x7f60dfab6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)

SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1ef852c5) in seastar::shared_mutex::unlock()
Shadow bytes around the buggy address:
0x0c2a8000b070: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c2a8000b080: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c2a8000b090: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c2a8000b0a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x0c2a8000b0b0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c2a8000b0c0: fd fd fd fd fd fd fd fd fd fd fd[fd]fd fd fd fd
0x0c2a8000b0d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a8000b0e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
0x0c2a8000b0f0: fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa fa
0x0c2a8000b100: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c2a8000b110: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
Container overflow: fc
Array cookie: ac
Intra object redzone: bb
ASan internal: fe
Left alloca redzone: ca
Right alloca redzone: cb
Shadow gap: cc
82480ABORTING
daemon-helper: command failed with exit status 1