Ceph » crimson

In `rados_api_tests`, we run `workunits/rados/test.sh`.
Not consistently, one of the osds crash with the following backtrace:

DEBUG 2023-03-26 09:08:44,237 [shard 0] osd - pg_advance_map(id=729864, detail=PGAdvanceMap(pg=31.1a from=1101 to=1102)): complete
DEBUG 2023-03-26 09:08:44,237 [shard 0] osd - pg_advance_map(id=730380, detail=PGAdvanceMap(pg=73.2 from=1101 to=1102)): complete
terminate called after throwing an instance of 'std::runtime_error'
  what():  read gave enoent on #-1:16a18850:::osdmap.1103:0#
Aborting on shard 0.
Backtrace:
Reactor stalled for 38 ms on shard 0.
 0# gsignal in /lib64/libc.so.6
 1# abort in /lib64/libc.so.6
 2# 0x00007F034A4AA09B in /lib64/libstdc++.so.6
 3# 0x00007F034A4B053C in /lib64/libstdc++.so.6
 4# 0x00007F034A4AF559 in /lib64/libstdc++.so.6
 5# __gxx_personality_v0 in /lib64/libstdc++.so.6
 6# 0x00007F034916EB03 in /lib64/libgcc_s.so.1
 7# _Unwind_Resume in /lib64/libgcc_s.so.1
 8# 0x0000560B05296C3C in ceph-osd
 9# 0x0000560B05298DA9 in ceph-osd
10# void seastar::futurize<seastar::future<ceph::buffer::v15_2_0::list> >::satisfy_with_result_of<seastar::future<ceph::buffer::v15_2_0::list>::then_wrapped_nrvo<seastar::future<ceph::buffer::v15_2_0::list>, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)> >(seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&, seastar::future_state<ceph::buffer::v15_2_0::list>&&)#1}::operator()(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&, seastar::future_state<ceph::buffer::v15_2_0::list>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::future<ceph::buffer::v15_2_0::list>::then_wrapped_nrvo<seastar::future<ceph::buffer::v15_2_0::list>, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)> >(seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&, seastar::future_state<ceph::buffer::v15_2_0::list>&&)#1}::operator()(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&, seastar::future_state<ceph::buffer::v15_2_0::list>&&) const::{lambda()#1}&&) in ceph-osd
11# seastar::continuation<seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>, seastar::future<ceph::buffer::v15_2_0::list>::then_wrapped_nrvo<seastar::future<ceph::buffer::v15_2_0::list>, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)> >(seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&&)::{lambda(seastar::internal::promise_base_with_type<ceph::buffer::v15_2_0::list>&&, seastar::noncopyable_function<seastar::future<ceph::buffer::v15_2_0::list> (seastar::future<ceph::buffer::v15_2_0::list>&&)>&, seastar::future_state<ceph::buffer::v15_2_0::list>&&)#1}, ceph::buffer::v15_2_0::list>::run_and_dispose() in ceph-osd
12# 0x0000560B1315EDDF in ceph-osd

First instance of this issue (AFAIK) is commented here [1], on main branch with head sha1 of fa8a9c73ae3a5cc905789e96e76c4f9d3a0b0573.
This crash continues to appear every now and then, see osd.1.log in [2], or osd.3.log in [3].

[1] https://github.com/ceph/ceph/pull/49286#discussion_r1095950772
[2] https://pulpito.ceph.com/matan-2023-03-26_08:06:20-crimson-rados-wip-matanb-crimson-only-testing-no_trim-23.3-v2-distro-crimson-smithi/7220607/
[3] https://pulpito.ceph.com/matan-2023-03-26_07:59:51-crimson-rados-wip-matanb-crimson-only-testing-no_trim-23.3-v2-distro-crimson-smithi/7220586/