Ceph » crimson

DEBUG 2020-09-05 06:42:44,746 [shard 0] osd - handle_peering_op on 2.7 from 3
DEBUG 2020-09-05 06:42:44,746 [shard 0] osd - peering_event(id=395, detail=PeeringEvent(from=3 pgid=2.7 sent=36 requested=36 evt=epoch_sent: 36 epoch_requested: 36 MNotifyRec 2.7 from 3 notify: (query:36 sent:36 2.7( empty local-lis/les=31/32 n=0 ec=11/11 lis/c=31/31 les/c/f=32/32/0 sis=36 pruub=12.000777655s) ([31,35] all_participants=0,1,3 intervals=([31,34] acting 0,1,3))) features: 0x3f01cfb8ffbdffff +create_info)): start
DEBUG 2020-09-05 06:42:44,746 [shard 0] ms - [osd.1(cluster) v2:172.21.15.136:6803/18949@64936 >> osd.3 v2:172.21.15.72:6803/18775] <== #446 === pg_notify2(3.5 3.5 (query:36 sent:36 3.5( v 33'65 (0'0,33'65] local-lis/les=31/32 n=0 ec=14/14 lis/c=31/31 les/c/f=32/33/0 sis=36) ([31,35] all_participants=0,1,3 intervals=([31,34] acting 0,1,3))) e36/36) v1 (130)
DEBUG 2020-09-05 06:42:44,746 [shard 0] osd - handle_peering_op on 3.5 from 3
DEBUG 2020-09-05 06:42:44,746 [shard 0] osd - peering_event(id=396, detail=PeeringEvent(from=3 pgid=3.5 sent=36 requested=36 evt=epoch_sent: 36 epoch_requested: 36 MNotifyRec 3.5 from 3 notify: (query:36 sent:36 3.5( v 33'65 (0'0,33'65] local-lis/les=31/32 n=0 ec=14/14 lis/c=31/31 les/c/f=32/33/0 sis=36) ([31,35] all_participants=0,1,3 intervals=([31,34] acting 0,1,3))) features: 0x3f01cfb8ffbdffff +create_info)): start
DEBUG 2020-09-05 06:42:44,746 [shard 0] ms - [osd.1(client) v2:172.21.15.136:6802/18949@51616 >> mon.0 v2:172.21.15.136:3300/0] <== #52 === osd_map(36..36 src has 1..36) v4 (41)
INFO  2020-09-05 06:42:44,746 [shard 0] osd - handle_osd_map osd_map(36..36 src has 1..36) v4
INFO  2020-09-05 06:42:44,746 [shard 0] osd - handle_osd_map epochs [36..36], i have 35, src has [1..36]
DEBUG 2020-09-05 06:42:44,746 [shard 0] filestore - do_transaction
Segmentation fault.
...
[Backtrace #0]
void seastar::backtrace<seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}>(seastar::backtrace_buffer::append_backtrace()::{lambda(seastar::frame)#1}&&) at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/include/seastar/util/backtrace.hh:59
non-virtual thunk to std::_Nested_exception<std::runtime_error>::~_Nested_exception() at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:730
 (inlined by) print_with_backtrace at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:751
print_with_backtrace at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:758
non-virtual thunk to std::_Nested_exception<std::runtime_error>::~_Nested_exception() at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:3513
 (inlined by) operator() at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:3499
 (inlined by) _FUN at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/seastar/src/core/reactor.cc:3495
.annobin_funlockfile.c_end at funlockfile.c:?
std::_Rb_tree<unsigned long, std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io>, std::_Select1st<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> >*) at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_tree.h:1907
std::_Rb_tree<unsigned long, std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io>, std::_Select1st<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> >, std::less<unsigned long>, std::allocator<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> > >::_M_erase(std::_Rb_tree_node<std::pair<unsigned long const, BlueStore::DeferredBatch::deferred_io> >*) at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_tree.h:1913
BlueStore::_kv_finalize_thread() at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_tree.h:995
 (inlined by) ?? at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_map.h:300
 (inlined by) ?? at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/os/bluestore/BlueStore.h:1819
 (inlined by) ?? at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/os/bluestore/BlueStore.h:1819
 (inlined by) BlueStore::_kv_finalize_thread() at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/os/bluestore/BlueStore.cc:11911
BlueStore::KVFinalizeThread::entry() at /usr/src/debug/ceph-16.0.0-5167.g0f0addb4d7.el8.x86_64/src/os/bluestore/BlueStore.h:1993
start_thread at pthread_create.c:?
__personality at :?

INFO  2020-09-05 06:42:44,778 [shard 0] osd - osd.1: committed_osd_maps(36, 36)
DEBUG 2020-09-05 06:42:44,778 [shard 0] osd - pg_advance_map(id=345, detail=PGAdvanceMap(pg=2.0 from=35 to=36)): start
DEBUG 2020-09-05 06:42:44,778 [shard 0] osd -  pg_epoch 35 pg[2.0( empty local-lis/les=31/32 n=0 ec=11/11 lis/c=31/31 les/c/f=32/32/0 sis=35 pruub=12.996028369s) [0,1]/[0,3,1] r=2 lpr=35 pi=[31,35)/1 crt=0'0 mlcod 0'0 remapped NOTIFY pruub 67.631926743s@ handle_advance_map {0, 1}/{0, 1, 3} -- 0/0
DEBUG 2020-09-05 06:42:44,778 [shard 0] osd -  pg_epoch 36 pg[2.0( empty local-lis/les=31/32 n=0 ec=11/11 lis/c=31/31 les/c/f=32/32/0 sis=35 pruub=12.996028369s) [0,1]/[0,3,1] r=2 lpr=35 pi=[31,35)/1 crt=0'0 mlcod 0'0 remapped NOTIFY pruub 67.631926743s@ state<Started>: Started advmap
daemon-helper: command crashed with signal 11