Bug #53740
Updated by Neha Ojha over 2 years ago
We have an openstack cluster.Last week we started clearing all openstack instances and deleting all ceph pools.All mon services crashed when the second pool was deleted, and the restart problem remained.
I noticed that the MON log showed that OPENSTACK requests were still being received, so I tried to shut down all openstack controller nodes.
After this, MON started normally. I then deleted the remaining Ceph pools and started the openstack controller node, and everything returned to normal.
<pre>
-39> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 mon.osd003@1(electing) e9 handle_command mon_command({"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"} v 0) v1
-38> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 log_channel(audit) log [DBG] : from='client.3605351 10.254.248.13:0/1538571016' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"}]: dispatch
...
0> 2021-12-24T09:52:16.348+0000 7f55cfe05700 -1 *** Caught signal (Segmentation fault) **
in thread 7f55cfe05700 thread_name:ms_dispatch
ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
1: /lib64/libpthread.so.0(+0x12c20) [0x7f55db482c20]
2: (OSDMonitor::preprocess_command(boost::intrusive_ptr<MonOpRequest>)+0x4949) [0x5651d4b07b09]
3: (OSDMonitor::preprocess_query(boost::intrusive_ptr<MonOpRequest>)+0x2f3) [0x5651d4b24213]
4: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x562) [0x5651d4ad1392]
5: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5651d4a24534]
6: (C_MonOp::finish(int)+0x49) [0x5651d49c18b9]
7: (Context::complete(int)+0xd) [0x5651d49bedbd]
8: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa5) [0x5651d49eb285]
9: (Paxos::handle_lease(boost::intrusive_ptr<MonOpRequest>)+0x604) [0x5651d4ac7674]
10: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x4d7) [0x5651d4acb1d7]
11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1324) [0x5651d49bc5f4]
12: (Monitor::_ms_dispatch(Message*)+0x670) [0x5651d49bd080]
13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5651d49ebf4c]
14: (DispatchQueue::entry()+0x126a) [0x7f55dd98aaba]
15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55dda3c5d1]
16: /lib64/libpthread.so.0(+0x817a) [0x7f55db47817a]
</pre>