Actions
Bug #53740
closedmon: all mon daemon always crash after rm pool
Status:
Resolved
Priority:
High
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
0%
Source:
Tags:
backport_processed
Backport:
quincy, pacific, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(RADOS):
Monitor
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We have an openstack cluster.Last week we started clearing all openstack instances and deleting all ceph pools.All mon services crashed when the second pool was deleted, and the restart problem remained.
I noticed that the MON log showed that OPENSTACK requests were still being received, so I tried to shut down all openstack controller nodes.
After this, MON started normally. I then deleted the remaining Ceph pools and started the openstack controller node, and everything returned to normal.
-39> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 mon.osd003@1(electing) e9 handle_command mon_command({"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"} v 0) v1 -38> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 log_channel(audit) log [DBG] : from='client.3605351 10.254.248.13:0/1538571016' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"}]: dispatch ... 0> 2021-12-24T09:52:16.348+0000 7f55cfe05700 -1 *** Caught signal (Segmentation fault) ** in thread 7f55cfe05700 thread_name:ms_dispatch ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f55db482c20] 2: (OSDMonitor::preprocess_command(boost::intrusive_ptr<MonOpRequest>)+0x4949) [0x5651d4b07b09] 3: (OSDMonitor::preprocess_query(boost::intrusive_ptr<MonOpRequest>)+0x2f3) [0x5651d4b24213] 4: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x562) [0x5651d4ad1392] 5: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5651d4a24534] 6: (C_MonOp::finish(int)+0x49) [0x5651d49c18b9] 7: (Context::complete(int)+0xd) [0x5651d49bedbd] 8: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa5) [0x5651d49eb285] 9: (Paxos::handle_lease(boost::intrusive_ptr<MonOpRequest>)+0x604) [0x5651d4ac7674] 10: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x4d7) [0x5651d4acb1d7] 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1324) [0x5651d49bc5f4] 12: (Monitor::_ms_dispatch(Message*)+0x670) [0x5651d49bd080] 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5651d49ebf4c] 14: (DispatchQueue::entry()+0x126a) [0x7f55dd98aaba] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55dda3c5d1] 16: /lib64/libpthread.so.0(+0x817a) [0x7f55db47817a]
Files
Actions