Actions
Bug #53740
closedmon: all mon daemon always crash after rm pool
Status:
Resolved
Priority:
High
Assignee:
Category:
Correctness/Safety
Target version:
% Done:
0%
Source:
Tags:
backport_processed
Backport:
quincy, pacific, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Component(RADOS):
Monitor
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
We have an openstack cluster.Last week we started clearing all openstack instances and deleting all ceph pools.All mon services crashed when the second pool was deleted, and the restart problem remained.
I noticed that the MON log showed that OPENSTACK requests were still being received, so I tried to shut down all openstack controller nodes.
After this, MON started normally. I then deleted the remaining Ceph pools and started the openstack controller node, and everything returned to normal.
-39> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 mon.osd003@1(electing) e9 handle_command mon_command({"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"} v 0) v1 -38> 2021-12-24T09:52:13.852+0000 7f55cfe05700 0 log_channel(audit) log [DBG] : from='client.3605351 10.254.248.13:0/1538571016' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"}]: dispatch ... 0> 2021-12-24T09:52:16.348+0000 7f55cfe05700 -1 *** Caught signal (Segmentation fault) ** in thread 7f55cfe05700 thread_name:ms_dispatch ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable) 1: /lib64/libpthread.so.0(+0x12c20) [0x7f55db482c20] 2: (OSDMonitor::preprocess_command(boost::intrusive_ptr<MonOpRequest>)+0x4949) [0x5651d4b07b09] 3: (OSDMonitor::preprocess_query(boost::intrusive_ptr<MonOpRequest>)+0x2f3) [0x5651d4b24213] 4: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x562) [0x5651d4ad1392] 5: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5651d4a24534] 6: (C_MonOp::finish(int)+0x49) [0x5651d49c18b9] 7: (Context::complete(int)+0xd) [0x5651d49bedbd] 8: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa5) [0x5651d49eb285] 9: (Paxos::handle_lease(boost::intrusive_ptr<MonOpRequest>)+0x604) [0x5651d4ac7674] 10: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x4d7) [0x5651d4acb1d7] 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1324) [0x5651d49bc5f4] 12: (Monitor::_ms_dispatch(Message*)+0x670) [0x5651d49bd080] 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5651d49ebf4c] 14: (DispatchQueue::entry()+0x126a) [0x7f55dd98aaba] 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55dda3c5d1] 16: /lib64/libpthread.so.0(+0x817a) [0x7f55db47817a]
Files
Updated by Neha Ojha over 2 years ago
Do you happen to have a coredump from this crash?
Updated by Taizeng Wu over 2 years ago
Neha Ojha wrote:
Do you happen to have a coredump from this crash?
No
Updated by Josh Durgin over 2 years ago
- Assignee set to Josh Durgin
- Pull request ID set to 44550
Updated by Josh Durgin over 2 years ago
- Category set to Correctness/Safety
- Backport set to quincy, pacific, octopus
Updated by Neha Ojha over 2 years ago
- Status changed from New to Pending Backport
Updated by Backport Bot over 2 years ago
- Copied to Backport #53942: pacific: mon: all mon daemon always crash after rm pool added
Updated by Backport Bot over 2 years ago
- Copied to Backport #53943: octopus: mon: all mon daemon always crash after rm pool added
Updated by Backport Bot over 2 years ago
- Copied to Backport #53977: quincy: mon: all mon daemon always crash after rm pool added
Updated by Radoslaw Zarzynski over 1 year ago
- Status changed from Pending Backport to Resolved
No need for backporting to quincy – the fix is already there (see the comment in the backport ticket). Resolving.
Actions