Project

General

Profile

Bug #53740

mon: all mon daemon always crash after rm pool

Added by Taizeng Wu over 1 year ago. Updated about 1 year ago.

Status:
Resolved
Priority:
High
Assignee:
Josh Durgin
Category:
Correctness/Safety
Target version:
Ceph - v16.2.7
% Done:

0%

Source:
Tags:
backport_processed
Backport:
quincy, pacific, octopus
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v16.2.7
ceph-qa-suite:
rados
Component(RADOS):
Monitor
Pull request ID:
44550
Crash signature (v1):
Crash signature (v2):

Description

We have an openstack cluster.Last week we started clearing all openstack instances and deleting all ceph pools.All mon services crashed when the second pool was deleted, and the restart problem remained.

I noticed that the MON log showed that OPENSTACK requests were still being received, so I tried to shut down all openstack controller nodes.

After this, MON started normally. I then deleted the remaining Ceph pools and started the openstack controller node, and everything returned to normal.

   -39> 2021-12-24T09:52:13.852+0000 7f55cfe05700  0 mon.osd003@1(electing) e9 handle_command mon_command({"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"} v 0) v1
   -38> 2021-12-24T09:52:13.852+0000 7f55cfe05700  0 log_channel(audit) log [DBG] : from='client.3605351 10.254.248.13:0/1538571016' entity='client.cinder' cmd=[{"prefix":"osd pool get-quota", "pool": "volumes", "format":"json"}]: dispatch
...
     0> 2021-12-24T09:52:16.348+0000 7f55cfe05700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f55cfe05700 thread_name:ms_dispatch

 ceph version 16.2.7 (dd0603118f56ab514f133c8d2e3adfc983942503) pacific (stable)
 1: /lib64/libpthread.so.0(+0x12c20) [0x7f55db482c20]
 2: (OSDMonitor::preprocess_command(boost::intrusive_ptr<MonOpRequest>)+0x4949) [0x5651d4b07b09]
 3: (OSDMonitor::preprocess_query(boost::intrusive_ptr<MonOpRequest>)+0x2f3) [0x5651d4b24213]
 4: (PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x562) [0x5651d4ad1392]
 5: (PaxosService::C_RetryMessage::_finish(int)+0x64) [0x5651d4a24534]
 6: (C_MonOp::finish(int)+0x49) [0x5651d49c18b9]
 7: (Context::complete(int)+0xd) [0x5651d49bedbd]
 8: (void finish_contexts<std::__cxx11::list<Context*, std::allocator<Context*> > >(ceph::common::CephContext*, std::__cxx11::list<Context*, std::allocator<Context*> >&, int)+0xa5) [0x5651d49eb285]
 9: (Paxos::handle_lease(boost::intrusive_ptr<MonOpRequest>)+0x604) [0x5651d4ac7674]
 10: (Paxos::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x4d7) [0x5651d4acb1d7]
 11: (Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x1324) [0x5651d49bc5f4]
 12: (Monitor::_ms_dispatch(Message*)+0x670) [0x5651d49bd080]
 13: (Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5c) [0x5651d49ebf4c]
 14: (DispatchQueue::entry()+0x126a) [0x7f55dd98aaba]
 15: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f55dda3c5d1]
 16: /lib64/libpthread.so.0(+0x817a) [0x7f55db47817a]

log View (79.4 KB) Taizeng Wu, 12/29/2021 07:54 AM

meta (1.93 KB) Taizeng Wu, 12/29/2021 07:54 AM

Related issues

Copied to RADOS - Backport #53942: pacific: mon: all mon daemon always crash after rm pool Resolved
Copied to RADOS - Backport #53943: octopus: mon: all mon daemon always crash after rm pool Resolved
Copied to RADOS - Backport #53977: quincy: mon: all mon daemon always crash after rm pool Rejected

History

#1 Updated by Neha Ojha over 1 year ago

  • Description updated (diff)

#2 Updated by Neha Ojha over 1 year ago

Do you happen to have a coredump from this crash?

#3 Updated by Neha Ojha over 1 year ago

  • Priority changed from Normal to High

#4 Updated by Taizeng Wu over 1 year ago

Neha Ojha wrote:

Do you happen to have a coredump from this crash?

No

#5 Updated by Josh Durgin over 1 year ago

  • Assignee set to Josh Durgin
  • Pull request ID set to 44550

#6 Updated by Josh Durgin over 1 year ago

  • Category set to Correctness/Safety
  • Backport set to quincy, pacific, octopus

#7 Updated by Neha Ojha over 1 year ago

  • Status changed from New to Pending Backport

#8 Updated by Backport Bot over 1 year ago

  • Copied to Backport #53942: pacific: mon: all mon daemon always crash after rm pool added

#9 Updated by Backport Bot over 1 year ago

  • Copied to Backport #53943: octopus: mon: all mon daemon always crash after rm pool added

#10 Updated by Backport Bot over 1 year ago

  • Copied to Backport #53977: quincy: mon: all mon daemon always crash after rm pool added

#11 Updated by Backport Bot about 1 year ago

  • Tags set to backport_processed

#12 Updated by Radoslaw Zarzynski about 1 year ago

  • Status changed from Pending Backport to Resolved

No need for backporting to quincy – the fix is already there (see the comment in the backport ticket). Resolving.

Also available in: Atom PDF