Project

General

Profile

Actions

Bug #59813

open

crash: void PaxosService::propose_pending(): assert(have_pending)

Added by Telemetry Bot 11 months ago. Updated 14 days ago.

Status:
Pending Backport
Priority:
Normal
Category:
-
Target version:
% Done:

0%

Source:
Telemetry
Tags:
backport_processed
Backport:
reef quincy
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):

2a49c9b59b3e73eb6df3eba6255b5f987b3b2a5263919468c35e65a5fe30eed6
7e097b8720bb64245dda2887c2c43a610ccd06126a37d10617a2f4adf70dd0ff
eb8737cc2f0a9176afa6b1225824d8b9b0e09d6b3a386f8979f97f6f8020f347


Description

New crash events were reported via Telemetry with newer versions (['17.2.5']) than encountered in Tracker (17.2.1).

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7a53cbc0bcdeffa2f26d71d0108e5f5ff3996898e40f82dacac9edba5a79a496

Assert condition: have_pending
Assert function: void PaxosService::propose_pending()

Sanitized backtrace:

    PaxosService::propose_pending()
    KVMonitor::do_osd_destroy(int, uuid_d&)
    OSDMonitor::prepare_command_osd_destroy(int, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)
    OSDMonitor::prepare_command_osd_purge(int, std::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)
    OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)
    OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)
    OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)
    PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)
    Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)
    Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)
    Monitor::_ms_dispatch(Message*)
    Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)
    Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)
    Monitor::_ms_dispatch(Message*)
    Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)
    DispatchQueue::entry()
    DispatchQueue::DispatchThread::entry()

Crash dump sample:
{
    "assert_condition": "have_pending",
    "assert_file": "mon/PaxosService.cc",
    "assert_func": "void PaxosService::propose_pending()",
    "assert_line": 193,
    "assert_msg": "mon/PaxosService.cc: In function 'void PaxosService::propose_pending()' thread 7fb17abb3700 time 2022-11-27T05:27:46.048417+0000\nmon/PaxosService.cc: 193: FAILED ceph_assert(have_pending)",
    "assert_thread_name": "ms_dispatch",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12cf0) [0x7fb184dc5cf0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fb186e1a499]",
        "/usr/lib64/ceph/libceph-common.so.2(+0x269605) [0x7fb186e1a605]",
        "(PaxosService::propose_pending()+0x472) [0x559456768802]",
        "(KVMonitor::do_osd_destroy(int, uuid_d&)+0x4d5) [0x559456704395]",
        "(OSDMonitor::prepare_command_osd_destroy(int, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0x27e) [0x55945678f4fe]",
        "(OSDMonitor::prepare_command_osd_purge(int, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&)+0xd7) [0x55945678fa37]",
        "(OSDMonitor::prepare_command_impl(boost::intrusive_ptr<MonOpRequest>, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&)+0x165bf) [0x5594567f212f]",
        "(OSDMonitor::prepare_command(boost::intrusive_ptr<MonOpRequest>)+0x38d) [0x5594567f8ccd]",
        "(OSDMonitor::prepare_update(boost::intrusive_ptr<MonOpRequest>)+0x17b) [0x5594567f931b]",
        "(PaxosService::dispatch(boost::intrusive_ptr<MonOpRequest>)+0x2ce) [0x559456768c0e]",
        "(Monitor::handle_command(boost::intrusive_ptr<MonOpRequest>)+0x1d39) [0x55945660d549]",
        "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x9e2) [0x5594566116d2]",
        "(Monitor::_ms_dispatch(Message*)+0x406) [0x5594566125e6]",
        "(Monitor::handle_forward(boost::intrusive_ptr<MonOpRequest>)+0xbf9) [0x559456613c39]",
        "(Monitor::dispatch_op(boost::intrusive_ptr<MonOpRequest>)+0x10c7) [0x559456611db7]",
        "(Monitor::_ms_dispatch(Message*)+0x406) [0x5594566125e6]",
        "(Dispatcher::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x5d) [0x559456642dad]",
        "(Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x478) [0x7fb187094c88]",
        "(DispatchQueue::entry()+0x50f) [0x7fb1870920cf]",
        "(DispatchQueue::DispatchThread::entry()+0x11) [0x7fb1871598f1]",
        "/lib64/libpthread.so.0(+0x81ca) [0x7fb184dbb1ca]",
        "clone()" 
    ],
    "ceph_version": "17.2.5",
    "crash_id": "2022-11-27T05:27:46.057187Z_eaac788c-5d4f-4289-9717-67a10cd31f24",
    "entity_name": "mon.7d82cd92777bc108bda33d3d314596e37a3efb71",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-mon",
    "stack_sig": "eb8737cc2f0a9176afa6b1225824d8b9b0e09d6b3a386f8979f97f6f8020f347",
    "timestamp": "2022-11-27T05:27:46.057187Z",
    "utsname_machine": "x86_64",
    "utsname_release": "6.0.9-arch1-1",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Wed, 16 Nov 2022 17:01:17 +0000" 
}


Related issues 4 (2 open2 closed)

Related to RADOS - Bug #56847: crash: void PaxosService::propose_pending(): assert(have_pending)Duplicate

Actions
Related to RADOS - Bug #56850: crash: void PaxosService::propose_pending(): assert(have_pending)ResolvedKefu Chai

Actions
Copied to RADOS - Backport #65376: quincy: crash: void PaxosService::propose_pending(): assert(have_pending)NewPatrick DonnellyActions
Copied to RADOS - Backport #65377: reef: crash: void PaxosService::propose_pending(): assert(have_pending)NewPatrick DonnellyActions
Actions #1

Updated by Telemetry Bot 11 months ago

  • Related to Bug #56847: crash: void PaxosService::propose_pending(): assert(have_pending) added
Actions #2

Updated by Telemetry Bot 11 months ago

  • Related to Bug #56850: crash: void PaxosService::propose_pending(): assert(have_pending) added
Actions #3

Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.10, v17.2.1, v17.2.5 added
Actions #4

Updated by Patrick Donnelly 9 months ago

  • Status changed from New to In Progress
  • Assignee set to Patrick Donnelly
  • Target version set to v19.0.0
  • Backport set to reef,quincy,pacific
  • Crash signature (v1) updated (diff)
Actions #5

Updated by Patrick Donnelly 9 months ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 52733
Actions #7

Updated by Konstantin Shalygin 14 days ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport changed from reef,quincy,pacific to reef quincy
Actions #8

Updated by Backport Bot 14 days ago

  • Copied to Backport #65376: quincy: crash: void PaxosService::propose_pending(): assert(have_pending) added
Actions #9

Updated by Backport Bot 14 days ago

  • Copied to Backport #65377: reef: crash: void PaxosService::propose_pending(): assert(have_pending) added
Actions #10

Updated by Backport Bot 14 days ago

  • Tags set to backport_processed
Actions

Also available in: Atom PDF