Project

General

Profile

Bug #54682

crash: void ReplicatedBackend::_do_push(OpRequestRef): abort

Added by Telemetry Bot about 2 years ago. Updated 10 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):

48ff2dcd6c66adb610683ed045faff0d1206df471a99f2673ba5f9e6bd8366ff
497236321b5e7779757f22db9c1819949d53a1b045e18d7254d1f1b5e2e7500b
62729a603ac67374aaee5ded43035b336f2e2da252b8a740a48df9bc3d33e7d9
675ce34474fe88f204012d16b109f90d180a412f8d6988f875b2aeb5c5862a86
6f824a40bb034c4e8be5e65bb8d7457b366b6a5ae4ea6533c18c30f70dae96ff
7f0e35b41078574a5d0551943a3022f29ff51420b07b9e09b02121f84f492dd4
dfbca27109e24f7900d033f625ea523d806141bf9823ecf6d701e93dcfd9a948
149f510c5d1855e32e2e641bf7c9f3b3f4c961c6124e7d87b12d6a991db5001c
8a2108e6453a3609f6b09363f97e60550b25abdf396563c2faea40f3305a71ba
31cb77570dd3b7353e23986a9e543cc4411a902409c02024b10c7b091f86cd8a
853ed090ee848c23a85468282f6824b3e418b2466eb4e441b9b538bf09e05b60
d2a66858e5f592adf1bc979462d4bb38b85c6b83fa500a834c14634151f6c806
f30d1c727d02e0a9ae9a1d849b74adac032ff7a3f537511dc7d7f37b51b116ff


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=c0b08204690d3af94bdee5cf298fbf4a46bcda39dd6725b844e409c400630c03

Assert condition: abort
Assert function: void ReplicatedBackend::_do_push(OpRequestRef)

Sanitized backtrace:

    ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)
    ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)
    PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)
    PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)
    OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)
    ceph::osd::scheduler::PGRecoveryMsg::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)
    OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)
    ShardedThreadPool::shardedthreadpool_worker(unsigned int)
    ShardedThreadPool::WorkThreadSharded::entry()

Crash dump sample:
{
    "assert_condition": "abort",
    "assert_file": "osd/ReplicatedBackend.cc",
    "assert_func": "void ReplicatedBackend::_do_push(OpRequestRef)",
    "assert_line": 789,
    "assert_msg": "osd/ReplicatedBackend.cc: In function 'void ReplicatedBackend::_do_push(OpRequestRef)' thread 7fba99303700 time 2022-03-12T11:34:39.034312+0000\nosd/ReplicatedBackend.cc: 789: ceph_abort_msg(\"abort() called\")",
    "assert_thread_name": "tp_osd_tp",
    "backtrace": [
        "/lib64/libpthread.so.0(+0x12ce0) [0x7fbabed2ece0]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b6) [0x5612e9f6b38f]",
        "(ReplicatedBackend::_do_push(boost::intrusive_ptr<OpRequest>)+0x198) [0x5612ea411ce8]",
        "(ReplicatedBackend::_handle_message(boost::intrusive_ptr<OpRequest>)+0x2a8) [0x5612ea413e88]",
        "(PGBackend::handle_message(boost::intrusive_ptr<OpRequest>)+0x52) [0x5612ea2467d2]",
        "(PrimaryLogPG::do_request(boost::intrusive_ptr<OpRequest>&, ThreadPool::TPHandle&)+0x5de) [0x5612ea1e995e]",
        "(OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x309) [0x5612ea0728e9]",
        "(ceph::osd::scheduler::PGRecoveryMsg::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x68) [0x5612ea2d0248]",
        "(OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xc28) [0x5612ea092d48]",
        "(ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5c4) [0x5612ea7045b4]",
        "(ShardedThreadPool::WorkThreadSharded::entry()+0x14) [0x5612ea707254]",
        "/lib64/libpthread.so.0(+0x81cf) [0x7fbabed241cf]",
        "clone()" 
    ],
    "ceph_version": "16.2.7",
    "crash_id": "2022-03-12T11:34:39.051075Z_373e1878-887f-40ba-967d-7c693cdcf4f9",
    "entity_name": "osd.e7c605252270a1fc806dfdd05928e06416f06f41",
    "os_id": "centos",
    "os_name": "CentOS Stream",
    "os_version": "8",
    "os_version_id": "8",
    "process_name": "ceph-osd",
    "stack_sig": "7f0e35b41078574a5d0551943a3022f29ff51420b07b9e09b02121f84f492dd4",
    "timestamp": "2022-03-12T11:34:39.051075Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.13.19-5-pve",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PVE 5.13.19-13 (Tue, 08 Mar 2022 07:32:25 +0100)" 
}

History

#1 Updated by Telemetry Bot about 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.1, v16.2.4, v16.2.5, v16.2.6, v16.2.7 added

#2 Updated by Telemetry Bot over 1 year ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v17.2.0, v17.2.1 added

#3 Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v16.2.9, v17.2.4, v17.2.5 added

#4 Updated by changzhi tan 10 months ago

I encountered the same problem in v15.2.8,

-7> 2023-05-19T15:16:16.593+0800 7f0c1beed700 -1 /SDS-CICD/rpmbuild/BUILD/ceph-15.2.8.1.0.18/src/osd/ReplicatedBackend.cc: In function 'void ReplicatedBackend::_do_push(OpRequestRef)' thread 7f0c1beed700 time 2023-05-19T15:16:16.588009+0800
/SDS-CICD/rpmbuild/BUILD/ceph-15.2.8.1.0.18/src/osd/ReplicatedBackend.cc: 844: ceph_abort_msg("abort() called")
ceph version v15.2.8.1.0.18 () octopus (stable)
1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xdf) [0x564fc3317d12]
2: (ReplicatedBackend::_do_push(boost::intrusive_ptr&lt;OpRequest&gt;)+0x19f) [0x564fc373f19f]
3: (ReplicatedBackend::_handle_message(boost::intrusive_ptr&lt;OpRequest&gt;)+0x298) [0x564fc3748198]
4: (PGBackend::handle_message(boost::intrusive_ptr&lt;OpRequest&gt;)+0x4a) [0x564fc35d789a]
5: (PrimaryLogPG::do_request(boost::intrusive_ptr&lt;OpRequest&gt;&, ThreadPool::TPHandle&)+0x5cb) [0x564fc357730b]
6: (OSD::dequeue_op(boost::intrusive_ptr&lt;PG&gt;, boost::intrusive_ptr&lt;OpRequest&gt;, ThreadPool::TPHandle&)+0x2f9) [0x564fc34038e9]
7: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr&lt;PG&gt;&, ThreadPool::TPHandle&)+0x69) [0x564fc365b0e9]
8: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0x1449) [0x564fc341f489]
9: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x5b6) [0x564fc3a53766]
10: (ShardedThreadPool::WorkThreadSharded::entry()+0x10) [0x564fc3a562b0]
11: (()+0x7ea5) [0x7f0c486e2ea5]
12: (clone()+0x6d) [0x7f0c475a5b0d]

Also available in: Atom PDF