Project

General

Profile

Bug #57932

Intermittent ceph-mgr segfault MgrStandby::ms_dispatch2()

Added by Peter Sabaini over 1 year ago. Updated about 1 year ago.

Status:
Need More Info
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We're seeing intermittent ceph-mgr segfaults in CI

2022-10-25T11:07:07.779+0000 7f0bd37fe700  0 [progress INFO root] Processing OSDMap change 613..613
2022-10-25T11:07:07.859+0000 7f0c0b7fe700 -1 *** Caught signal (Segmentation fault) **
 in thread 7f0c0b7fe700 thread_name:ms_dispatch

 ceph version 16.2.9 (4c3647a322c0ff5a1dd2344e039859dcbd28c830) pacific (stable)
 1: /lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f0c1df81420]
 2: (std::_Rb_tree_increment(std::_Rb_tree_node_base*)+0x13) [0x7f0c1e1be1c3]
 3: (PGMap::apply_incremental(ceph::common::CephContext*, PGMap::Incremental const&)+0xae8) [0x5568996e5ab8]
 4: (ClusterState::notify_osdmap(OSDMap const&)+0x26d) [0x55689978647d]
 5: (Mgr::handle_osd_map()+0x35c) [0x5568997e32fc]
 6: (Mgr::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x4f0) [0x5568997e6640]
 7: (MgrStandby::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0xb1) [0x5568997f19c1]
 8: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0x448) [0x7f0c1e793a68]
 9: (DispatchQueue::entry()+0x58f) [0x7f0c1e79135f]
 10: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f0c1e85c011]
 11: /lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f0c1df75609]
 12: clone()

ceph-mgr.juju-9174b2-zaza-f23fe2833233-6.log.xz (108 KB) Peter Sabaini, 10/26/2022 08:59 AM

History

#1 Updated by Peter Sabaini over 1 year ago

Please let me know if I can provide any more detail. If it's helpful I can provide a crash dump

#2 Updated by Radoslaw Zarzynski about 1 year ago

  • Status changed from New to Need More Info

Hello! I went over the attached log. Unfortunate, I can't see an obvious stating point for the investigation. Could you please either:

1. provide coredump (the best and simplest) or
2. provide more verbose log (debug_mgr=20 and debug_ms=5) or
3. output of objdump -D <path to your libstdc++> where the path can be figured out by ldd.

Also available in: Atom PDF