Project

General

Profile

Actions

Bug #51885

open

crash: tc_new()

Added by Telemetry Bot over 2 years ago. Updated about 2 years ago.

Status:
Triaged
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

3144fe78fcc25f1a03c1fb83213d0689afea71058dc43abaace982f319172cc0
46d1e62cc67ac874e7dbcd0dab20039566eddf95449fd01e720009c941f05c0b
4e45a4aa6e86f27a8df606d0020ee1bafe00ef7dc42fa5d2edbd8d31eb2efa6b
f43f5ef17957808f576fc7438e44ce40aaa8eeec1fa2cb92f3f1eca5e6ccb676
fb9836410eb2f2c7954620d27fa74e33b60a25b071c915afb00701a4d86b42bc


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=fb9836410eb2f2c7954620d27fa74e33b60a25b071c915afb00701a4d86b42bc

Sanitized backtrace:

    tc_new()
    ActivePyModules::notify_all(LogEntry const&)
    Mgr::handle_log(boost::intrusive_ptr<MLog>)
    Mgr::ms_dispatch2(boost::intrusive_ptr<Message> const&)
    MgrStandby::ms_dispatch2(boost::intrusive_ptr<Message> const&)
    DispatchQueue::entry()
    DispatchQueue::DispatchThread::entry()
    clone()

Crash dump sample:
{
    "archived": "2021-06-14 09:21:39.604010",
    "backtrace": [
        "(()+0x12980) [0x7f20ce5a9980]",
        "(gsignal()+0xc7) [0x7f20cd6a1fb7]",
        "(abort()+0x141) [0x7f20cd6a3921]",
        "(()+0x8c957) [0x7f20ce096957]",
        "(()+0x92ae6) [0x7f20ce09cae6]",
        "(()+0x92b21) [0x7f20ce09cb21]",
        "(()+0x92d54) [0x7f20ce09cd54]",
        "(()+0x1424b) [0x7f20ce7ca24b]",
        "(tc_new()+0x283) [0x7f20ce7eb943]",
        "(()+0x142a22) [0x55b87d491a22]",
        "(ActivePyModules::notify_all(LogEntry const&)+0x444) [0x55b87d498964]",
        "(Mgr::handle_log(boost::intrusive_ptr<MLog>)+0x53) [0x55b87d51b6a3]",
        "(Mgr::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0x187) [0x55b87d524a77]",
        "(MgrStandby::ms_dispatch2(boost::intrusive_ptr<Message> const&)+0xac) [0x55b87d53ac7c]",
        "(DispatchQueue::entry()+0x11ea) [0x7f20cf535f2a]",
        "(DispatchQueue::DispatchThread::entry()+0xd) [0x7f20cf5d624d]",
        "(()+0x76db) [0x7f20ce59e6db]",
        "(clone()+0x3f) [0x7f20cd78471f]" 
    ],
    "ceph_version": "15.2.13",
    "crash_id": "2021-06-13T23:06:43.841979Z_3f839ec2-5ab2-4ebd-911e-b1ecd5d6d3d6",
    "entity_name": "mgr.3e4e613a0ea0e7d2115664e743d9889ae2969c7a",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "18.04.5 LTS (Bionic Beaver)",
    "os_version_id": "18.04",
    "process_name": "ceph-mgr",
    "stack_sig": "3144fe78fcc25f1a03c1fb83213d0689afea71058dc43abaace982f319172cc0",
    "timestamp": "2021-06-13T23:06:43.841979Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.15.0-140-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#144-Ubuntu SMP Fri Mar 19 14:12:35 UTC 2021" 
}


Related issues 1 (0 open1 closed)

Has duplicate mgr - Bug #51917: crash: ceph::BackTrace::print(std::ostream&) constDuplicate

Actions
Actions #1

Updated by Telemetry Bot over 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v15.2.13, v15.2.3, v15.2.4, v15.2.5, v15.2.7, v15.2.8 added
Actions #2

Updated by Sage Weil over 2 years ago

  • Has duplicate Bug #51917: crash: ceph::BackTrace::print(std::ostream&) const added
Actions #3

Updated by Sage Weil over 2 years ago

  • Crash signature (v1) updated (diff)

There are a bunch of crashes in the mgr that run into tc_new() segv. Probably some heap corruption or locking issue? http://telemetry.front.sepia.ceph.com:4000/d/Nvj6XTaMk/spec-search?orgId=1&var-substr_1=tc_new()

Actions #4

Updated by Josh Durgin over 2 years ago

  • Status changed from New to Triaged
  • Priority changed from Normal to High
Actions #5

Updated by Josh Durgin over 2 years ago

All reported cases so far are from ubuntu bionic - maybe a bug with the version of tcmalloc or another library there?

Actions #6

Updated by Telemetry Bot over 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
Actions #7

Updated by Telemetry Bot about 2 years ago

  • Affected Versions v15.2.14 added
Actions #8

Updated by Telemetry Bot about 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
Actions

Also available in: Atom PDF