Project

General

Profile

Bug #56274

crash: pthread_mutex_lock()

Added by Telemetry Bot 5 months ago. Updated 4 months ago.

Status:
Pending Backport
Priority:
High
Target version:
% Done:

0%

Source:
Telemetry
Tags:
backport_processed
Backport:
quincy,pacific
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

1301087589e4b5967aa0dce3c10c5ea2a5d576eb6900d8325c24e9aa1b5b114f


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=ae29a2dc9a72f6b7bc2a83d735ed13625bef428796efc5a3ed5524fd8248ddf9

Sanitized backtrace:

    pthread_mutex_lock()
    cephsqlite_setcct()
    Mgr::init()
    Context::complete(int)
    Finisher::finisher_thread_entry()

Crash dump sample:
{
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f7ca7f4b420]",
        "pthread_mutex_lock()",
        "(ceph::logging::Log::submit_entry(ceph::logging::Entry&&)+0x45) [0x7f7ca87c8975]",
        "cephsqlite_setcct()",
        "(Mgr::init()+0x19c5) [0x557fcdda6a85]",
        "/usr/bin/ceph-mgr(+0x2789b1) [0x557fcdda79b1]",
        "(Context::complete(int)+0xd) [0x557fcdcf244d]",
        "(Finisher::finisher_thread_entry()+0x19d) [0x7f7ca847deed]",
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f7ca7f3f609]",
        "clone()" 
    ],
    "ceph_version": "17.2.0",
    "crash_id": "2022-06-13T16:27:27.048191Z_19ce4dc9-111f-4a5b-97ac-7986716e4ab5",
    "entity_name": "mgr.82df13b301fa0a76d46f6a23c7761010ca88e6b9",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "20.04.4 LTS (Focal Fossa)",
    "os_version_id": "20.04",
    "process_name": "ceph-mgr",
    "stack_sig": "1301087589e4b5967aa0dce3c10c5ea2a5d576eb6900d8325c24e9aa1b5b114f",
    "timestamp": "2022-06-13T16:27:27.048191Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.13.19-6-pve",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PVE 5.13.19-15 (Tue, 29 Mar 2022 15:59:50 +0200)" 
}


Related issues

Duplicated by CephFS - Bug #56659: mgr: crash after upgrade pacific to main Duplicate
Copied to cephsqlite - Backport #57183: pacific: crash: pthread_mutex_lock() Resolved
Copied to cephsqlite - Backport #57184: quincy: crash: pthread_mutex_lock() Resolved

History

#1 Updated by Telemetry Bot 5 months ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v17.1.0, v17.2.0 added

#2 Updated by Laura Flores 4 months ago

  • Duplicated by Bug #56659: mgr: crash after upgrade pacific to main added

#3 Updated by Laura Flores 4 months ago

  • Priority changed from Normal to High

#5 Updated by Patrick Donnelly 4 months ago

Thanks for linking these tickets Laura! It helps to know this is not an obscure problem Milind found maybe somehow caused by his changes.

So I looked at a run Milind did:

/ceph/teuthology-archive/mchangir-2022-07-26_10:40:37-upgrade-wip-mchangir-test-pacific-to-quincy-upgrade-with-snap_schedule-test-pacific-distro-default-smithi/6947433/teuthology.log

It seems that libsqlite3-mod-ceph is not upgraded to 17.2.0 which causes this (symbol locations will differ, causing the segtfault). I'm not exactly sure yet why that doesn't happen.

#6 Updated by Patrick Donnelly 4 months ago

  • Status changed from New to Fix Under Review
  • Assignee set to Patrick Donnelly
  • Target version set to v18.0.0
  • Backport set to quincy,pacific
  • Pull request ID set to 47288

#7 Updated by Laura Flores 4 months ago

Thanks Patrick! I'll test this commit against Yuri's quincy p2p PR: https://github.com/ceph/ceph/pull/46896
(This is where we were experiencing this failure).

#8 Updated by Patrick Donnelly 4 months ago

  • Status changed from Fix Under Review to Pending Backport

#9 Updated by Backport Bot 4 months ago

#10 Updated by Backport Bot 4 months ago

#11 Updated by Backport Bot 4 months ago

  • Tags set to backport_processed

Also available in: Atom PDF