Bug #56274
crash: pthread_mutex_lock()
0%
1301087589e4b5967aa0dce3c10c5ea2a5d576eb6900d8325c24e9aa1b5b114f
Description
Sanitized backtrace:
pthread_mutex_lock() cephsqlite_setcct() Mgr::init() Context::complete(int) Finisher::finisher_thread_entry()
Crash dump sample:
{ "backtrace": [ "/lib/x86_64-linux-gnu/libpthread.so.0(+0x14420) [0x7f7ca7f4b420]", "pthread_mutex_lock()", "(ceph::logging::Log::submit_entry(ceph::logging::Entry&&)+0x45) [0x7f7ca87c8975]", "cephsqlite_setcct()", "(Mgr::init()+0x19c5) [0x557fcdda6a85]", "/usr/bin/ceph-mgr(+0x2789b1) [0x557fcdda79b1]", "(Context::complete(int)+0xd) [0x557fcdcf244d]", "(Finisher::finisher_thread_entry()+0x19d) [0x7f7ca847deed]", "/lib/x86_64-linux-gnu/libpthread.so.0(+0x8609) [0x7f7ca7f3f609]", "clone()" ], "ceph_version": "17.2.0", "crash_id": "2022-06-13T16:27:27.048191Z_19ce4dc9-111f-4a5b-97ac-7986716e4ab5", "entity_name": "mgr.82df13b301fa0a76d46f6a23c7761010ca88e6b9", "os_id": "ubuntu", "os_name": "Ubuntu", "os_version": "20.04.4 LTS (Focal Fossa)", "os_version_id": "20.04", "process_name": "ceph-mgr", "stack_sig": "1301087589e4b5967aa0dce3c10c5ea2a5d576eb6900d8325c24e9aa1b5b114f", "timestamp": "2022-06-13T16:27:27.048191Z", "utsname_machine": "x86_64", "utsname_release": "5.13.19-6-pve", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PVE 5.13.19-15 (Tue, 29 Mar 2022 15:59:50 +0200)" }
Related issues
History
#1 Updated by Telemetry Bot almost 2 years ago
#2 Updated by Laura Flores over 1 year ago
- Duplicated by Bug #56659: mgr: crash after upgrade pacific to main added
#3 Updated by Laura Flores over 1 year ago
- Priority changed from Normal to High
#4 Updated by Laura Flores over 1 year ago
This issue is a blocker for upgrade tests: http://pulpito.front.sepia.ceph.com/lflores-2022-07-26_19:56:29-upgrade:quincy-p2p:quincy-p2p-stress-split-quincy-distro-default-smithi/
See http://pulpito.front.sepia.ceph.com/lflores-2022-07-26_19:55:12-upgrade:quincy-p2p:quincy-p2p-parallel-quincy-distro-default-smithi/. This seems to strictly affect ubuntu and debian.
#5 Updated by Patrick Donnelly over 1 year ago
Thanks for linking these tickets Laura! It helps to know this is not an obscure problem Milind found maybe somehow caused by his changes.
So I looked at a run Milind did:
/ceph/teuthology-archive/mchangir-2022-07-26_10:40:37-upgrade-wip-mchangir-test-pacific-to-quincy-upgrade-with-snap_schedule-test-pacific-distro-default-smithi/6947433/teuthology.log
It seems that libsqlite3-mod-ceph is not upgraded to 17.2.0 which causes this (symbol locations will differ, causing the segtfault). I'm not exactly sure yet why that doesn't happen.
#6 Updated by Patrick Donnelly over 1 year ago
- Status changed from New to Fix Under Review
- Assignee set to Patrick Donnelly
- Target version set to v18.0.0
- Backport set to quincy,pacific
- Pull request ID set to 47288
#7 Updated by Laura Flores over 1 year ago
Thanks Patrick! I'll test this commit against Yuri's quincy p2p PR: https://github.com/ceph/ceph/pull/46896
(This is where we were experiencing this failure).
#8 Updated by Patrick Donnelly over 1 year ago
- Status changed from Fix Under Review to Pending Backport
#9 Updated by Backport Bot over 1 year ago
- Copied to Backport #57183: pacific: crash: pthread_mutex_lock() added
#10 Updated by Backport Bot over 1 year ago
- Copied to Backport #57184: quincy: crash: pthread_mutex_lock() added
#11 Updated by Backport Bot over 1 year ago
- Tags set to backport_processed
#12 Updated by Patrick Donnelly about 1 year ago
- Status changed from Pending Backport to Resolved