Project

General

Profile

Actions

Bug #40698

closed

mgr: openSSL SSL_library_init() failed to initialize

Added by Xuehan Xu almost 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
Urgent
Category:
ceph-mgr
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Recently, when trying to do some test, the ceph-mgr failed to boot:

2019-07-09T13:29:45.164+0800 7ff56b7e3b80 -1 /home/xuxuehan/mnt/github-ceph/src/common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int, bool, bool)' thread 7ff56b7e3b80 time 2019-07-09T13:29:45.155129+0800
/home/xuxuehan/mnt/github-ceph/src/common/lockdep.cc: 305: ceph_abort_msg("abort() called")

 ceph version v15.0.0-2554-g221d976e3c (221d976e3c21e0d99c82f3d808b7bd1f7e5a406d) octopus (dev)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xfe) [0x7ff5623ea874]
 2: (lockdep_will_lock(char const*, int, bool, bool)+0x44b) [0x7ff5624d1b6f]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::_will_lock(bool)+0x4a) [0x7ff5624dd89c]
 4: (ceph::shared_mutex_debug::lock()+0x3a) [0x7ff56257bbee]
 5: (()+0x143a532) [0x7ff56241b532]
 6: (CRYPTO_mem_ctrl()+0x39) [0x7ff55fdd1be9]
 7: (()+0x4a60e) [0x7ff54a51b60e]
 8: (SSL_COMP_get_compression_methods()+0x9) [0x7ff54a51d899]
 9: (SSL_library_init()+0x179) [0x7ff54a5237b9]
 10: (init_hashlib()+0x1f) [0x7ff54a74614f]
 11: (_PyImport_LoadDynamicModule()+0x99) [0x7ff560d0ed69]
 12: (()+0xf7e41) [0x7ff560d0ce41]
 13: (()+0xf808d) [0x7ff560d0d08d]
 14: (PyImport_ImportModuleLevel()+0x15e) [0x7ff560d0da6e]
 15: (()+0xdbd1f) [0x7ff560cf0d1f]
 16: (PyObject_Call()+0x43) [0x7ff560c60a63]
 17: (PyEval_CallObjectWithKeywords()+0x47) [0x7ff560cf28f7]
 18: (PyEval_EvalFrameEx()+0x46b5) [0x7ff560cf75b5]
 19: (PyEval_EvalCodeEx()+0x7ed) [0x7ff560cfc03d]

Files

mgr.x.log (86.6 KB) mgr.x.log Xuehan Xu, 07/09/2019 08:33 AM
mgr.x.log.lockdep_force_backtrace (115 KB) mgr.x.log.lockdep_force_backtrace Xuehan Xu, 07/10/2019 07:25 AM

Related issues 3 (1 open2 closed)

Related to Dashboard - Bug #40688: mgr/dashboard: ceph dashboard tests Jenkins job fails with "*** Caught signal (Aborted) ** in thread 7f9d14518b80 thread_name:ceph-mgr"ResolvedLaura Paduano

Actions
Related to Dashboard - Bug #40813: mgr/dashboard: Remove --nolockdep option from e2e script and vstart_runner.pyNew

Actions
Has duplicate mgr - Bug #40724: mgr: recursive lock in vstartDuplicate

Actions
Actions #1

Updated by Sebastian Wagner almost 5 years ago

  • Subject changed from mgr: failed to boot to mgr: openSSL SSL_library_init() failed to initialize

Could you please attach the full mgr logs?

Actions #2

Updated by Xuehan Xu almost 5 years ago

Sebastian Wagner wrote:

Could you please attach the full mgr logs?

Yes, it's in the attached file:-)

Actions #3

Updated by Radoslaw Zarzynski almost 5 years ago

Could we have a log with lockdep_force_backtrace set to true?
The idea is to figure out the place the where the lock has been taken initially.

Actions #4

Updated by Kefu Chai almost 5 years ago

  • Status changed from New to Need More Info
  • Assignee set to Xuehan Xu
Actions #5

Updated by Xuehan Xu almost 5 years ago

Radoslaw Zarzynski wrote:

Could we have a log with lockdep_force_backtrace set to true?
The idea is to figure out the place the where the lock has been taken initially.

Here is the log with "lockdep_force_backtrace = true"

Actions #6

Updated by Sebastian Wagner almost 5 years ago

  • Has duplicate Bug #40724: mgr: recursive lock in vstart added
Actions #7

Updated by Sebastian Wagner almost 5 years ago

  • Priority changed from Normal to Urgent
Actions #8

Updated by Sebastian Wagner almost 5 years ago

  • Related to Bug #40688: mgr/dashboard: ceph dashboard tests Jenkins job fails with "*** Caught signal (Aborted) ** in thread 7f9d14518b80 thread_name:ceph-mgr" added
Actions #9

Updated by Radoslaw Zarzynski almost 5 years ago

  • Status changed from Need More Info to In Progress
  • Assignee changed from Xuehan Xu to Radoslaw Zarzynski
Actions #10

Updated by Radoslaw Zarzynski almost 5 years ago

  • Status changed from In Progress to Fix Under Review
Actions #11

Updated by Sebastian Wagner almost 5 years ago

  • Pull request ID set to 28987
Actions #12

Updated by Lenz Grimmer almost 5 years ago

Radoslaw Zarzynski wrote:

https://gist.github.com/rzarzynski/b2e73a4fefd86c1cbbafc76f8e3ee55f

Thanks for the detailed analysis! Impressive :)

Actions #13

Updated by Sage Weil almost 5 years ago

  • Status changed from Fix Under Review to Resolved
Actions #14

Updated by Laura Paduano almost 5 years ago

  • Related to Bug #40813: mgr/dashboard: Remove --nolockdep option from e2e script and vstart_runner.py added
Actions

Also available in: Atom PDF