Project

General

Profile

Bug #40698

mgr: openSSL SSL_library_init() failed to initialize

Added by Xuehan Xu about 1 month ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Urgent
Category:
ceph-mgr
Target version:
Start date:
07/09/2019
Due date:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

Recently, when trying to do some test, the ceph-mgr failed to boot:

2019-07-09T13:29:45.164+0800 7ff56b7e3b80 -1 /home/xuxuehan/mnt/github-ceph/src/common/lockdep.cc: In function 'int lockdep_will_lock(const char*, int, bool, bool)' thread 7ff56b7e3b80 time 2019-07-09T13:29:45.155129+0800
/home/xuxuehan/mnt/github-ceph/src/common/lockdep.cc: 305: ceph_abort_msg("abort() called")

 ceph version v15.0.0-2554-g221d976e3c (221d976e3c21e0d99c82f3d808b7bd1f7e5a406d) octopus (dev)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::string const&)+0xfe) [0x7ff5623ea874]
 2: (lockdep_will_lock(char const*, int, bool, bool)+0x44b) [0x7ff5624d1b6f]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::_will_lock(bool)+0x4a) [0x7ff5624dd89c]
 4: (ceph::shared_mutex_debug::lock()+0x3a) [0x7ff56257bbee]
 5: (()+0x143a532) [0x7ff56241b532]
 6: (CRYPTO_mem_ctrl()+0x39) [0x7ff55fdd1be9]
 7: (()+0x4a60e) [0x7ff54a51b60e]
 8: (SSL_COMP_get_compression_methods()+0x9) [0x7ff54a51d899]
 9: (SSL_library_init()+0x179) [0x7ff54a5237b9]
 10: (init_hashlib()+0x1f) [0x7ff54a74614f]
 11: (_PyImport_LoadDynamicModule()+0x99) [0x7ff560d0ed69]
 12: (()+0xf7e41) [0x7ff560d0ce41]
 13: (()+0xf808d) [0x7ff560d0d08d]
 14: (PyImport_ImportModuleLevel()+0x15e) [0x7ff560d0da6e]
 15: (()+0xdbd1f) [0x7ff560cf0d1f]
 16: (PyObject_Call()+0x43) [0x7ff560c60a63]
 17: (PyEval_CallObjectWithKeywords()+0x47) [0x7ff560cf28f7]
 18: (PyEval_EvalFrameEx()+0x46b5) [0x7ff560cf75b5]
 19: (PyEval_EvalCodeEx()+0x7ed) [0x7ff560cfc03d]

mgr.x.log View (86.6 KB) Xuehan Xu, 07/09/2019 08:33 AM

mgr.x.log.lockdep_force_backtrace (115 KB) Xuehan Xu, 07/10/2019 07:25 AM


Related issues

Related to mgr - Bug #40688: mgr/dashboard: ceph dashboard tests Jenkins job fails with "*** Caught signal (Aborted) ** in thread 7f9d14518b80 thread_name:ceph-mgr" Resolved 07/08/2019
Related to mgr - Bug #40813: mgr/dashboard: Remove --nolockdep option from e2e script and vstart_runner.py New 07/12/2019
Duplicated by mgr - Bug #40724: mgr: recursive lock in vstart Duplicate

History

#1 Updated by Sebastian Wagner about 1 month ago

  • Subject changed from mgr: failed to boot to mgr: openSSL SSL_library_init() failed to initialize

Could you please attach the full mgr logs?

#2 Updated by Xuehan Xu about 1 month ago

Sebastian Wagner wrote:

Could you please attach the full mgr logs?

Yes, it's in the attached file:-)

#3 Updated by Radoslaw Zarzynski about 1 month ago

Could we have a log with lockdep_force_backtrace set to true?
The idea is to figure out the place the where the lock has been taken initially.

#4 Updated by Kefu Chai about 1 month ago

  • Status changed from New to Need More Info
  • Assignee set to Xuehan Xu

#5 Updated by Xuehan Xu about 1 month ago

Radoslaw Zarzynski wrote:

Could we have a log with lockdep_force_backtrace set to true?
The idea is to figure out the place the where the lock has been taken initially.

Here is the log with "lockdep_force_backtrace = true"

#6 Updated by Sebastian Wagner about 1 month ago

  • Duplicated by Bug #40724: mgr: recursive lock in vstart added

#7 Updated by Sebastian Wagner about 1 month ago

  • Priority changed from Normal to Urgent

#8 Updated by Sebastian Wagner about 1 month ago

  • Related to Bug #40688: mgr/dashboard: ceph dashboard tests Jenkins job fails with "*** Caught signal (Aborted) ** in thread 7f9d14518b80 thread_name:ceph-mgr" added

#9 Updated by Radoslaw Zarzynski about 1 month ago

  • Status changed from Need More Info to In Progress
  • Assignee changed from Xuehan Xu to Radoslaw Zarzynski

#10 Updated by Radoslaw Zarzynski about 1 month ago

  • Status changed from In Progress to Need Review

#11 Updated by Sebastian Wagner about 1 month ago

  • Pull request ID set to 28987

#12 Updated by Lenz Grimmer about 1 month ago

Radoslaw Zarzynski wrote:

https://gist.github.com/rzarzynski/b2e73a4fefd86c1cbbafc76f8e3ee55f

Thanks for the detailed analysis! Impressive :)

#13 Updated by Sage Weil about 1 month ago

  • Status changed from Need Review to Resolved

#14 Updated by Laura Paduano about 1 month ago

  • Related to Bug #40813: mgr/dashboard: Remove --nolockdep option from e2e script and vstart_runner.py added

Also available in: Atom PDF