Actions
Bug #61140
opencrash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)
Status:
Pending Backport
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Telemetry
Tags:
backport_processed
Backport:
quincy, reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):
7985ad40f80529b5f15babb32c6b7f23ef34f6d1e8c32f91e9aa6d2baf8b507d
Crash signature (v2):
Description
Assert condition: end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout
Assert function: int OSD::shutdown()
Sanitized backtrace:
raise() OSD::shutdown() SignalHandler::entry()
Crash dump sample:
{ "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout", "assert_file": "osd/OSD.cc", "assert_func": "int OSD::shutdown()", "assert_line": 4340, "assert_msg": "osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f65b14cc640 time 2023-05-08T06:31:17.573014-0500\nosd/OSD.cc: 4340: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)", "assert_thread_name": "signal_handler", "backtrace": [ "/lib/x86_64-linux-gnu/libc.so.6(+0x3daf0) [0x7f65b4e33af0]", "/lib/x86_64-linux-gnu/libc.so.6(+0x8983c) [0x7f65b4e7f83c]", "raise()", "abort()", "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x5644a013afca]", "/usr/bin/ceph-osd(+0xc2310e) [0x5644a013b10e]", "(OSD::shutdown()+0x135d) [0x5644a023087d]", "(SignalHandler::entry()+0x648) [0x5644a08b6408]", "/lib/x86_64-linux-gnu/libc.so.6(+0x87b27) [0x7f65b4e7db27]", "/lib/x86_64-linux-gnu/libc.so.6(+0x10a78c) [0x7f65b4f0078c]" ], "ceph_version": "17.2.5", "crash_id": "2023-05-08T11:31:17.584022Z_72dfa87b-5999-4ef7-847d-a88fbc37e171", "entity_name": "osd.2708ef1a39b11e5ae43934bfe9869e66050afdc8", "os_id": "11", "os_name": "Debian GNU/Linux 11 (bullseye)", "os_version": "11 (bullseye)", "os_version_id": "11", "process_name": "ceph-osd", "stack_sig": "7985ad40f80529b5f15babb32c6b7f23ef34f6d1e8c32f91e9aa6d2baf8b507d", "timestamp": "2023-05-08T11:31:17.584022Z", "utsname_machine": "x86_64", "utsname_release": "6.2.11-1-pve", "utsname_sysname": "Linux", "utsname_version": "#1 SMP PREEMPT_DYNAMIC PVE 6.2.11-1 (2023-04-20T09:59Z)" }
Updated by Matan Breizman 10 months ago
- Has duplicate Bug #56292: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Updated by Matan Breizman 10 months ago
- Has duplicate Bug #56873: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Updated by Matan Breizman 10 months ago
- Has duplicate Bug #56896: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Updated by Matan Breizman 10 months ago
Seems to be introduced by:
https://github.com/ceph/ceph/pull/44913/commits/9b2a64a5f6ea743b2a4f4c2dbd703248d88b2a96
Updated by Gabriel BenHanokh 10 months ago
This error means that fast-shutdown exceeded the 15 second timeout.
It might happen if log-level is set too high
Might be a good idea to check the log-level in the assert
Updated by Christian Rohmann 8 months ago
We just observed this "noise" for quite a few OSDs on rolling reboots.... would be nice to have this "not" treated an a crash anymore.
Updated by Igor Fedotov 7 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 53530
Updated by Igor Fedotov 5 months ago
- Status changed from Fix Under Review to Pending Backport
Updated by Backport Bot 5 months ago
- Copied to Backport #63526: quincy: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Updated by Backport Bot 5 months ago
- Copied to Backport #63527: reef: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Updated by Yuri Weinstein about 2 months ago
Actions