Project

General

Profile

Actions

Bug #61140

open

crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)

Added by Telemetry Bot 11 months ago. Updated about 2 months ago.

Status:
Pending Backport
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
backport_processed
Backport:
quincy, reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):

7985ad40f80529b5f15babb32c6b7f23ef34f6d1e8c32f91e9aa6d2baf8b507d


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=8a0ffcca7ae094e79a916d2fc1361dbb0e86fc103fa8569c325c1a93bb68b801

Assert condition: end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout
Assert function: int OSD::shutdown()

Sanitized backtrace:

    raise()
    OSD::shutdown()
    SignalHandler::entry()

Crash dump sample:
{
    "assert_condition": "end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout",
    "assert_file": "osd/OSD.cc",
    "assert_func": "int OSD::shutdown()",
    "assert_line": 4340,
    "assert_msg": "osd/OSD.cc: In function 'int OSD::shutdown()' thread 7f65b14cc640 time 2023-05-08T06:31:17.573014-0500\nosd/OSD.cc: 4340: FAILED ceph_assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)",
    "assert_thread_name": "signal_handler",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libc.so.6(+0x3daf0) [0x7f65b4e33af0]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x8983c) [0x7f65b4e7f83c]",
        "raise()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17e) [0x5644a013afca]",
        "/usr/bin/ceph-osd(+0xc2310e) [0x5644a013b10e]",
        "(OSD::shutdown()+0x135d) [0x5644a023087d]",
        "(SignalHandler::entry()+0x648) [0x5644a08b6408]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x87b27) [0x7f65b4e7db27]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x10a78c) [0x7f65b4f0078c]" 
    ],
    "ceph_version": "17.2.5",
    "crash_id": "2023-05-08T11:31:17.584022Z_72dfa87b-5999-4ef7-847d-a88fbc37e171",
    "entity_name": "osd.2708ef1a39b11e5ae43934bfe9869e66050afdc8",
    "os_id": "11",
    "os_name": "Debian GNU/Linux 11 (bullseye)",
    "os_version": "11 (bullseye)",
    "os_version_id": "11",
    "process_name": "ceph-osd",
    "stack_sig": "7985ad40f80529b5f15babb32c6b7f23ef34f6d1e8c32f91e9aa6d2baf8b507d",
    "timestamp": "2023-05-08T11:31:17.584022Z",
    "utsname_machine": "x86_64",
    "utsname_release": "6.2.11-1-pve",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC PVE 6.2.11-1 (2023-04-20T09:59Z)" 
}


Related issues 5 (4 open1 closed)

Has duplicate RADOS - Bug #56292: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)New

Actions
Has duplicate RADOS - Bug #56873: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)New

Actions
Has duplicate RADOS - Bug #56896: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)NewRadoslaw Zarzynski

Actions
Copied to RADOS - Backport #63526: quincy: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)ResolvedActions
Copied to RADOS - Backport #63527: reef: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout)In ProgressActions
Actions #1

Updated by Telemetry Bot 11 months ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v17.2.5 added
Actions #2

Updated by Matan Breizman 10 months ago

  • Has duplicate Bug #56292: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Actions #3

Updated by Matan Breizman 10 months ago

  • Has duplicate Bug #56873: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Actions #4

Updated by Matan Breizman 10 months ago

  • Has duplicate Bug #56896: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Actions #6

Updated by Gabriel BenHanokh 10 months ago

This error means that fast-shutdown exceeded the 15 second timeout.
It might happen if log-level is set too high
Might be a good idea to check the log-level in the assert

Actions #7

Updated by Christian Rohmann 8 months ago

We just observed this "noise" for quite a few OSDs on rolling reboots.... would be nice to have this "not" treated an a crash anymore.

Actions #8

Updated by Igor Fedotov 7 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 53530
Actions #9

Updated by Igor Fedotov 7 months ago

  • Backport set to quincy, reef
Actions #10

Updated by Igor Fedotov 5 months ago

  • Status changed from Fix Under Review to Pending Backport
Actions #11

Updated by Backport Bot 5 months ago

  • Copied to Backport #63526: quincy: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Actions #12

Updated by Backport Bot 5 months ago

  • Copied to Backport #63527: reef: crash: int OSD::shutdown(): assert(end_time - start_time_func < cct->_conf->osd_fast_shutdown_timeout) added
Actions #13

Updated by Backport Bot 5 months ago

  • Tags set to backport_processed
Actions

Also available in: Atom PDF