Project

General

Profile

Actions
Copy link

Bug #56194

open

crash: OpTracker::~OpTracker(): assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())

Added by Telemetry Bot almost 2 years ago. Updated 11 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v13.2.8, Ceph - v14.1.0, Ceph - v14.2.0, Ceph - v14.2.1, Ceph - v14.2.10, Ceph - v14.2.11, Ceph - v14.2.18, Ceph - v14.2.4, Ceph - v14.2.6, Ceph - v15.2.13, Ceph - v15.2.16, Ceph - v15.2.8, Ceph - v16.2.7, Ceph - v16.2.9
ceph-qa-suite:
Component(RADOS):
Pull request ID:
Crash signature (v1):

04d9f3d47f8b6c31c4a7bbb86963f78f192ba55382ceb6f803b96e6d64826249
1412f2c9f4ff467425a1b38907b896488627cba4d516b0e74af767b4c2fabe6f
181659e0a467b58588f5f0ff30fb1e44e8089642e4dab5276b67a986fd1eb25f
260995e374d7a3a79e28b7655e7b668e41ed271543d21e78105b2a8aaa7e0927
2d28de22352ffbab7935e792a16681b52a232f4fe24a3853c6252caac8f13c76
3f938f4ec18c11e1270405e3d5619c9e4dd8332ba3099c818976cca4567ce6aa
4746e7ed03ef91e4a7a1b4d70c512474cfafbe89386400a9301e53d62ad21b35
54ebe531ea147f2517522379b181a809a6ddf2e0ed14bbcb96d83e4ba9cf4f2a
55cedb4a0f7e2a6136596bacce6c9b811478ae7bd1a116f7921186c63f0874bf
72e3066bb0802a798218887a051a2a2698278abc9bf78d6b1bf1950ca4bd296f
7de92ab6e07cb7e6dfe75a2c66f716e5687c588df1df66a4344dc1f5b1e7b0d7
834d96d15b01f799e928c27fc85a248e289a65e43a8f11b50a4028884776c8f9
8ac4259cf34a2bff5255d08a539d67618427d9901d62aea40410314ef1b43b4e
94dee4b639b5c7cadeecbe7681fd1baebda2aed38df2ed58522367bca1657341
acf5960aae5c179be3c7f7d8914176a3eee4f84f3fa1a535303529f84381d318
cb09b68b8e5d892a04e08e74425507a5fa25c91f75eff4641509753310950365
dc6a93ee77f22485a7c2287052576721690cf280318c2bb53aa847d15da3a63c
3d6644ff99b9d46528379bf202a5f48cb2a95090323253c06242d163fd749519
47d8e292605e0f4c9f66c521d0c38143709bae82dd639ff4678cc84d7fc81289

Crash signature (v2):

e86e0b84f213b49af7dc5555afc83eb9c6ffffdab3b5f1083227207843b08cdc

Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=e86e0b84f213b49af7dc5555afc83eb9c6ffffdab3b5f1083227207843b08cdc

Assert condition: (sharded_in_flight_list.back())->ops_in_flight_sharded.empty()
Assert function: OpTracker::~OpTracker()

Sanitized backtrace:

    Monitor::~Monitor()
    Monitor::~Monitor()

Crash dump sample:
{
    "archived": "2022-05-26 12:32:48.037354",
    "assert_condition": "(sharded_in_flight_list.back())->ops_in_flight_sharded.empty()",
    "assert_file": "common/TrackedOp.cc",
    "assert_func": "OpTracker::~OpTracker()",
    "assert_line": 173,
    "assert_msg": "common/TrackedOp.cc: In function 'OpTracker::~OpTracker()' thread 7fb28d3f46c0 time 2022-05-26T20:22:59.005991+0800\ncommon/TrackedOp.cc: 173: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())",
    "assert_thread_name": "ceph-mon",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb2833f7980]",
        "pthread_getname_np()",
        "(ceph::logging::Log::dump_recent()+0x58f) [0x7fb2848bc74f]",
        "/usr/bin/ceph-mon(+0x9fe681) [0x55a4e727d681]",
        "/lib/x86_64-linux-gnu/libpthread.so.0(+0x12980) [0x7fb2833f7980]",
        "gsignal()",
        "abort()",
        "(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x19c) [0x7fb28457e3be]",
        "(ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...)+0) [0x7fb28457e548]",
        "/usr/lib/ceph/libceph-common.so.2(+0x31448d) [0x7fb28462e48d]",
        "(Monitor::~Monitor()+0xae) [0x55a4e702519e]",
        "(Monitor::~Monitor()+0x9) [0x55a4e70260c9]",
        "main()",
        "__libc_start_main()",
        "_start()" 
    ],
    "ceph_version": "16.2.9",
    "crash_id": "2022-05-26T12:22:59.083679Z_1f3be27d-d898-452b-8bbe-c47026a5dc94",
    "entity_name": "mon.c64b3f340dd5b0bfb2e7728782b4869d5b327a5a",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "18.04.6 LTS (Bionic Beaver)",
    "os_version_id": "18.04",
    "process_name": "ceph-mon",
    "stack_sig": "acf5960aae5c179be3c7f7d8914176a3eee4f84f3fa1a535303529f84381d318",
    "timestamp": "2022-05-26T12:22:59.083679Z",
    "utsname_machine": "x86_64",
    "utsname_release": "4.15.0-180-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#189-Ubuntu SMP Wed May 18 14:13:57 UTC 2022" 
}

Actions
Copy link
 #1

Updated by Telemetry Bot almost 2 years ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v13.2.8, v14.1.0, v14.2.0, v14.2.1, v14.2.10, v14.2.11, v14.2.18, v14.2.4, v14.2.6, v15.2.13, v15.2.16, v15.2.8, v16.2.9 added
Actions
Copy link
 #2

Updated by Telemetry Bot over 1 year ago

  • Crash signature (v1) updated (diff)
  • Affected Versions v16.2.7 added
Actions
Copy link
 #3

Updated by can zhu 11 months ago

is there any updates for this issue?

Actions
Copy link
 #4

Updated by can zhu 11 months ago

ceph v16.2.10

{
"assert_condition": "(sharded_in_flight_list.back())->ops_in_flight_sharded.empty()",
"assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc",
"assert_func": "OpTracker::~OpTracker()",
"assert_line": 173,
"assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc: In function 'OpTracker::~OpTracker()' thread 7fa6361e4880 time 2023-05-12T02:24:43.162422+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc: 173: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())\n",
"assert_thread_name": "ceph-mon",
"backtrace": [
"/lib64/libpthread.so.0(+0x12ce0) [0x7fa62b099ce0]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fa62d35fe39]",
"/usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7fa62d360002]",
"(OpTracker::~OpTracker()+0x39) [0x7fa62d45dd59]",
"(Monitor::~Monitor()+0xb2) [0x555b026eb272]",
"(Monitor::~Monitor()+0xd) [0x555b026ec05d]",
"main()",
"__libc_start_main()",
"_start()"
],
"ceph_version": "16.2.10",
"crash_id": "2023-05-12T02:24:43.165745Z_fe8216ac-7c83-4b9b-994c-0b63bbc4fba1",
"entity_name": "mon.c",
"os_id": "centos",
"os_name": "CentOS Stream",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mon",
"stack_sig": "f356906c2e640adf2ac0749d754e41bdb688375d36e0eb749511918a88204664",
"timestamp": "2023-05-12T02:24:43.165745Z",
"utsname_hostname": "smd-node04",
"utsname_machine": "x86_64",
"utsname_release": "3.10.0-1160.45.1.el7.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021"
}

Actions
Copy link
 #5

Updated by can zhu 11 months ago

can zhu wrote:

ceph v16.2.10

{
"assert_condition": "(sharded_in_flight_list.back())->ops_in_flight_sharded.empty()",
"assert_file": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc",
"assert_func": "OpTracker::~OpTracker()",
"assert_line": 173,
"assert_msg": "/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc: In function 'OpTracker::~OpTracker()' thread 7fa6361e4880 time 2023-05-12T02:24:43.162422+0000\n/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.10/rpm/el8/BUILD/ceph-16.2.10/src/common/TrackedOp.cc: 173: FAILED ceph_assert((sharded_in_flight_list.back())->ops_in_flight_sharded.empty())\n",
"assert_thread_name": "ceph-mon",
"backtrace": [
"/lib64/libpthread.so.0(+0x12ce0) [0x7fa62b099ce0]",
"gsignal()",
"abort()",
"(ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x1a9) [0x7fa62d35fe39]",
"/usr/lib64/ceph/libceph-common.so.2(+0x277002) [0x7fa62d360002]",
"(OpTracker::~OpTracker()+0x39) [0x7fa62d45dd59]",
"(Monitor::~Monitor()+0xb2) [0x555b026eb272]",
"(Monitor::~Monitor()+0xd) [0x555b026ec05d]",
"main()",
"__libc_start_main()",
"_start()"
],
"ceph_version": "16.2.10",
"crash_id": "2023-05-12T02:24:43.165745Z_fe8216ac-7c83-4b9b-994c-0b63bbc4fba1",
"entity_name": "mon.c",
"os_id": "centos",
"os_name": "CentOS Stream",
"os_version": "8",
"os_version_id": "8",
"process_name": "ceph-mon",
"stack_sig": "f356906c2e640adf2ac0749d754e41bdb688375d36e0eb749511918a88204664",
"timestamp": "2023-05-12T02:24:43.165745Z",
"utsname_hostname": "smd-node04",
"utsname_machine": "x86_64",
"utsname_release": "3.10.0-1160.45.1.el7.x86_64",
"utsname_sysname": "Linux",
"utsname_version": "#1 SMP Wed Oct 13 17:20:51 UTC 2021"
}

Actions
Copy link

Also available in: Atom PDF