Bug #51919: crash: ceph::common::PerfCounters::inc(int, unsigned long) (in RGWAsyncFetchRemoteObj::_send_request()) - rgw - Ceph

Actions

Copy link

Bug #51919

closed

crash: ceph::common::PerfCounters::inc(int, unsigned long) (in RGWAsyncFetchRemoteObj::_send_request())

Added by Telemetry Bot over 2 years ago. Updated over 1 year ago.

Status:

Duplicate

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Telemetry

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v15.2.13, Ceph - v15.2.8, Ceph - v15.2.9

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

2bf1b3e02038e06d50abb448410d2c59001d10861a18e5c7cf1f3e8c1926b924
7d6ca6057edf55e9e3dea0fd7cdcd6e4f11f13c4a5d00a883206d07a1e5fdae0

Crash signature (v2):

853cf7f363a1445c83117d475dce60d305f1caaaed16406284277ec39e4e0161

Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=7d6ca6057edf55e9e3dea0fd7cdcd6e4f11f13c4a5d00a883206d07a1e5fdae0

Sanitized backtrace:

    ceph::common::PerfCounters::inc(int, unsigned long)
    RGWAsyncFetchRemoteObj::_send_request()
    RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)
    RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)
    ThreadPool::worker(ThreadPool::WorkThread*)
    ThreadPool::WorkThread::entry()
    clone()

Crash dump sample:

{
    "backtrace": [
        "(()+0x12b30) [0x7fdc44025b30]",
        "(ceph::common::PerfCounters::inc(int, unsigned long)+0x7) [0x7fdc44dc5187]",
        "(RGWAsyncFetchRemoteObj::_send_request()+0x3bc) [0x7fdc4f04859c]",
        "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x24) [0x7fdc4f042a74]",
        "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0x11) [0x7fdc4f04a5a1]",
        "(ThreadPool::worker(ThreadPool::WorkThread*)+0xe64) [0x7fdc44cc5d14]",
        "(ThreadPool::WorkThread::entry()+0x15) [0x7fdc44cc6575]",
        "(()+0x815a) [0x7fdc4401b15a]",
        "(clone()+0x43) [0x7fdc42728dd3]" 
    ],
    "ceph_version": "15.2.13",
    "crash_id": "2021-07-02T10:44:41.719078Z_05f92b3c-5ae9-4023-981d-22c96b652272",
    "entity_name": "client.9f5ba328f57e893aca80108d5e05c226d0071626",
    "os_id": "ol",
    "os_name": "Oracle Linux Server",
    "os_version": "8.4",
    "os_version_id": "8.4",
    "process_name": "radosgw",
    "stack_sig": "2bf1b3e02038e06d50abb448410d2c59001d10861a18e5c7cf1f3e8c1926b924",
    "timestamp": "2021-07-02T10:44:41.719078Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.4.17-2102.202.5.el8uek.x86_64",
    "utsname_sysname": "Linux",
    "utsname_version": "#2 SMP Sat May 22 16:16:03 PDT 2021" 
}

Related issues 1 (1 open — 0 closed)

Actions

Copy link

Updated by Telemetry Bot over 2 years ago

Crash signature (v1) updated (diff)
Crash signature (v2) updated (diff)
Affected Versions v15.2.13, v15.2.8, v15.2.9 added

Actions

Copy link

Updated by Sage Weil over 2 years ago

Subject changed from crash: ceph::common::PerfCounters::inc(int, unsigned long) to crash: ceph::common::PerfCounters::inc(int, unsigned long) (in RGWAsyncFetchRemoteObj::_send_request())

Actions

Copy link

Updated by Christian Rohmann over 2 years ago

I just wanted to report two more of those stack traces on a cluster serving as secondary for RADOSGW multisite.
No user traffic or even writes go to this cluster, just sync. Potentially there could have been restarts on the other end (master zone), but this is not certain.

{
    "backtrace": [
        "(()+0x3f040) [0x7fa54ba50040]",
        "(ceph::common::PerfCounters::inc(int, unsigned long)+0x3) [0x7fa5424fa643]",
        "(RGWAsyncFetchRemoteObj::_send_request()+0x436) [0x7fa54c54c596]",
        "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7fa54c546470]",
        "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7fa54c54e83d]",
        "(ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7fa5423e765a]",
        "(ThreadPool::WorkThread::entry()+0x11) [0x7fa5423e8541]",
        "(()+0x76db) [0x7fa54139e6db]",
        "(clone()+0x3f) [0x7fa54bb3271f]" 
    ],
    "ceph_version": "15.2.15",
    "crash_id": "2021-12-16T11:41:42.873320Z_51db945a-a15b-435f-bd14-79f46d8abff4",
    "entity_name": "client.rgw.fra-rgn-bckp-03",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "18.04.6 LTS (Bionic Beaver)",
    "os_version_id": "18.04",
    "process_name": "radosgw",
    "stack_sig": "2bf1b3e02038e06d50abb448410d2c59001d10861a18e5c7cf1f3e8c1926b924",
    "timestamp": "2021-12-16T11:41:42.873320Z",
    "utsname_hostname": "fra-rgn-bckp-03",
    "utsname_machine": "x86_64",
    "utsname_release": "4.15.0-163-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#171-Ubuntu SMP Fri Nov 5 11:55:11 UTC 2021" 
}

{
    "archived": "2021-12-06 22:36:55.334411",
    "backtrace": [
        "(()+0x3f040) [0x7f5ca33df040]",
        "(ceph::common::PerfCounters::inc(int, unsigned long)+0x3) [0x7f5c99e89643]",
        "(RGWAsyncFetchRemoteObj::_send_request()+0x615) [0x7f5ca3edb775]",
        "(RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f5ca3ed5470]",
        "(RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f5ca3edd83d]",
        "(ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f5c99d7665a]",
        "(ThreadPool::WorkThread::entry()+0x11) [0x7f5c99d77541]",
        "(()+0x76db) [0x7f5c98d2d6db]",
        "(clone()+0x3f) [0x7f5ca34c171f]" 
    ],
    "ceph_version": "15.2.15",
    "crash_id": "2021-12-06T22:34:04.683159Z_011665f3-887c-47bc-9af0-023e6bd03b8d",
    "entity_name": "client.rgw.fra-rgn-bckp-02",
    "os_id": "ubuntu",
    "os_name": "Ubuntu",
    "os_version": "18.04.6 LTS (Bionic Beaver)",
    "os_version_id": "18.04",
    "process_name": "radosgw",
    "stack_sig": "2bf1b3e02038e06d50abb448410d2c59001d10861a18e5c7cf1f3e8c1926b924",
    "timestamp": "2021-12-06T22:34:04.683159Z",
    "utsname_hostname": "fra-rgn-bckp-02",
    "utsname_machine": "x86_64",
    "utsname_release": "4.15.0-159-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#167-Ubuntu SMP Tue Sep 21 08:55:05 UTC 2021" 
}

Actions

Copy link