Project

General

Profile

Actions

Bug #56788

open

crash: void KernelDevice::_aio_thread(): abort

Added by Telemetry Bot over 1 year ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Telemetry
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):

9e2d4644ff6c73397924fdfb6a4800a2c1b370e40fea5775166dc1d99c32875c
eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7


Description

http://telemetry.front.sepia.ceph.com:4000/d/jByk5HaMz/crash-spec-x-ray?orgId=1&var-sig_v2=c1fd5f4de1d415baece7e11982350afcdc4bd5d9fcebaf92de8aa1cd60c38f34

Assert condition: abort
Assert function: void KernelDevice::_aio_thread()

Sanitized backtrace:

    pthread_kill()
    raise()
    KernelDevice::_aio_thread()
    KernelDevice::AioCompletionThread::entry()

Crash dump sample:
{
    "archived": "2022-07-09 21:01:05.232857",
    "assert_condition": "abort",
    "assert_file": "blk/kernel/KernelDevice.cc",
    "assert_func": "void KernelDevice::_aio_thread()",
    "assert_line": 617,
    "assert_msg": "blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f58457fa640 time 2022-07-09T16:53:00.980762-0400\nblk/kernel/KernelDevice.cc: 617: ceph_abort_msg(\"Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!\")",
    "assert_thread_name": "bstore_aio",
    "backtrace": [
        "/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5861dce520]",
        "pthread_kill()",
        "raise()",
        "abort()",
        "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x5640326a3e9e]",
        "(KernelDevice::_aio_thread()+0xe59) [0x564033285c39]",
        "(KernelDevice::AioCompletionThread::entry()+0x11) [0x56403328f551]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5861e20b43]",
        "/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5861eb2a00]" 
    ],
    "ceph_version": "17.2.0",
    "crash_id": "2022-07-09T20:53:00.988485Z_2a09a4a9-254f-4ce6-bc45-2f703d2c4f1c",
    "entity_name": "osd.bafedfc7637c5f9baa507101a851d388a1d22b0e",
    "io_error": true,
    "io_error_code": -5,
    "io_error_devname": "dm-2",
    "io_error_length": 4096,
    "io_error_offset": 22167552,
    "io_error_optype": 8,
    "io_error_path": "/var/lib/ceph/osd/ceph-6/block",
    "os_id": "22.04",
    "os_name": "Ubuntu 22.04 LTS",
    "os_version": "22.04 LTS (Jammy Jellyfish)",
    "os_version_id": "22.04",
    "process_name": "ceph-osd",
    "stack_sig": "eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7",
    "timestamp": "2022-07-09T20:53:00.988485Z",
    "utsname_machine": "x86_64",
    "utsname_release": "5.15.0-39-generic",
    "utsname_sysname": "Linux",
    "utsname_version": "#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022" 
}

Actions #1

Updated by Telemetry Bot over 1 year ago

  • Crash signature (v1) updated (diff)
  • Crash signature (v2) updated (diff)
  • Affected Versions v16.2.6, v16.2.7, v17.2.0 added
Actions #2

Updated by Laura Flores about 1 year ago

  • Translation missing: en.field_tag_list set to test-failure
  • Crash signature (v1) updated (diff)

/a/yuriw-2023-02-13_21:53:12-rados-wip-yuri-testing-2023-02-06-1155-quincy-distro-default-smithi/7172130

Actions #3

Updated by Laura Flores 4 months ago

/a/yuriw-2023-12-07_16:42:12-rados-wip-yuri2-testing-2023-12-06-1239-distro-default-smithi/7482250

Actions #4

Updated by Laura Flores 3 months ago

/a/lflores-2024-01-10_23:43:40-rados-wip-yuri11-testing-2024-01-10-1124-pacific-distro-default-smithi/7512920

Actions #5

Updated by Igor Fedotov 3 months ago

Reported error code (similar to the one seen before) is 28 (ENOSPC) so I believe this isn't a BlueStore problem but rather something wrong with the environment.

Log snippet:
-80> 2024-01-11T00:31:01.389+0000 7f02f6d2f700 -1 bdev(0x558e87df6400 /var/lib/ceph/osd/ceph-0/block) _aio_thread got r=-28 ((28) No space left on device)

Actions #6

Updated by Laura Flores 3 months ago

/a/yuriw-2024-01-19_15:23:50-rados-wip-yuri7-testing-2024-01-18-1327-distro-default-smithi/7522671

Actions #7

Updated by Laura Flores about 2 months ago

/a/yuriw-2024-02-28_15:47:41-rados-wip-yuri4-testing-2024-02-27-1111-quincy-distro-default-smithi/7575637

Actions

Also available in: Atom PDF