Bug #56788
opencrash: void KernelDevice::_aio_thread(): abort
0%
9e2d4644ff6c73397924fdfb6a4800a2c1b370e40fea5775166dc1d99c32875c
eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7
Description
Assert condition: abort
Assert function: void KernelDevice::_aio_thread()
Sanitized backtrace:
pthread_kill() raise() KernelDevice::_aio_thread() KernelDevice::AioCompletionThread::entry()
Crash dump sample:
{ "archived": "2022-07-09 21:01:05.232857", "assert_condition": "abort", "assert_file": "blk/kernel/KernelDevice.cc", "assert_func": "void KernelDevice::_aio_thread()", "assert_line": 617, "assert_msg": "blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f58457fa640 time 2022-07-09T16:53:00.980762-0400\nblk/kernel/KernelDevice.cc: 617: ceph_abort_msg(\"Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!\")", "assert_thread_name": "bstore_aio", "backtrace": [ "/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5861dce520]", "pthread_kill()", "raise()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x5640326a3e9e]", "(KernelDevice::_aio_thread()+0xe59) [0x564033285c39]", "(KernelDevice::AioCompletionThread::entry()+0x11) [0x56403328f551]", "/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5861e20b43]", "/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5861eb2a00]" ], "ceph_version": "17.2.0", "crash_id": "2022-07-09T20:53:00.988485Z_2a09a4a9-254f-4ce6-bc45-2f703d2c4f1c", "entity_name": "osd.bafedfc7637c5f9baa507101a851d388a1d22b0e", "io_error": true, "io_error_code": -5, "io_error_devname": "dm-2", "io_error_length": 4096, "io_error_offset": 22167552, "io_error_optype": 8, "io_error_path": "/var/lib/ceph/osd/ceph-6/block", "os_id": "22.04", "os_name": "Ubuntu 22.04 LTS", "os_version": "22.04 LTS (Jammy Jellyfish)", "os_version_id": "22.04", "process_name": "ceph-osd", "stack_sig": "eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7", "timestamp": "2022-07-09T20:53:00.988485Z", "utsname_machine": "x86_64", "utsname_release": "5.15.0-39-generic", "utsname_sysname": "Linux", "utsname_version": "#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022" }
Updated by Telemetry Bot over 1 year ago
Updated by Laura Flores about 1 year ago
- Translation missing: en.field_tag_list set to test-failure
- Crash signature (v1) updated (diff)
/a/yuriw-2023-02-13_21:53:12-rados-wip-yuri-testing-2023-02-06-1155-quincy-distro-default-smithi/7172130
Updated by Laura Flores 4 months ago
/a/yuriw-2023-12-07_16:42:12-rados-wip-yuri2-testing-2023-12-06-1239-distro-default-smithi/7482250
Updated by Laura Flores 3 months ago
/a/lflores-2024-01-10_23:43:40-rados-wip-yuri11-testing-2024-01-10-1124-pacific-distro-default-smithi/7512920
Updated by Igor Fedotov 3 months ago
Reported error code (similar to the one seen before) is 28 (ENOSPC) so I believe this isn't a BlueStore problem but rather something wrong with the environment.
Log snippet:
-80> 2024-01-11T00:31:01.389+0000 7f02f6d2f700 -1 bdev(0x558e87df6400 /var/lib/ceph/osd/ceph-0/block) _aio_thread got r=-28 ((28) No space left on device)
Updated by Laura Flores 3 months ago
/a/yuriw-2024-01-19_15:23:50-rados-wip-yuri7-testing-2024-01-18-1327-distro-default-smithi/7522671
Updated by Laura Flores about 2 months ago
/a/yuriw-2024-02-28_15:47:41-rados-wip-yuri4-testing-2024-02-27-1111-quincy-distro-default-smithi/7575637