Bug #56788
opencrash: void KernelDevice::_aio_thread(): abort
0%
9e2d4644ff6c73397924fdfb6a4800a2c1b370e40fea5775166dc1d99c32875c
eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7
Description
Assert condition: abort
Assert function: void KernelDevice::_aio_thread()
Sanitized backtrace:
pthread_kill() raise() KernelDevice::_aio_thread() KernelDevice::AioCompletionThread::entry()
Crash dump sample:
{ "archived": "2022-07-09 21:01:05.232857", "assert_condition": "abort", "assert_file": "blk/kernel/KernelDevice.cc", "assert_func": "void KernelDevice::_aio_thread()", "assert_line": 617, "assert_msg": "blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7f58457fa640 time 2022-07-09T16:53:00.980762-0400\nblk/kernel/KernelDevice.cc: 617: ceph_abort_msg(\"Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!\")", "assert_thread_name": "bstore_aio", "backtrace": [ "/lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f5861dce520]", "pthread_kill()", "raise()", "abort()", "(ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x190) [0x5640326a3e9e]", "(KernelDevice::_aio_thread()+0xe59) [0x564033285c39]", "(KernelDevice::AioCompletionThread::entry()+0x11) [0x56403328f551]", "/lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f5861e20b43]", "/lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f5861eb2a00]" ], "ceph_version": "17.2.0", "crash_id": "2022-07-09T20:53:00.988485Z_2a09a4a9-254f-4ce6-bc45-2f703d2c4f1c", "entity_name": "osd.bafedfc7637c5f9baa507101a851d388a1d22b0e", "io_error": true, "io_error_code": -5, "io_error_devname": "dm-2", "io_error_length": 4096, "io_error_offset": 22167552, "io_error_optype": 8, "io_error_path": "/var/lib/ceph/osd/ceph-6/block", "os_id": "22.04", "os_name": "Ubuntu 22.04 LTS", "os_version": "22.04 LTS (Jammy Jellyfish)", "os_version_id": "22.04", "process_name": "ceph-osd", "stack_sig": "eceb646d37a5aa7dec4dd9a0298f64e5e5519f44e9589e68836110dcd62e82a7", "timestamp": "2022-07-09T20:53:00.988485Z", "utsname_machine": "x86_64", "utsname_release": "5.15.0-39-generic", "utsname_sysname": "Linux", "utsname_version": "#42-Ubuntu SMP Thu Jun 9 23:42:32 UTC 2022" }
Updated by Telemetry Bot almost 2 years ago
Updated by Laura Flores about 1 year ago
- Translation missing: en.field_tag_list set to test-failure
- Crash signature (v1) updated (diff)
/a/yuriw-2023-02-13_21:53:12-rados-wip-yuri-testing-2023-02-06-1155-quincy-distro-default-smithi/7172130
Updated by Laura Flores 5 months ago
/a/yuriw-2023-12-07_16:42:12-rados-wip-yuri2-testing-2023-12-06-1239-distro-default-smithi/7482250
Updated by Laura Flores 4 months ago
/a/lflores-2024-01-10_23:43:40-rados-wip-yuri11-testing-2024-01-10-1124-pacific-distro-default-smithi/7512920
Updated by Igor Fedotov 4 months ago
Reported error code (similar to the one seen before) is 28 (ENOSPC) so I believe this isn't a BlueStore problem but rather something wrong with the environment.
Log snippet:
-80> 2024-01-11T00:31:01.389+0000 7f02f6d2f700 -1 bdev(0x558e87df6400 /var/lib/ceph/osd/ceph-0/block) _aio_thread got r=-28 ((28) No space left on device)
Updated by Laura Flores 3 months ago
/a/yuriw-2024-01-19_15:23:50-rados-wip-yuri7-testing-2024-01-18-1327-distro-default-smithi/7522671
Updated by Laura Flores 2 months ago
/a/yuriw-2024-02-28_15:47:41-rados-wip-yuri4-testing-2024-02-27-1111-quincy-distro-default-smithi/7575637
Updated by Laura Flores 10 days ago
/a/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668449
2024-04-22T18:43:31.793 INFO:tasks.ceph.osd.1.smithi028.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1240-g6a7528e4/rpm/el9/BUILD/ceph-18.2.2-1240-g6a7528e4/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fd2bd3d5640 time 2024-04-22T18:43:31.793839+0000 2024-04-22T18:43:31.793 INFO:tasks.ceph.osd.1.smithi028.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1240-g6a7528e4/rpm/el9/BUILD/ceph-18.2.2-1240-g6a7528e4/src/blk/kernel/KernelDevice.cc: 633: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!") 2024-04-22T18:43:31.793 INFO:tasks.ceph.osd.1.smithi028.stderr: ceph version 18.2.2-1240-g6a7528e4 (6a7528e4aecd36b18c4b41cee6012e9f92aa7ab0) reef (stable) 2024-04-22T18:43:31.794 INFO:tasks.ceph.osd.1.smithi028.stderr: 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc6) [0x5619211f99ae] 2024-04-22T18:43:31.794 INFO:tasks.ceph.osd.1.smithi028.stderr: 2: (KernelDevice::_aio_thread()+0x11e0) [0x561921b922e0] 2024-04-22T18:43:31.794 INFO:tasks.ceph.osd.1.smithi028.stderr: 3: ceph-osd(+0xd62451) [0x561921b92451] 2024-04-22T18:43:31.794 INFO:tasks.ceph.osd.1.smithi028.stderr: 4: /lib64/libc.so.6(+0x89c02) [0x7fd2caa89c02] 2024-04-22T18:43:31.794 INFO:tasks.ceph.osd.1.smithi028.stderr: 5: /lib64/libc.so.6(+0x10ec40) [0x7fd2cab0ec40]
/a/yuriw-2024-04-22_18:19:58-rados-wip-yuri2-testing-2024-04-17-0823-reef-distro-default-smithi/7668449$ find . -name \*gz -print0 | xargs -0 zgrep "No space left on device"
./remote/smithi028/log/ceph-osd.3.log.gz:2024-04-22T18:43:23.309+0000 7f767abce640 -1 bdev(0x561db7dc6e00 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-28 ((28) No space left on device) ./remote/smithi028/log/ceph-osd.3.log.gz: -8> 2024-04-22T18:43:23.309+0000 7f767abce640 -1 bdev(0x561db7dc6e00 /var/lib/ceph/osd/ceph-3/block) _aio_thread got r=-28 ((28) No space left on device)