Project

General

Profile

Actions
Copy link

Bug #54370

open

RuntimeError: could not find osd.1 with osd_fsid 620375df-6d0b-4104-bef9-240da4a6ec3d

Added by Kamoltat (Junior) Sirivadhna about 2 years ago. Updated about 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Crash signature (v1):
Crash signature (v2):

Description

yuriw-2022-02-17_23:23:56-rados-wip-yuri7-testing-2022-02-17-0852-pacific-distro-default-smithi/6693042

022-02-20T13:17:04.205 INFO:journalctl@ceph.iscsi.iscsi.a.smithi143.stdout:Feb 20 13:17:04 smithi143 systemd[1]: Started Ceph iscsi.iscsi.a for 26a88fdc-924e-11ec-8c35-001a4aab830c.
2022-02-20T13:17:04.837 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]: Traceback (most recent call last):
2022-02-20T13:17:04.837 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/sbin/ceph-volume", line 11, in <module>
2022-02-20T13:17:04.838 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
2022-02-20T13:17:04.838 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in __init__
2022-02-20T13:17:04.838 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     self.main(self.argv)
2022-02-20T13:17:04.838 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
2022-02-20T13:17:04.839 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     return f(*a, **kw)
2022-02-20T13:17:04.839 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
2022-02-20T13:17:04.839 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     terminal.dispatch(self.mapper, subcommand_args)
2022-02-20T13:17:04.839 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2022-02-20T13:17:04.840 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     instance.main()
2022-02-20T13:17:04.840 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 46, in main
2022-02-20T13:17:04.840 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     terminal.dispatch(self.mapper, self.argv)
2022-02-20T13:17:04.841 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
2022-02-20T13:17:04.841 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     instance.main()
2022-02-20T13:17:04.841 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", line 377, in main
2022-02-20T13:17:04.841 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     self.activate(args)
2022-02-20T13:17:04.841 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
2022-02-20T13:17:04.842 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     return func(*a, **kw)
2022-02-20T13:17:04.842 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:   File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/activate.py", line 282, in activate
2022-02-20T13:17:04.842 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]:     (osd_id, osd_fsid))
2022-02-20T13:17:04.842 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:17:04 smithi068 bash[30817]: RuntimeError: could not find osd.1 with osd_fsid 620375df-6d0b-4104-bef9-240da4a6ec3d

Actions
Copy link
 #1

Updated by Neha Ojha about 2 years ago

  • Project changed from RADOS to Infrastructure

smithi068 seems to have a bad disk

2022-02-20T13:16:49.874 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]: debug 2022-02-20T13:16:49.589+0000 7fc457b4f700 -1 bdev(0x5592c832c400 /var/lib/ceph/osd/ceph-1/block) _aio_thread got r=-5 ((5) Input/output error)
2022-02-20T13:16:49.874 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7-445-g9f91d3ca/rpm/el8/BUILD/ceph-16.2.7-445-g9f91d3ca/src/blk/kernel/KernelDevice.cc: In function 'void KernelDevice::_aio_thread()' thread 7fc457b4f700 time 2022-02-20T13:16:49.592967+0000
2022-02-20T13:16:49.874 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/16.2.7-445-g9f91d3ca/rpm/el8/BUILD/ceph-16.2.7-445-g9f91d3ca/src/blk/kernel/KernelDevice.cc: 600: ceph_abort_msg("Unexpected IO error. This may suggest a hardware issue. Please check your kernel log!")
2022-02-20T13:16:49.874 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  ceph version 16.2.7-445-g9f91d3ca (9f91d3caa3f16637a5668f2b678fb3a44b6977ba) pacific (stable)
2022-02-20T13:16:49.875 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xe5) [0x5592bb76bfbc]
2022-02-20T13:16:49.875 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  2: (KernelDevice::_aio_thread()+0x1254) [0x5592bc2c3c54]
2022-02-20T13:16:49.875 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  3: (KernelDevice::AioCompletionThread::entry()+0x11) [0x5592bc2cef51]
2022-02-20T13:16:49.875 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  4: /lib64/libpthread.so.0(+0x81cf) [0x7fc466bd11cf]
2022-02-20T13:16:49.875 INFO:journalctl@ceph.osd.1.smithi068.stdout:Feb 20 13:16:49 smithi068 bash[21685]:  5: clone()
Actions
Copy link

Also available in: Atom PDF