Actions
Bug #52079
closedbluefs mount failed to replay log: (5) Input/output error
Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
pacific, octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
In the testlab after simultaneous power off of all OSD nodes (3) two of them can not start.
h2 node:
...
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluefs _replay 0xb5000: stop: failed to decode: bad crc 1492738775 expected 0: Malformed input
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluefs mount failed to replay log: (5) Input/output error
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_bluefs failed bluefs mount: (5) Input/output error
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_db failed to prepare db environment:
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 1 bdev(0x5621b7fce400 /var/lib/ceph/osd/ceph-1/block) close
debug 2021-08-06T04:49:15.226+0000 7f42e40d5080 -1 osd.1 0 OSD:init: unable to mount object store
debug 2021-08-06T04:49:15.226+0000 7f42e40d5080 -1 ** ERROR: osd init failed: (5) Input/output error@
h3 node:
...
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluefs _replay 0xc07000: stop: failed to decode: bad crc 3449997429 expected 0: Malformed input
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluefs mount failed to replay log: (5) Input/output error
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _open_bluefs failed bluefs mount: (5) Input/output error
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db failed to prepare db environment:
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 1 bdev(0x55898df4e400 /var/lib/ceph/osd/ceph-2/block) close
debug 2021-08-06T04:38:55.686+0000 7f139f28e080 -1 osd.2 0 OSD:init: unable to mount object store
debug 2021-08-06T04:38:55.686+0000 7f139f28e080 -1 ** ERROR: osd init failed: (5) Input/output error
Full log files attached.
ceph-bluestore-tool and ceph-objectstore-tool outputs the same error messages.
I am not sure if this problem related to existing expecially BUG #50965.
Files
Actions