Project

General

Profile

Actions
Copy link

Bug #52079

closed

bluefs mount failed to replay log: (5) Input/output error

Added by Viktor Svecov almost 3 years ago. Updated over 2 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
pacific, octopus
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v16.2.1
ceph-qa-suite:
Pull request ID:
42830
Crash signature (v1):
Crash signature (v2):

Description

In the testlab after simultaneous power off of all OSD nodes (3) two of them can not start.

h2 node:

...
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluefs _replay 0xb5000: stop: failed to decode: bad crc 1492738775 expected 0: Malformed input
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluefs mount failed to replay log: (5) Input/output error
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_bluefs failed bluefs mount: (5) Input/output error
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _open_db failed to prepare db environment:
debug 2021-08-06T04:49:14.966+0000 7f42e40d5080  1 bdev(0x5621b7fce400 /var/lib/ceph/osd/ceph-1/block) close
debug 2021-08-06T04:49:15.226+0000 7f42e40d5080 -1 osd.1 0 OSD:init: unable to mount object store
debug 2021-08-06T04:49:15.226+0000 7f42e40d5080 -1  ** ERROR: osd init failed: (5) Input/output error@

h3 node:

...
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluefs _replay 0xc07000: stop: failed to decode: bad crc 3449997429 expected 0: Malformed input
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluefs mount failed to replay log: (5) Input/output error
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _open_bluefs failed bluefs mount: (5) Input/output error
debug 2021-08-06T04:38:55.526+0000 7f139f28e080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _open_db failed to prepare db environment:
debug 2021-08-06T04:38:55.526+0000 7f139f28e080  1 bdev(0x55898df4e400 /var/lib/ceph/osd/ceph-2/block) close
debug 2021-08-06T04:38:55.686+0000 7f139f28e080 -1 osd.2 0 OSD:init: unable to mount object store
debug 2021-08-06T04:38:55.686+0000 7f139f28e080 -1  ** ERROR: osd init failed: (5) Input/output error

Full log files attached.

ceph-bluestore-tool and ceph-objectstore-tool outputs the same error messages.

I am not sure if this problem related to existing expecially BUG #50965.

Files

Download all files
bluefs mount failed to replay log (5) on h2.txt (11.6 KB) bluefs mount failed to replay log (5) on h2.txt Viktor Svecov, 08/06/2021 05:09 AM
bluefs mount failed to replay log (5) on h3.txt (11.6 KB) bluefs mount failed to replay log (5) on h3.txt Viktor Svecov, 08/06/2021 05:09 AM
debug_bluefs_20.7z (59.6 KB) debug_bluefs_20.7z Viktor Svecov, 08/14/2021 05:46 AM
h3_debug_bluefs_20_2.7z (55.3 KB) h3_debug_bluefs_20_2.7z Viktor Svecov, 08/16/2021 04:57 AM

Related issues 2 (0 open — 2 closed)

Copied to bluestore - Backport #52492: pacific: bluefs mount failed to replay log: (5) Input/output errorResolvedActions
Copied to bluestore - Backport #52493: octopus: bluefs mount failed to replay log: (5) Input/output errorResolvedActions
Actions
Copy link

Also available in: Atom PDF