Project

General

Profile

Bug #55049

Can't start multiple OSD

Added by Karol Murawski about 2 years ago. Updated almost 2 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-disk, fs
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Problem occurred after multiple osd crash.
kernel: 5.15.27-1-pve

2022-03-24T17:25:41.901+0100 7efc49fc2d80 20 bluefs _read left 0x385d len 0x20000
2022-03-24T17:25:41.901+0100 7efc49fc2d80 20 bluefs _read fetching 0x23000~1d000 of 1:0x29eb2750000~a10000
2022-03-24T17:25:41.901+0100 7efc49fc2d80 10 bluefs read dev 1: 0x29eb2773000~1d000
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read left 0x1d000 len 0x1c7a3
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read got 131072
2022-03-24T17:25:41.905+0100 7efc49fc2d80 10 bluefs _read h 0x55cbb49a5800 0x2a7ff~40000 from file(ino 25618 size 0x41b6629 mtime 2022-03-17T14:37:18.652125+0100 allocated 41c0000 extents [1:0x29eb2750000~a10000,1:0x29f57080000~1300000,1:0x29f3ed50000~1300000,1:0x29ecac40000~11b0000]) prefetch
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read left 0x15801 len 0x40000
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read fetching 0x40000~2b000 of 1:0x29eb2750000~a10000
2022-03-24T17:25:41.905+0100 7efc49fc2d80 10 bluefs read dev 1: 0x29eb2790000~2b000
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read left 0x2b000 len 0x2a7ff
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read got 262144
2022-03-24T17:25:41.905+0100 7efc49fc2d80 10 bluefs _read h 0x55cbb49a5200 0x1f6f~2000 from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000]) prefetch
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read left 0x2091 len 0x2000
2022-03-24T17:25:41.905+0100 7efc49fc2d80 20 bluefs _read got 8192
2022-03-24T17:25:41.905+0100 7efc49fc2d80 10 bluefs _read_random h 0x55cbb49a5200 0x1f6f~fc5 from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000])
2022-03-24T17:25:41.909+0100 7efc49fc2d80 20 bluefs _read_random left 0x2091 0x1f6f~fc5
2022-03-24T17:25:41.909+0100 7efc49fc2d80 20 bluefs _read_random got 4037
2022-03-24T17:25:41.909+0100 7efc49fc2d80 10 bluefs _read_random h 0x55cbb49a5200 0x2f34~fa0 from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000])
2022-03-24T17:25:41.909+0100 7efc49fc2d80 20 bluefs _read_random left 0x10cc 0x2f34~fa0
2022-03-24T17:25:41.909+0100 7efc49fc2d80 20 bluefs _read_random got 4000
2022-03-24T17:25:41.909+0100 7efc49fc2d80  2 osd.117 0 journal looks like hdd
2022-03-24T17:25:41.909+0100 7efc49fc2d80  2 osd.117 0 boot
2022-03-24T17:25:41.909+0100 7efc49fc2d80 20 osd.117 0 configured osd_max_object_name[space]_len looks ok
2022-03-24T17:25:41.913+0100 7efc49fc2d80 10 bluefs _read_random h 0x55cbb49a5200 0x315ca39~cecca from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000])
2022-03-24T17:25:41.913+0100 7efc49fc2d80 20 bluefs _read_random read random 0x14ca39~cecca of 1:0x29f2dc00000~220000
2022-03-24T17:25:41.913+0100 7efc49fc2d80 10 bluefs read_random dev 1: 0x29f2dd4ca39~cecca
2022-03-24T17:25:41.921+0100 7efc49fc2d80 20 bluefs _read_random got 847050
2022-03-24T17:25:41.921+0100 7efc49fc2d80 10 bluefs _read_random h 0x55cbb49a5200 0x322b703~dd97f from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000])
2022-03-24T17:25:41.921+0100 7efc49fc2d80 20 bluefs _read_random read random 0x21b703~48fd of 1:0x29f2dc00000~220000
2022-03-24T17:25:41.921+0100 7efc49fc2d80 10 bluefs read_random dev 1: 0x29f2de1b703~48fd
2022-03-24T17:25:41.921+0100 7efc49fc2d80 20 bluefs _read_random read random 0x0~d9082 of 1:0x29f265a0000~e0000
2022-03-24T17:25:41.921+0100 7efc49fc2d80 10 bluefs read_random dev 1: 0x29f265a0000~d9082
2022-03-24T17:25:41.925+0100 7efc49fc2d80 20 bluefs _read_random got 907647
2022-03-24T17:25:41.929+0100 7efc49fc2d80 10 bluefs _read_random h 0x55cbb49a5200 0x59e555~fdc from file(ino 32953 size 0x33093da mtime 2022-03-22T21:02:39.241808+0100 allocated 3310000 extents [1:0x29eda510000~110000,1:0x29eefe80000~100000,1:0x29f1a3a0000~100000,1:0x29ec1f40000~100000,1:0x29f3e8d0000~100000,1:0x29f46b90000~100000,1:0x29f60950000~100000,1:0x29ecbff0000~100000,1:0x29f50df0000~100000,1:0x29f5e920000~100000,1:0x29f97c90000~100000,1:0x29f264a0000~100000,1:0x29ebfb50000~200000,1:0x29f58700000~200000,1:0x29f3b240000~200000,1:0x29f58380000~200000,1:0x29f0d400000~200000,1:0x29f02250000~200000,1:0x29ef8330000~400000,1:0x29f22be0000~400000,1:0x29eeb8a0000~400000,1:0x29ecd0e0000~400000,1:0x29eecfd0000~400000,1:0x29ebcf30000~400000,1:0x29f2dc00000~220000,1:0x29f265a0000~e0000])
2022-03-24T17:25:41.929+0100 7efc49fc2d80 20 bluefs _read_random read random 0x8e555~fdc of 1:0x29f46b90000~100000
2022-03-24T17:25:41.929+0100 7efc49fc2d80 10 bluefs read_random dev 1: 0x29f46c1e555~fdc
2022-03-24T17:25:41.929+0100 7efc3e4cb700 10 rocksdb: commit_cache_size old: 13421772 new: 67108864
2022-03-24T17:25:41.929+0100 7efc3e4cb700 10 rocksdb: commit_cache_size High Pri Pool Ratio set to 0.1
2022-03-24T17:25:41.929+0100 7efc49fc2d80 20 bluefs _read_random got 4060
2022-03-24T17:25:41.933+0100 7efc49fc2d80 -1 bluestore(/var/lib/ceph/osd/ceph-117) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xc7287920, expected 0x9e518d37, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0#
2022-03-24T17:25:41.933+0100 7efc49fc2d80 -1 bluestore(/var/lib/ceph/osd/ceph-117) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xc7287920, expected 0x9e518d37, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0#
2022-03-24T17:25:41.933+0100 7efc49fc2d80 -1 bluestore(/var/lib/ceph/osd/ceph-117) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xc7287920, expected 0x9e518d37, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0#
2022-03-24T17:25:41.933+0100 7efc49fc2d80 -1 bluestore(/var/lib/ceph/osd/ceph-117) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0xc7287920, expected 0x9e518d37, device location [0x10000~1000], logical extent 0x0~1000, object #-1:7b3f43c4:::osd_superblock:0#
2022-03-24T17:25:41.933+0100 7efc49fc2d80 -1 osd.117 0 OSD::init() : unable to read osd superblock
2022-03-24T17:25:41.933+0100 7efc49fc2d80  1 bluestore(/var/lib/ceph/osd/ceph-117) umount
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  allocation stats probe 1: cnt: 0 frags: 0 size: 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  probe -1: 0,  0, 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  probe -3: 0,  0, 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  probe -5: 0,  0, 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  probe -9: 0,  0, 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117)  probe -17: 0,  0, 0
2022-03-24T17:25:41.941+0100 7efc3e4cb700  0 bluestore(/var/lib/ceph/osd/ceph-117) ------------
2022-03-24T17:25:41.941+0100 7efc49fc2d80  4 rocksdb: [db/db_impl.cc:389] Shutdown: canceling all background work
2022-03-24T17:25:41.941+0100 7efc49fc2d80 10 bluefs readdir db
2022-03-24T17:25:41.945+0100 7efc49fc2d80 10 bluefs readdir db.slow
2022-03-24T17:25:41.945+0100 7efc49fc2d80 10 bluefs readdir db.wal
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579ba40 ignoring, length 0 < min_flush_size 524288
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579ba40 ignoring, length 0 < min_flush_size 524288
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579ba40 ignoring, length 0 < min_flush_size 524288
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _fsync 0x55cbb579ba40 file(ino 33058 size 0x0 mtime 2022-03-24T17:25:41.806970+0100 allocated 0 extents [])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579ba40 no dirty data on file(ino 33058 size 0x0 mtime 2022-03-24T17:25:41.806970+0100 allocated 0 extents [])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _claim_completed_aios got 0 aios
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579ba40
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579ba40 done in 0.000001
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs flush_bdev
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _close_writer 0x55cbb579ba40 type 1
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579b180 ignoring, length 0 < min_flush_size 524288
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579b180 ignoring, length 0 < min_flush_size 524288
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _fsync 0x55cbb579b180 file(ino 33056 size 0x1d8b mtime 2022-03-24T17:25:41.805455+0100 allocated 10000 extents [1:0x29f26680000~10000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579b180 no dirty data on file(ino 33056 size 0x1d8b mtime 2022-03-24T17:25:41.805455+0100 allocated 10000 extents [1:0x29f26680000~10000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _claim_completed_aios got 0 aios
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579b180
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579b180 done in 0.000001
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs flush_bdev
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _truncate 0x1d8b file file(ino 33056 size 0x1d8b mtime 2022-03-24T17:25:41.805455+0100 allocated 10000 extents [1:0x29f26680000~10000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _close_writer 0x55cbb579b180 type 0
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs unlock_file 0x55cbb4a1b748 on file(ino 2 size 0x0 mtime 2021-07-06T02:37:34.777592+0200 allocated 0 extents [])
2022-03-24T17:25:41.949+0100 7efc49fc2d80  4 rocksdb: [db/db_impl.cc:563] Shutdown complete
2022-03-24T17:25:41.949+0100 7efc49fc2d80  1 bluefs umount
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs sync_metadata
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs flush_bdev
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush_and_sync_log txn(seq 1923760 len 0x62 crc 0x38217df4)
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush 0x55cbb579aa80 0xae6000~1000 to file(ino 1 size 0xae6000 mtime 0.000000 allocated c10000 extents [1:0x29ed42e0000~10000,1:0x29f0a7f0000~400000,1:0x29eaeeb0000~400000,1:0x29ed3ee0000~400000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _flush_range 0x55cbb579aa80 pos 0xae6000 0xae6000~1000 to file(ino 1 size 0xae6000 mtime 0.000000 allocated c10000 extents [1:0x29ed42e0000~10000,1:0x29f0a7f0000~400000,1:0x29eaeeb0000~400000,1:0x29ed3ee0000~400000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs _flush_range file now, unflushed file(ino 1 size 0xae7000 mtime 0.000000 allocated c10000 extents [1:0x29ed42e0000~10000,1:0x29f0a7f0000~400000,1:0x29eaeeb0000~400000,1:0x29ed3ee0000~400000])
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs _flush_range in 1:0x29ed3ee0000~400000 x_off 0x2d6000
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs _flush_range h 0x55cbb579aa80 pos now 0xae7000
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _claim_completed_aios got 1 aios
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579aa80
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs wait_for_aio 0x55cbb579aa80 done in 0.000237
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs flush_bdev
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs _flush_and_sync_log log_seq_stable 1923760
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs sync_metadata done in 0.000350
2022-03-24T17:25:41.949+0100 7efc49fc2d80 10 bluefs _close_writer 0x55cbb579aa80 type 0
2022-03-24T17:25:41.949+0100 7efc49fc2d80 20 bluefs _stop_alloc
2022-03-24T17:25:41.949+0100 7efc49fc2d80  1 bdev(0x55cbb564e380 /var/lib/ceph/osd/ceph-117/block) close
2022-03-24T17:25:42.401+0100 7efc49fc2d80  1 fbmap_alloc 0x55cbb4973600 shutdown
2022-03-24T17:25:42.401+0100 7efc49fc2d80  1 freelist shutdown
2022-03-24T17:25:42.401+0100 7efc49fc2d80  1 bdev(0x55cbb564e000 /var/lib/ceph/osd/ceph-117/block) close
2022-03-24T17:25:42.469+0100 7efc49fc2d80 -1  ** ERROR: osd init failed: (22) Invalid argument
2022-03-24T17:24:22.774+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.774+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2914194~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.774+0100 7f9df8aa8d80 20 bluefs _read_random left 0x37e6c 0x2914194~fe0
2022-03-24T17:24:22.774+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2915174~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x36e8c 0x2915174~fe1
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2916155~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x35eab 0x2916155~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2917135~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x34ecb 0x2917135~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2918115~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x33eeb 0x2918115~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x29190f5~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x32f0b 0x29190f5~fe1
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291a0d6~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x31f2a 0x291a0d6~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291b0b6~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x30f4a 0x291b0b6~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291c096~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2ff6a 0x291c096~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291d076~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2ef8a 0x291d076~fe1
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291e057~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2dfa9 0x291e057~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x291f037~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2cfc9 0x291f037~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2920017~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2bfe9 0x2920017~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2920ff7~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2b009 0x2920ff7~fe1
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2921fd8~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random left 0x2a028 0x2921fd8~fe0
2022-03-24T17:24:22.778+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2922fb8~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x29048 0x2922fb8~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2923f98~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x28068 0x2923f98~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2924f78~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x27088 0x2924f78~fe1
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2925f59~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x260a7 0x2925f59~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2926f39~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x250c7 0x2926f39~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2927f19~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x240e7 0x2927f19~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2928ef9~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x23107 0x2928ef9~fe1
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x2929eda~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x22126 0x2929eda~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x292aeba~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x21146 0x292aeba~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x292be9a~fe0 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x20166 0x292be9a~fe0
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4064
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 10 bluefs _read_random h 0x56477c797480 0x292ce7a~fe1 from file(ino 29193 size 0x2b77c3f mtime 2022-03-20T07:20:15.380517+0100 allocated 2b80000 extents [1:0x29f02420000~2b10000,1:0x29f49940000~70000])
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random left 0x1f186 0x292ce7a~fe1
2022-03-24T17:24:22.782+0100 7f9df8aa8d80 20 bluefs _read_random got 4065
2022-03-24T17:24:22.798+0100 7f9df8aa8d80  1 bluestore(/var/lib/ceph/osd/ceph-116) _fsck_on_open <<<FINISH>>> with 20 errors, 0 warnings, 0 repaired, 20 remaining in 107.871781 seconds
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  allocation stats probe 0: cnt: 0 frags: 0 size: 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  probe -1: 0,  0, 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  probe -2: 0,  0, 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  probe -4: 0,  0, 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  probe -8: 0,  0, 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116)  probe -16: 0,  0, 0
2022-03-24T17:24:22.798+0100 7f9de67a4700  0 bluestore(/var/lib/ceph/osd/ceph-116) ------------
2022-03-24T17:24:22.798+0100 7f9df8aa8d80  4 rocksdb: [db/db_impl.cc:389] Shutdown: canceling all background work
2022-03-24T17:24:22.802+0100 7f9df8aa8d80  4 rocksdb: [db/db_impl.cc:563] Shutdown complete
2022-03-24T17:24:22.802+0100 7f9df8aa8d80  1 bluefs umount
2022-03-24T17:24:22.802+0100 7f9df8aa8d80 10 bluefs sync_metadata - no pending log events
2022-03-24T17:24:22.802+0100 7f9df8aa8d80 10 bluefs _close_writer 0x56477c4b2a80 type 0
2022-03-24T17:24:22.802+0100 7f9df8aa8d80 20 bluefs _stop_alloc
2022-03-24T17:24:22.802+0100 7f9df8aa8d80  1 bdev(0x56477c384380 /var/lib/ceph/osd/ceph-116/block) close
2022-03-24T17:24:23.086+0100 7f9df8aa8d80  1 fbmap_alloc 0x56477b686f00 shutdown
2022-03-24T17:24:23.086+0100 7f9df8aa8d80  1 freelist shutdown
2022-03-24T17:24:23.086+0100 7f9df8aa8d80  1 bdev(0x56477c384000 /var/lib/ceph/osd/ceph-116/block) close
2022-03-24T17:24:23.170+0100 7f9df8aa8d80 -1 bluestore(/var/lib/ceph/osd/ceph-116) _mount fsck found 20 errors
2022-03-24T17:24:23.170+0100 7f9df8aa8d80 -1 osd.116 0 OSD:init: unable to mount object store
2022-03-24T17:24:23.170+0100 7f9df8aa8d80 -1  ** ERROR: osd init failed: (5) Input/output error
2022-03-24T18:15:47.213+0100 7f37ad260d80  0 set uid:gid to 64045:64045 (ceph:ceph)
2022-03-24T18:15:47.213+0100 7f37ad260d80  0 ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable), process ceph-osd, pid 2694795
2022-03-24T18:15:47.213+0100 7f37ad260d80  0 pidfile_write: ignore empty --pid-file
2022-03-24T18:15:47.213+0100 7f37ad260d80  5 object store type is bluestore
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev create path /var/lib/ceph/osd/ceph-80/block type kernel
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open path /var/lib/ceph/osd/ceph-80/block
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open size 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bluestore(/var/lib/ceph/osd/ceph-80) _set_cache_sizes cache_size 33554432 meta 0.4 kv 0.4 data 0.2
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev create path /var/lib/ceph/osd/ceph-80/block type kernel
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev(0x55d830a34380 /var/lib/ceph/osd/ceph-80/block) open path /var/lib/ceph/osd/ceph-80/block
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev(0x55d830a34380 /var/lib/ceph/osd/ceph-80/block) open size 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-80/block size 5.5 TiB
2022-03-24T18:15:47.213+0100 7f37ad260d80  1 bdev(0x55d830a34380 /var/lib/ceph/osd/ceph-80/block) close
2022-03-24T18:15:47.513+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) close
2022-03-24T18:15:47.757+0100 7f37ad260d80  0 starting osd.80 osd_data /var/lib/ceph/osd/ceph-80 /var/lib/ceph/osd/ceph-80/journal
2022-03-24T18:15:47.773+0100 7f37ad260d80  0 load: jerasure load: lrc load: isa
2022-03-24T18:15:47.773+0100 7f37ad260d80  1 bdev create path /var/lib/ceph/osd/ceph-80/block type kernel
2022-03-24T18:15:47.773+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open path /var/lib/ceph/osd/ceph-80/block
2022-03-24T18:15:47.773+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open size 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-03-24T18:15:47.773+0100 7f37ad260d80  1 bluestore(/var/lib/ceph/osd/ceph-80) _set_cache_sizes cache_size 33554432 meta 0.4 kv 0.4 data 0.2
2022-03-24T18:15:47.773+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) close
2022-03-24T18:15:48.069+0100 7f37ad260d80  0 osd.80:0.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-03-24T18:15:48.069+0100 7f37ad260d80  0 osd.80:1.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-03-24T18:15:48.069+0100 7f37ad260d80  0 osd.80:2.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-03-24T18:15:48.069+0100 7f37ad260d80  0 osd.80:3.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-03-24T18:15:48.069+0100 7f37ad260d80  0 osd.80:4.OSDShard using op scheduler ClassedOpQueueScheduler(queue=WeightedPriorityQueue, cutoff=196)
2022-03-24T18:15:48.069+0100 7f37ad260d80  2 osd.80 0 init /var/lib/ceph/osd/ceph-80 (looks like hdd)
2022-03-24T18:15:48.069+0100 7f37ad260d80  2 osd.80 0 journal /var/lib/ceph/osd/ceph-80/journal
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bluestore(/var/lib/ceph/osd/ceph-80) _mount path /var/lib/ceph/osd/ceph-80
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bluestore(/var/lib/ceph/osd/ceph-80) _fsck check (regular)
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bdev create path /var/lib/ceph/osd/ceph-80/block type kernel
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open path /var/lib/ceph/osd/ceph-80/block
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) open size 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bluestore(/var/lib/ceph/osd/ceph-80) _set_cache_sizes cache_size 33554432 meta 0.4 kv 0.4 data 0.2
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bdev create path /var/lib/ceph/osd/ceph-80/block type kernel
2022-03-24T18:15:48.069+0100 7f37ad260d80  1 bdev(0x55d830a34380 /var/lib/ceph/osd/ceph-80/block) open path /var/lib/ceph/osd/ceph-80/block
2022-03-24T18:15:48.073+0100 7f37ad260d80  1 bdev(0x55d830a34380 /var/lib/ceph/osd/ceph-80/block) open size 6001172414464 (0x57541c00000, 5.5 TiB) block_size 4096 (4 KiB) rotational discard not supported
2022-03-24T18:15:48.073+0100 7f37ad260d80  1 bluefs add_block_device bdev 1 path /var/lib/ceph/osd/ceph-80/block size 5.5 TiB
2022-03-24T18:15:48.073+0100 7f37ad260d80  1 bluefs mount
2022-03-24T18:15:48.073+0100 7f37ad260d80  1 bluefs _init_alloc id 1 alloc_size 0x10000 size 0x57541c00000
2022-03-24T18:15:48.081+0100 7f37ad260d80 -1 bluefs _replay file with link count 0: file(ino 22817 size 0xd2085f mtime 2022-03-22T12:16:10.970459+0100 allocated d30000 extents [1:0x29ec3cd0000~10000,1:0x29f3ccb0000~80000,1:0x29eb6910000~10000,1:0x29eb0590000~40000,1:0x29ec3cc0000~10000,1:0x29edd3a0000~10000,1:0x29eec440000~10000,1:0x29ef6f00000~10000,1:0x29f2d550000~10000,1:0x29f3c4a0000~10000,1:0x29f3f610000~10000,1:0x29f8bcc0000~10000,1:0x29f263b0000~20000,1:0x29f799b0000~20000,1:0x29eb3ae0000~30000,1:0x29eb7a00000~30000,1:0x29ee0270000~30000,1:0x29ee6f30000~30000,1:0x29f49730000~30000,1:0x29f8aec0000~30000,1:0x29f1fad0000~40000,1:0x29f64480000~40000,1:0x29ee3270000~50000,1:0x29f01f20000~50000,1:0x29edad40000~60000,1:0x29f33bd0000~60000,1:0x29ec8bd0000~70000,1:0x29f25a40000~70000,1:0x29f6afe0000~a0000,1:0x29eed250000~b0000,1:0x29f01a50000~b0000,1:0x29ede7c0000~e0000,1:0x29f3f8d0000~f0000,1:0x29f8baf0000~d0000,1:0x29f3f620000~10000,1:0x29f3f640000~180000,1:0x29edfba0000~180000])
2022-03-24T18:15:48.085+0100 7f37ad260d80 -1 bluefs mount failed to replay log: (5) Input/output error
2022-03-24T18:15:48.085+0100 7f37ad260d80 -1 bluestore(/var/lib/ceph/osd/ceph-80) _open_bluefs failed bluefs mount: (5) Input/output error
2022-03-24T18:15:48.085+0100 7f37ad260d80  1 bdev(0x55d830a34000 /var/lib/ceph/osd/ceph-80/block) close
2022-03-24T18:15:48.361+0100 7f37ad260d80 -1 osd.80 0 OSD:init: unable to mount object store
2022-03-24T18:15:48.361+0100 7f37ad260d80 -1  ** ERROR: osd init failed: (5) Input/output error

mount-fail.txt View (57.8 KB) Karol Murawski, 03/26/2022 11:21 AM

History

#1 Updated by Karol Murawski about 2 years ago

HEALTH_WARN 1 filesystem is degraded; 2 osds down; 2 hosts (8 osds) down; 2 nearfull osd(s); Reduced data availability: 140 pgs inactive, 102 pgs down, 4 pgs incomplete, 90 pgs stale; Degraded data redundancy: 14917043/34948917 objects degraded (42.682%), 832 pgs degraded, 836 pgs undersized; 24 pgs not deep-scrubbed in time; 4 pool(s) nearfull; 87 daemons have recently crashed; 2 slow ops, oldest one blocked for 1223 sec, osd.112 has slow ops
[WRN] FS_DEGRADED: 1 filesystem is degraded
    fs cephfs is degraded
[WRN] OSD_DOWN: 2 osds down
    osd.109 (root=default,host=pve1-14) is down
    osd.117 (root=default,host=pve1-16) is down
[WRN] OSD_HOST_DOWN: 2 hosts (8 osds) down
    host pve1-14 (root=default) (4 osds) is down
    host pve1-16 (root=default) (4 osds) is down
[WRN] OSD_NEARFULL: 2 nearfull osd(s)
    osd.27 is near full
    osd.46 is near full
[WRN] PG_AVAILABILITY: Reduced data availability: 140 pgs inactive, 102 pgs down, 4 pgs incomplete, 90 pgs stale
    pg 1.2eb is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [112]
    pg 1.2f3 is stuck inactive for 2d, current state unknown, last acting []
    pg 1.2f9 is stuck stale for 47h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [133]
    pg 1.2fe is stuck inactive for 2d, current state unknown, last acting []
    pg 1.2ff is down, acting [60,106,44]
    pg 1.307 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [112,23]
    pg 1.308 is stuck inactive for 2d, current state unknown, last acting []
    pg 1.30f is down, acting [17,77]
    pg 1.31f is stuck inactive for 2d, current state unknown, last acting []
    pg 1.321 is down, acting [73,51,60]
    pg 1.327 is down, acting [131,76]
    pg 1.328 is down, acting [129,37,78]
    pg 1.32a is stuck inactive for 2d, current state unknown, last acting []
    pg 1.32c is stuck stale for 47h, current state stale+down, last acting [133]
    pg 1.32d is stuck stale for 47h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [133,8]
    pg 1.32e is incomplete, acting [105,75]
    pg 1.339 is stuck stale for 47h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [133]
    pg 1.343 is stuck stale for 26h, current state stale+active+undersized+degraded, last acting [79,90]
    pg 1.346 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79,42]
    pg 1.34a is stuck inactive for 2d, current state unknown, last acting []
    pg 1.34e is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79]
    pg 1.35b is down, acting [132,120,136]
    pg 1.363 is down, acting [44,131,86]
    pg 1.366 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [112]
    pg 1.36c is down, acting [136,77]
    pg 1.36d is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [112]
    pg 1.36f is down, acting [129,78,31]
    pg 1.379 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79]
    pg 1.37d is stuck stale for 47h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [133]
    pg 1.388 is stuck inactive for 2d, current state unknown, last acting []
    pg 1.391 is down, acting [27,130]
    pg 1.395 is incomplete, acting [87,120,40]
    pg 1.396 is stuck stale for 26h, current state stale+active+undersized+degraded, last acting [79,131]
    pg 1.398 is stuck stale for 47h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [133]
    pg 1.39b is down, acting [10,93,120]
    pg 1.39e is down, acting [74,87]
    pg 1.3a1 is down, acting [50,85,2]
    pg 1.3b1 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79,75]
    pg 1.3b2 is stuck stale for 26h, current state stale+active+clean, last acting [112,87,73]
    pg 1.3b9 is down, acting [132,90]
    pg 1.3c1 is stuck inactive for 2d, current state unknown, last acting []
    pg 1.3c2 is stuck stale for 26h, current state stale+active+clean, last acting [79,132,121]
    pg 1.3c8 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfilling, last acting [79]
    pg 1.3cd is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79,73]
    pg 1.3d1 is stuck stale for 47h, current state stale+active+undersized+degraded, last acting [133,106]
    pg 1.3d4 is stuck stale for 47h, current state stale+down, last acting [133]
    pg 1.3e9 is stuck stale for 26h, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79]
    pg 1.3f1 is stuck stale for 26h, current state stale+down, last acting [79,40,74]
    pg 1.3f2 is down, acting [66,85]
    pg 1.3f5 is down, acting [105,16]
    pg 1.3f6 is down, acting [105,133,7]
[WRN] PG_DEGRADED: Degraded data redundancy: 14917043/34948917 objects degraded (42.682%), 832 pgs degraded, 836 pgs undersized
    pg 1.3bf is active+undersized+degraded+remapped+backfilling, acting [46]
    pg 1.3c0 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [78]
    pg 1.3c3 is stuck undersized for 2d, current state active+undersized+degraded, last acting [45,59]
    pg 1.3c4 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [74]
    pg 1.3c5 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [73]
    pg 1.3c6 is stuck undersized for 2d, current state active+undersized+degraded, last acting [56,121]
    pg 1.3c7 is stuck undersized for 2d, current state active+undersized+degraded, last acting [105,90]
    pg 1.3c8 is stuck undersized for 2d, current state stale+active+undersized+degraded+remapped+backfilling, last acting [79]
    pg 1.3c9 is stuck undersized for 2d, current state active+undersized+degraded, last acting [78,61]
    pg 1.3ca is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [6,129]
    pg 1.3cb is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [75]
    pg 1.3cc is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [78]
    pg 1.3cd is stuck undersized for 2d, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79,73]
    pg 1.3ce is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [52]
    pg 1.3cf is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [85]
    pg 1.3d0 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [136,28]
    pg 1.3d1 is stuck undersized for 2d, current state stale+active+undersized+degraded, last acting [133,106]
    pg 1.3d2 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [78,130]
    pg 1.3d3 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [14]
    pg 1.3d5 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [45,27]
    pg 1.3d6 is stuck undersized for 2d, current state active+undersized+degraded, last acting [55,73]
    pg 1.3d7 is stuck undersized for 2d, current state active+undersized+degraded, last acting [29,112]
    pg 1.3d9 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [24,5]
    pg 1.3da is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [77,76]
    pg 1.3db is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [45,105]
    pg 1.3dc is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [41,75]
    pg 1.3dd is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [106]
    pg 1.3de is stuck undersized for 2d, current state active+undersized+degraded, last acting [86,106]
    pg 1.3df is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [77]
    pg 1.3e0 is stuck undersized for 2d, current state active+undersized+degraded, last acting [106,84]
    pg 1.3e1 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [73]
    pg 1.3e2 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [78,130]
    pg 1.3e3 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [90]
    pg 1.3e5 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [46]
    pg 1.3e6 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [136,120]
    pg 1.3e7 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [84,132]
    pg 1.3e8 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [130]
    pg 1.3e9 is stuck undersized for 2d, current state stale+active+undersized+degraded+remapped+backfill_wait, last acting [79]
    pg 1.3ea is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [66,76]
    pg 1.3ec is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [90,85]
    pg 1.3ed is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [50,75]
    pg 1.3ee is stuck undersized for 2d, current state active+undersized+degraded, last acting [77]
    pg 1.3f0 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [106]
    pg 1.3f3 is stuck undersized for 2d, current state active+undersized+degraded, last acting [75]
    pg 1.3f4 is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [32,85]
    pg 1.3f8 is stuck undersized for 2d, current state active+undersized+degraded, last acting [35,105]
    pg 1.3fa is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [33,75]
    pg 1.3fb is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfilling, last acting [38]
    pg 1.3fc is stuck undersized for 2d, current state active+undersized+degraded, last acting [74,42]
    pg 1.3fd is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [74]
    pg 1.3fe is stuck undersized for 2d, current state active+undersized+degraded+remapped+backfill_wait, last acting [71]
[WRN] PG_NOT_DEEP_SCRUBBED: 24 pgs not deep-scrubbed in time
    pg 1.3ce not deep-scrubbed since 2022-03-12T06:48:46.887120+0100
    pg 1.3b8 not deep-scrubbed since 2022-03-12T09:14:55.010776+0100
    pg 1.3b6 not deep-scrubbed since 2022-03-12T10:38:57.113172+0100
    pg 1.3b1 not deep-scrubbed since 2022-03-12T01:50:57.020809+0100
    pg 1.34d not deep-scrubbed since 2022-03-12T12:20:58.338572+0100
    pg 1.324 not deep-scrubbed since 2022-03-12T10:59:49.938077+0100
    pg 1.30c not deep-scrubbed since 2022-03-12T03:46:22.259459+0100
    pg 1.2ea not deep-scrubbed since 2022-03-12T03:47:59.842766+0100
    pg 1.2e1 not deep-scrubbed since 2022-03-12T12:08:06.187218+0100
    pg 1.2de not deep-scrubbed since 2022-03-11T22:50:04.665438+0100
    pg 1.2ac not deep-scrubbed since 2022-03-11T23:37:33.072248+0100
    pg 1.234 not deep-scrubbed since 2022-03-12T09:17:33.382834+0100
    pg 1.233 not deep-scrubbed since 2022-03-12T10:12:37.738661+0100
    pg 1.b1 not deep-scrubbed since 2022-03-12T09:51:58.137368+0100
    pg 5.b not deep-scrubbed since 2022-03-12T04:33:05.928065+0100
    pg 1.6b not deep-scrubbed since 2022-03-12T11:31:41.021794+0100
    pg 1.124 not deep-scrubbed since 2022-03-12T06:42:40.733719+0100
    pg 1.13b not deep-scrubbed since 2022-03-12T09:11:47.176684+0100
    pg 1.15b not deep-scrubbed since 2022-03-11T22:47:04.747883+0100
    pg 1.173 not deep-scrubbed since 2022-03-12T12:13:06.434862+0100
    pg 1.195 not deep-scrubbed since 2022-03-12T12:13:48.608588+0100
    pg 1.1be not deep-scrubbed since 2022-03-12T11:03:03.971295+0100
    pg 1.1da not deep-scrubbed since 2022-03-12T07:28:53.498812+0100
    pg 1.1df not deep-scrubbed since 2022-03-12T07:04:21.912327+0100
[WRN] POOL_NEARFULL: 4 pool(s) nearfull
    pool 'ceph-vm' is nearfull
    pool 'device_health_metrics' is nearfull
    pool 'cephfs_data' is nearfull
    pool 'cephfs_metadata' is nearfull
[WRN] RECENT_CRASH: 87 daemons have recently crashed
    osd.133 crashed on host pve1-20 at 2022-03-20T01:36:35.552832Z
    osd.131 crashed on host pve1-20 at 2022-03-20T01:36:59.735630Z
    osd.132 crashed on host pve1-20 at 2022-03-20T01:38:07.615568Z
    osd.132 crashed on host pve1-20 at 2022-03-20T01:38:19.426817Z
    osd.129 crashed on host pve1-19 at 2022-03-20T01:52:16.227723Z
    osd.112 crashed on host pve1-19 at 2022-03-20T01:52:16.279567Z
    osd.112 crashed on host pve1-19 at 2022-03-20T01:52:44.441392Z
    osd.112 crashed on host pve1-19 at 2022-03-20T01:52:56.476468Z
    osd.130 crashed on host pve1-20 at 2022-03-20T02:01:11.197624Z
    osd.131 crashed on host pve1-20 at 2022-03-20T02:07:09.519778Z
    osd.131 crashed on host pve1-20 at 2022-03-20T02:07:20.931583Z
    osd.133 crashed on host pve1-20 at 2022-03-20T02:07:22.925567Z
    osd.131 crashed on host pve1-20 at 2022-03-20T02:07:32.311876Z
    osd.133 crashed on host pve1-20 at 2022-03-20T02:07:48.047323Z
    osd.130 crashed on host pve1-20 at 2022-03-20T02:08:04.664581Z
    osd.131 crashed on host pve1-20 at 2022-03-20T03:16:35.368838Z
    osd.131 crashed on host pve1-20 at 2022-03-20T03:16:46.579830Z
    osd.131 crashed on host pve1-20 at 2022-03-20T03:16:57.778490Z
    osd.90 crashed on host pve1-19 at 2022-03-20T04:12:14.237437Z
    osd.131 crashed on host pve1-20 at 2022-03-20T04:16:51.081891Z
    osd.131 crashed on host pve1-20 at 2022-03-20T04:17:34.813316Z
    osd.131 crashed on host pve1-20 at 2022-03-20T04:17:46.646229Z
    osd.112 crashed on host pve1-19 at 2022-03-20T04:28:01.693454Z
    osd.93 crashed on host pve1-19 at 2022-03-20T04:28:01.705094Z
    osd.93 crashed on host pve1-19 at 2022-03-20T04:28:15.762905Z
    osd.112 crashed on host pve1-19 at 2022-03-20T04:28:42.485572Z
    osd.130 crashed on host pve1-20 at 2022-03-20T04:40:48.780968Z
    osd.112 crashed on host pve1-19 at 2022-03-20T05:00:47.376508Z
    osd.90 crashed on host pve1-19 at 2022-03-20T05:00:52.898968Z
    osd.112 crashed on host pve1-19 at 2022-03-20T05:01:02.201987Z
    and 57 more
[WRN] SLOW_OPS: 2 slow ops, oldest one blocked for 1223 sec, osd.112 has slow ops

#2 Updated by Karol Murawski about 2 years ago

ID   CLASS  WEIGHT     TYPE NAME         STATUS  REWEIGHT  PRI-AFF
 -1         410.22058  root default
 -3          15.45456      host pve1-1
  0    hdd    0.90919          osd.0         up   1.00000  1.00000
  1    hdd    0.90919          osd.1         up   1.00000  1.00000
  2    hdd    0.90919          osd.2         up   1.00000  1.00000
  3    hdd    0.90919          osd.3         up   1.00000  1.00000
  4    hdd    0.90919          osd.4         up   1.00000  1.00000
  5    hdd    0.90919          osd.5         up   1.00000  1.00000
  6    hdd    0.90919          osd.6         up   1.00000  1.00000
  7    hdd    0.90919          osd.7         up   1.00000  1.00000
 32    hdd    0.81810          osd.32        up   1.00000  1.00000
 39    hdd    0.81810          osd.39        up   1.00000  1.00000
 40    hdd    0.81810          osd.40        up   1.00000  1.00000
 44    hdd    0.81810          osd.44        up   1.00000  1.00000
 48    hdd    0.81810          osd.48        up   1.00000  1.00000
 49    hdd    0.81810          osd.49        up   1.00000  1.00000
 58    hdd    0.81810          osd.58        up   1.00000  1.00000
 59    hdd    0.81810          osd.59        up   1.00000  1.00000
 68    hdd    0.81810          osd.68        up   1.00000  1.00000
 69    hdd    0.81810          osd.69        up   1.00000  1.00000
-21          21.83197      host pve1-10
 70    hdd    5.45799          osd.70      down         0  1.00000
 92    hdd    5.45799          osd.92      down         0  1.00000
 94    hdd    5.45799          osd.94      down         0  1.00000
 95    hdd    5.45799          osd.95      down         0  1.00000
-23          21.83197      host pve1-11
 96    hdd    5.45799          osd.96      down         0  1.00000
 97    hdd    5.45799          osd.97      down         0  1.00000
 98    hdd    5.45799          osd.98      down         0  1.00000
 99    hdd    5.45799          osd.99      down         0  1.00000
-25          21.83197      host pve1-12
100    hdd    5.45799          osd.100       up   1.00000  1.00000
101    hdd    5.45799          osd.101     down         0  1.00000
102    hdd    5.45799          osd.102     down         0  1.00000
103    hdd    5.45799          osd.103       up   1.00000  1.00000
-27          21.83197      host pve1-13
104    hdd    5.45799          osd.104     down         0  1.00000
105    hdd    5.45799          osd.105       up   1.00000  1.00000
106    hdd    5.45799          osd.106       up   1.00000  1.00000
107    hdd    5.45799          osd.107     down         0  1.00000
-29          21.83197      host pve1-14
108    hdd    5.45799          osd.108     down         0  1.00000
109    hdd    5.45799          osd.109     down   1.00000  1.00000
110    hdd    5.45799          osd.110     down         0  1.00000
111    hdd    5.45799          osd.111     down         0  1.00000
-31          21.83197      host pve1-15
 88    hdd    5.45799          osd.88      down         0  1.00000
113    hdd    5.45799          osd.113     down         0  1.00000
114    hdd    5.45799          osd.114     down         0  1.00000
115    hdd    5.45799          osd.115     down         0  1.00000
-33          21.83197      host pve1-16
116    hdd    5.45799          osd.116     down         0  1.00000
117    hdd    5.45799          osd.117     down   1.00000  1.00000
118    hdd    5.45799          osd.118     down         0  1.00000
119    hdd    5.45799          osd.119     down         0  1.00000
-35          21.83197      host pve1-17
120    hdd    5.45799          osd.120       up   1.00000  1.00000
121    hdd    5.45799          osd.121       up   1.00000  1.00000
122    hdd    5.45799          osd.122     down         0  1.00000
123    hdd    5.45799          osd.123     down         0  1.00000
-37          21.83197      host pve1-18
124    hdd    5.45799          osd.124     down         0  1.00000
125    hdd    5.45799          osd.125     down         0  1.00000
126    hdd    5.45799          osd.126     down         0  1.00000
136    hdd    5.45799          osd.136       up   1.00000  1.00000
-39          21.83197      host pve1-19
 93    hdd    5.45799          osd.93        up   1.00000  1.00000
129    hdd    5.45799          osd.129       up   1.00000  1.00000
130    hdd    5.45799          osd.130       up   1.00000  1.00000
132    hdd    5.45799          osd.132       up   1.00000  1.00000
 -5          15.36346      host pve1-2
 11    hdd    0.90919          osd.11        up   1.00000  1.00000
 14    hdd    0.90919          osd.14        up   1.00000  1.00000
 17    hdd    0.90919          osd.17        up   1.00000  1.00000
 20    hdd    0.90919          osd.20        up   1.00000  1.00000
 23    hdd    0.90919          osd.23        up   1.00000  1.00000
 28    hdd    0.90919          osd.28        up   1.00000  1.00000
 29    hdd    0.90919          osd.29        up   1.00000  1.00000
 33    hdd    0.81810          osd.33        up   1.00000  1.00000
 36    hdd    0.81810          osd.36        up   1.00000  1.00000
 43    hdd    0.81810          osd.43        up   1.00000  1.00000
 47    hdd    0.81810          osd.47        up   1.00000  1.00000
 51    hdd    0.81810          osd.51        up   1.00000  1.00000
 54    hdd    0.81810          osd.54        up   1.00000  1.00000
 57    hdd    0.81810          osd.57        up   1.00000  1.00000
 61    hdd    0.81810          osd.61        up   1.00000  1.00000
 64    hdd    0.81810          osd.64        up   1.00000  1.00000
 71    hdd    0.81810          osd.71        up   1.00000  1.00000
 72    hdd    0.81810          osd.72        up   1.00000  1.00000
-41          21.83197      host pve1-20
 90    hdd    5.45799          osd.90        up   1.00000  1.00000
112    hdd    5.45799          osd.112       up   1.00000  1.00000
131    hdd    5.45799          osd.131       up   1.00000  1.00000
133    hdd    5.45799          osd.133       up   1.00000  1.00000
 -7          13.81836      host pve1-3
  9    hdd    0.90919          osd.9         up   1.00000  1.00000
 12    hdd    0.90919          osd.12        up   1.00000  1.00000
 15    hdd    0.90919          osd.15        up   1.00000  1.00000
 18    hdd    0.90919          osd.18        up   1.00000  1.00000
 21    hdd    0.90919          osd.21        up   1.00000  1.00000
 24    hdd    0.90919          osd.24        up   1.00000  1.00000
 26    hdd    0.90919          osd.26        up   1.00000  1.00000
 27    hdd    0.90919          osd.27        up   1.00000  1.00000
 34    hdd    0.81810          osd.34        up   1.00000  1.00000
 37    hdd    0.81810          osd.37        up   1.00000  1.00000
 41    hdd    0.81810          osd.41        up   1.00000  1.00000
 46    hdd    0.81810          osd.46        up   1.00000  1.00000
 53    hdd    0.81810          osd.53        up   1.00000  1.00000
 56    hdd    0.81810          osd.56        up   1.00000  1.00000
 60    hdd    0.81810          osd.60        up   1.00000  1.00000
 66    hdd    0.81810          osd.66        up   1.00000  1.00000
 -9          16.27266      host pve1-4
 10    hdd    0.90919          osd.10        up   1.00000  1.00000
 13    hdd    0.90919          osd.13        up   1.00000  1.00000
 16    hdd    0.90919          osd.16        up   1.00000  1.00000
 19    hdd    0.90919          osd.19        up   1.00000  1.00000
 22    hdd    0.90919          osd.22        up   1.00000  1.00000
 25    hdd    0.90919          osd.25        up   1.00000  1.00000
 30    hdd    0.90919          osd.30        up   1.00000  1.00000
 31    hdd    0.90919          osd.31        up   1.00000  1.00000
 35    hdd    0.81810          osd.35        up   1.00000  1.00000
 38    hdd    0.81810          osd.38        up   1.00000  1.00000
 42    hdd    0.81810          osd.42        up   1.00000  1.00000
 45    hdd    0.81810          osd.45        up   1.00000  1.00000
 50    hdd    0.81810          osd.50        up   1.00000  1.00000
 52    hdd    0.81810          osd.52        up   1.00000  1.00000
 55    hdd    0.81810          osd.55        up   1.00000  1.00000
 62    hdd    0.81810          osd.62        up   1.00000  1.00000
 63    hdd    0.81810          osd.63        up   1.00000  1.00000
 65    hdd    0.81810          osd.65        up   1.00000  1.00000
 67    hdd    0.81810          osd.67        up   1.00000  1.00000
-11          21.83197      host pve1-5
 73    hdd    5.45799          osd.73        up   1.00000  1.00000
 74    hdd    5.45799          osd.74        up   1.00000  1.00000
 75    hdd    5.45799          osd.75        up   1.00000  1.00000
 76    hdd    5.45799          osd.76        up   1.00000  1.00000
-13          21.83197      host pve1-6
  8    hdd    5.45799          osd.8         up   1.00000  1.00000
 77    hdd    5.45799          osd.77        up   1.00000  1.00000
 78    hdd    5.45799          osd.78        up   1.00000  1.00000
 79    hdd    5.45799          osd.79        up   1.00000  1.00000
-15          21.83197      host pve1-7
 80    hdd    5.45799          osd.80      down         0  1.00000
 81    hdd    5.45799          osd.81      down         0  1.00000
 82    hdd    5.45799          osd.82      down         0  1.00000
 83    hdd    5.45799          osd.83        up   1.00000  1.00000
-17          21.83197      host pve1-8
 84    hdd    5.45799          osd.84        up   1.00000  1.00000
 85    hdd    5.45799          osd.85        up   1.00000  1.00000
 86    hdd    5.45799          osd.86        up   1.00000  1.00000
 87    hdd    5.45799          osd.87        up   1.00000  1.00000
-19          21.83197      host pve1-9
 89    hdd    5.45799          osd.89      down         0  1.00000
 91    hdd    5.45799          osd.91      down         0  1.00000
127    hdd    5.45799          osd.127     down         0  1.00000
128    hdd    5.45799          osd.128     down         0  1.00000

#3 Updated by Dan van der Ster about 2 years ago

Could you provide an example osd log, backtrace from the first crash?
Is there anything relevant in the kernel logs (e.g. IO errors in dmesg, journalctl -rk) ?

#4 Updated by Pawel Korzybski about 2 years ago

osd.log

    -9> 2022-03-20T03:07:07.166+0100 7f6f55a40700  5 prioritycache tune_memory target: 939524096 mapped: 549773312 unmapped: 145088512 heap: 694861824 old mem: 134217728 new mem: 134217728
    -8> 2022-03-20T03:07:08.174+0100 7f6f55a40700  5 prioritycache tune_memory target: 939524096 mapped: 553869312 unmapped: 140992512 heap: 694861824 old mem: 134217728 new mem: 134217728
    -7> 2022-03-20T03:07:09.178+0100 7f6f55a40700  5 prioritycache tune_memory target: 939524096 mapped: 553885696 unmapped: 140976128 heap: 694861824 old mem: 134217728 new mem: 134217728
    -6> 2022-03-20T03:07:09.178+0100 7f6f55a40700  5 bluestore.MempoolThread(0x5588e1830a98) _resize_shards cache_size: 134217728 kv_alloc: 67108864 kv_used: 65939760 meta_alloc: 67108864 meta_used: 10896915 data_alloc: 67108864 data_used: 0
    -5> 2022-03-20T03:07:09.486+0100 7f6f5261c700 -1 bluestore(/var/lib/ceph/osd/ceph-131) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8, device location [0x24c2c0000~1000], logical extent 0x0~1000, object #-1:ba296019:::osdmap.6551884:0#
    -4> 2022-03-20T03:07:09.494+0100 7f6f5261c700 -1 bluestore(/var/lib/ceph/osd/ceph-131) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8, device location [0x24c2c0000~1000], logical extent 0x0~1000, object #-1:ba296019:::osdmap.6551884:0#
    -3> 2022-03-20T03:07:09.494+0100 7f6f5261c700 -1 bluestore(/var/lib/ceph/osd/ceph-131) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8, device location [0x24c2c0000~1000], logical extent 0x0~1000, object #-1:ba296019:::osdmap.6551884:0#
    -2> 2022-03-20T03:07:09.494+0100 7f6f5261c700 -1 bluestore(/var/lib/ceph/osd/ceph-131) _verify_csum bad crc32c/0x1000 checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8, device location [0x24c2c0000~1000], logical extent 0x0~1000, object #-1:ba296019:::osdmap.6551884:0#
    -1> 2022-03-20T03:07:09.506+0100 7f6f5261c700 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 7f6f5261c700 time 2022-03-20T03:07:09.499097+0100
./src/osd/OSD.cc: 7913: FAILED ceph_assert(p != added_maps_bl.end())

 ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x5588d618006f]
 2: (()+0x9d01fa) [0x5588d61801fa]
 3: (OSD::handle_osd_map(MOSDMap*)+0x1461) [0x5588d623fb81]
 4: (OSD::ms_dispatch(Message*)+0x7c) [0x5588d6254b9c]
 5: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&)+0xc3) [0x5588d6bc61b3]
 6: (DispatchQueue::entry()+0x5ef) [0x5588d6bc4aff]
 7: (DispatchQueue::DispatchThread::entry()+0xd) [0x5588d6a0c41d]
 8: (()+0x8ea7) [0x7f6f61b83ea7]
 9: (clone()+0x3f) [0x7f6f6171cdef]

     0> 2022-03-20T03:07:09.514+0100 7f6f5261c700 -1 *** Caught signal (Aborted) **
 in thread 7f6f5261c700 thread_name:ms_dispatch

-1> 2022-03-20T03:07:09.506+0100 7f6f5261c700 -1 ./src/osd/OSD.cc: In function 'void OSD::handle_osd_map(MOSDMap*)' thread 7f6f5261c700 time 2022-03-20T03:07:09.499097+0100

some dmesg from one of servers wigh Nvidia cards and iommu

[Sun Mar 20 05:28:27 2022] DMAR: ERROR: DMA PTE for vPFN 0x63527 already set (to 63527003 not bab005803)
[Sun Mar 20 05:28:27 2022] ------------[ cut here ]------------
[Sun Mar 20 05:28:27 2022] WARNING: CPU: 19 PID: 118214 at drivers/iommu/intel/iommu.c:2405 __domain_mapping.cold+0x30/0x37
[Sun Mar 20 05:28:27 2022] Modules linked in: 8021q garp mrp rbd ceph libceph fscache netfs ebtable_filter ebtables ip_set ip6table_raw iptable_raw ip6table_filter ip6_tables sctp ip6_udp_tunnel udp_tunnel iptable_filter bpfilter bonding tls softdog nfnetlink_log nfnetlink ipmi_ssif intel_rapl_msr intel_rapl_common sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm crct10dif_pclmul ghash_clmulni_intel aesni_intel crypto_simd cryptd rapl intel_cstate pcspkr efi_pstore mgag200 drm_kms_helper cec rc_core i2c_algo_bit fb_sys_fops syscopyarea sysfillrect sysimgblt hpilo ioatdma dca acpi_ipmi ipmi_si ipmi_devintf acpi_tad ipmi_msghandler mac_hid vhost_net vhost vhost_iotlb tap vfio_pci vfio_virqfd irqbypass vfio_iommu_type1 vfio drm sunrpc ip_tables x_tables autofs4 zfs(PO) zunicode(PO) zzstd(O) zlua(O) zavl(PO) icp(PO) zcommon(PO) znvpair(PO) spl(O) btrfs blake2b_generic xor zstd_compress raid6_pq uas usb_storage crc32_pclmul xhci_pci i2c_i801 xhci_pci_renesas i2c_smbus uhci_hcd lpc_ich
[Sun Mar 20 05:28:27 2022]  ehci_pci ahci tg3 bnx2x xhci_hcd ehci_hcd libahci nvme mdio libcrc32c nvme_core wmi
[Sun Mar 20 05:28:27 2022] CPU: 19 PID: 118214 Comm: ceph-osd Tainted: P        W  O      5.13.19-3-pve #1
[Sun Mar 20 05:28:27 2022] Hardware name: HP ProLiant DL380 Gen9/ProLiant DL380 Gen9, BIOS P89 10/21/2019
[Sun Mar 20 05:28:27 2022] RIP: 0010:__domain_mapping.cold+0x30/0x37
[Sun Mar 20 05:28:27 2022] Code: 4c 89 fe 48 c7 c7 68 03 84 b3 4c 89 55 b8 e8 ae 5a fd ff 8b 05 b1 f8 28 01 4c 8b 55 b8 85 c0 74 09 83 e8 01 89 05 a0 f8 28 01 <0f> 0b e9 e3 9c ba ff 4c 89 d1 44 89 e2 48 c7 c6 a0 60 4c b3 48 c7
[Sun Mar 20 05:28:27 2022] RSP: 0018:ffffa957e0dd7408 EFLAGS: 00010046
[Sun Mar 20 05:28:27 2022] RAX: 0000000000000000 RBX: ffff8dc18c035938 RCX: ffff8dd0df2a09c8
[Sun Mar 20 05:28:27 2022] RDX: 0000000000000000 RSI: 0000000000000027 RDI: ffff8dd0df2a09c0
[Sun Mar 20 05:28:27 2022] RBP: ffffa957e0dd7480 R08: 0000000000000000 R09: ffffa957e0dd71f0
[Sun Mar 20 05:28:27 2022] R10: ffff8dc18c047080 R11: ffff8dd0fff2d0d8 R12: 0000000000000001
[Sun Mar 20 05:28:27 2022] R13: 0000000bab005803 R14: ffff8dc18c047080 R15: 0000000000063527
[Sun Mar 20 05:28:27 2022] FS:  00007f2ebed7fd80(0000) GS:ffff8dd0df280000(0000) knlGS:0000000000000000
[Sun Mar 20 05:28:27 2022] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sun Mar 20 05:28:27 2022] CR2: 000056509acc3120 CR3: 0000000734542001 CR4: 00000000003726e0
[Sun Mar 20 05:28:27 2022] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[Sun Mar 20 05:28:27 2022] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[Sun Mar 20 05:28:27 2022] Call Trace:
[Sun Mar 20 05:28:27 2022]  <TASK>
[Sun Mar 20 05:28:27 2022]  intel_iommu_map+0x74/0x90
[Sun Mar 20 05:28:27 2022]  __iommu_map+0xf7/0x1e0
[Sun Mar 20 05:28:27 2022]  __iommu_map_sg+0x8e/0x110
[Sun Mar 20 05:28:27 2022]  iommu_map_sg_atomic+0x14/0x20
[Sun Mar 20 05:28:27 2022]  iommu_dma_map_sg+0x333/0x4c0
[Sun Mar 20 05:28:27 2022]  ? ata_scsi_mode_select_xlat+0x4e0/0x4e0
[Sun Mar 20 05:28:27 2022]  dma_map_sg_attrs+0x4a/0x50
[Sun Mar 20 05:28:27 2022]  ata_qc_issue+0x14d/0x210
[Sun Mar 20 05:28:27 2022]  __ata_scsi_queuecmd+0x156/0x440
[Sun Mar 20 05:28:27 2022]  ata_scsi_queuecmd+0x45/0x90
[Sun Mar 20 05:28:27 2022]  scsi_queue_rq+0x393/0xb10
[Sun Mar 20 05:28:27 2022]  blk_mq_dispatch_rq_list+0x13c/0x7e0
[Sun Mar 20 05:28:27 2022]  ? __sbitmap_get_word+0x36/0x80
[Sun Mar 20 05:28:27 2022]  ? elv_rb_del+0x24/0x30
[Sun Mar 20 05:28:27 2022]  ? dd_dispatch_request+0x1/0x1e0
[Sun Mar 20 05:28:27 2022]  __blk_mq_do_dispatch_sched+0xba/0x2c0
[Sun Mar 20 05:28:27 2022]  ? blk_account_io_start+0x49/0x60
[Sun Mar 20 05:28:27 2022]  __blk_mq_sched_dispatch_requests+0x144/0x1a0
[Sun Mar 20 05:28:27 2022]  blk_mq_sched_dispatch_requests+0x35/0x60
[Sun Mar 20 05:28:27 2022]  __blk_mq_run_hw_queue+0x34/0x70
[Sun Mar 20 05:28:27 2022]  __blk_mq_delay_run_hw_queue+0x162/0x170
[Sun Mar 20 05:28:27 2022]  blk_mq_run_hw_queue+0x78/0xe0
[Sun Mar 20 05:28:27 2022]  blk_mq_sched_insert_requests+0x6e/0xf0
[Sun Mar 20 05:28:27 2022]  blk_mq_flush_plug_list+0x106/0x1b0
[Sun Mar 20 05:28:27 2022]  blk_flush_plug_list+0xdd/0x100
[Sun Mar 20 05:28:27 2022]  blk_finish_plug+0x29/0x40
[Sun Mar 20 05:28:27 2022]  read_pages+0x117/0x270
[Sun Mar 20 05:28:27 2022]  page_cache_ra_unbounded+0x146/0x1f0
[Sun Mar 20 05:28:27 2022]  force_page_cache_ra+0xe1/0x140
[Sun Mar 20 05:28:27 2022]  page_cache_sync_ra+0x40/0xe0
[Sun Mar 20 05:28:27 2022]  filemap_get_pages+0xda/0x5d0
[Sun Mar 20 05:28:27 2022]  filemap_read+0xb9/0x3c0
[Sun Mar 20 05:28:27 2022]  ? __do_sys_msync+0x110/0x2a0
[Sun Mar 20 05:28:27 2022]  ? get_page_from_freelist+0xa7c/0xd50
[Sun Mar 20 05:28:27 2022]  generic_file_read_iter+0xe2/0x140
[Sun Mar 20 05:28:27 2022]  blkdev_read_iter+0x4a/0x60
[Sun Mar 20 05:28:27 2022]  new_sync_read+0x10d/0x190
[Sun Mar 20 05:28:27 2022]  vfs_read+0x100/0x190
[Sun Mar 20 05:28:27 2022]  __x64_sys_pread64+0x93/0xc0
[Sun Mar 20 05:28:27 2022]  do_syscall_64+0x61/0xb0
[Sun Mar 20 05:28:27 2022]  ? handle_mm_fault+0xda/0x2c0
[Sun Mar 20 05:28:27 2022]  ? exit_to_user_mode_prepare+0x37/0x1b0
[Sun Mar 20 05:28:27 2022]  ? irqentry_exit_to_user_mode+0x9/0x20
[Sun Mar 20 05:28:27 2022]  ? irqentry_exit+0x19/0x30
[Sun Mar 20 05:28:27 2022]  ? exc_page_fault+0x8f/0x170
[Sun Mar 20 05:28:27 2022]  ? asm_exc_page_fault+0x8/0x30
[Sun Mar 20 05:28:27 2022]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[Sun Mar 20 05:28:27 2022] RIP: 0033:0x7f2ebf3d6917
[Sun Mar 20 05:28:27 2022] Code: 08 89 3c 24 48 89 4c 24 18 e8 05 f4 ff ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 04 24 e8 35 f4 ff ff 48 8b
[Sun Mar 20 05:28:27 2022] RSP: 002b:00007ffd02d59570 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
[Sun Mar 20 05:28:27 2022] RAX: ffffffffffffffda RBX: 0000565099e86b08 RCX: 00007f2ebf3d6917
[Sun Mar 20 05:28:27 2022] RDX: 0000000000100000 RSI: 000056509888a000 RDI: 000000000000002d
[Sun Mar 20 05:28:27 2022] RBP: 0000565098754380 R08: 0000000000000000 R09: 0000000000000020
[Sun Mar 20 05:28:27 2022] R10: 0000029ec7a30000 R11: 0000000000000293 R12: 0000000000100000
[Sun Mar 20 05:28:27 2022] R13: 0000029ec7a30000 R14: 0000565097a780c0 R15: 00007ffd02d59600
[Sun Mar 20 05:28:27 2022]  </TASK>
[Sun Mar 20 05:28:27 2022] ---[ end trace 41635a3823f2c0f5 ]---
[Sun Mar 20 05:28:27 2022] DMAR: ERROR: DMA PTE for vPFN 0x63528 already set (to 63528003 not bab006803)
[Sun Mar 20 05:28:27 2022] ------------[ cut here ]------------

[207749.540778] DMAR: DRHD: handling fault status reg 102
[207749.543614] DMAR: [DMA Write] Request device [04:00.0] PASID ffffffff fault addr 791f4000 [fault reason 05] PTE Write access is not set
[207754.541528] DMAR: DRHD: handling fault status reg 202
[207754.543842] DMAR: [DMA Write] Request device [04:00.0] PASID ffffffff fault addr 791f4000 [fault reason 05] PTE Write access is not set
04:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet [14e4:168e] (rev 10)

It's 20 node cluster
4x Dell R730xdE5-2640 (stable nodes)
16x HP DL380 gen9 E5-2690 v4 (failing nodes)

No IO errors.
It all started after upgrade, kernel version changed from 5.10 to 5.11, 5.13 and 5.15
Ceph was also upgraded 15.2.12 to 15.2.15

Now we are back on kernel 5.10, looks stable but damage has already been done.

#5 Updated by Dan van der Ster about 2 years ago

checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8

0x6706be76 is apparently the checksum of all zeroes, indicating indeed that this was a kernel or hardware issue. See https://tracker.ceph.com/issues/22464 and https://tracker.ceph.com/issues/22102
Do you understand from the timeline why this impacted so many OSDs all at the same time? Were the crashes soon after the kernel upgrade?

Now, are you able to fuse mount the disk and browse the objects? Something like `ceph-objecstore-tool /var/lib/osd/ceph-0/ --op fuse --mountpoint /mnt/` might be useful eventually if you need to replace some key corrupted objects. (E.g. the osdmap).
But in your first report even the superblock looks corrupted -- maybe you can't even mount.

Also, can you share an entire fsck log? (We don't see the errors themselves in your first message).

#6 Updated by Karol Murawski about 2 years ago

Few OSD more up: fsck -> repair -> OSDMap reimport

Maybe you know how to deal with:

2022-03-25T11:51:38.240+0100 7f6d6fd67b80 10 bluefs _replay 0x40f000: txn(seq 453972 len 0x3b4 crc 0xb807a71)
2022-03-25T11:51:38.240+0100 7f6d6fd67b80 10 bluefs _read h 0x55f6455bbc80 0x410000~1000 from file(ino 1 size 0x410000 mtime 0.000000 allocated 810000 extents [1:0x29ee0b30000~10000,1:0x29f60350000~400000,1:0x29f0c630000~400000])
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 10 bluefs _replay 0x410000: stop: seq 456021 != expected 453973
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 10 bluefs _replay log file size was 0x410000
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 -1 bluefs _replay file with link count 0: file(ino 1773 size 0x31767cc mtime 2022-03-22T11:58:47.579514+0100 allocated 3180000 extents [1:0x29f5da20000~10000,1:0x29eb0ff0000~10000,1:0x29ebee20000~40000,1:0x29ee4ef0000~10000,1:0x29f17b90000~10000,1:0x29f344a0000~10000,1:0x29f40470000~10000,1:0x29f51280000~10000,1:0x29f91ae0000~10000,1:0x29ecdb60000~80000,1:0x29ef7d80000~80000,1:0x29f3c1d0000~80000,1:0x29ec4e60000~20000,1:0x29ecd540000~20000,1:0x29ee1100000~10000,1:0x29ec00c0000~40000,1:0x29ee1110000~10000,1:0x29ee8400000~20000,1:0x29efc7d0000~20000,1:0x29f5da30000~10000,1:0x29ec9550000~30000,1:0x29f1c340000~40000,1:0x29f5f050000~40000,1:0x29f6f2e0000~30000,1:0x29f38ff0000~50000,1:0x29f6f310000~10000,1:0x29ebeab0000~70000,1:0x29f33930000~70000,1:0x29f37c80000~70000,1:0x29ebd600000~80000,1:0x29f192a0000~80000,1:0x29f4ab70000~80000,1:0x29ebd680000~20000,1:0x29f19320000~20000,1:0x29f4abf0000~20000,1:0x29f3bb50000~70000,1:0x29f4d3e0000~a0000,1:0x29f5e7e0000~a0000,1:0x29ee1e30000~b0000,1:0x29f602a0000~b0000,1:0x29f7cb60000~b0000,1:0x29ebf060000~20000,1:0x29ebf090000~90000,1:0x29f24260000~c0000,1:0x29f8c710000~c0000,1:0x29f660d0000~d0000,1:0x29f775c0000~d0000,1:0x29f89ab0000~f0000,1:0x29f94df0000~f0000,1:0x29fa88e0000~f0000,1:0x29ebffc0000~100000,1:0x29eca180000~100000,1:0x29f3c250000~110000,1:0x29f08640000~130000,1:0x29ebf520000~150000,1:0x29ebfb70000~150000,1:0x29f6b070000~190000,1:0x29f5ec80000~160000,1:0x29ed8ec0000~10000,1:0x29f5ede0000~60000,1:0x29ebec20000~80000,1:0x29ebf080000~10000,1:0x29ebeca0000~180000,1:0x29f05800000~200000,1:0x29f90ce0000~200000,1:0x29f5da60000~280000,1:0x29f8ff00000~2e0000,1:0x29edcb00000~300000,1:0x29ee09f0000~140000,1:0x29ee0b40000~1b0000,1:0x29ee4bf0000~300000,1:0x29f341a0000~e0000])
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 -1 bluefs mount failed to replay log: (5) Input/output error
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 -1 bluestore(/var/lib/ceph/osd/ceph-101) _open_bluefs failed bluefs mount: (5) Input/output error
2022-03-25T11:51:38.244+0100 7f6d6fd67b80 10 bluefs maybe_verify_layout no memorized_layout in bluefs superblock
2022-03-25T11:51:38.244+0100 7f6d6fd67b80  1 bdev(0x55f646278380 /var/lib/ceph/osd/ceph-101/block) close
fsck failed: (5) Input/output error

Check that mount also - there are files with pgs numbers '.head' . All '0' size and 1970 date.

#7 Updated by Karol Murawski about 2 years ago

Is there any option to bring up OSD from extracted RocksDB into dir? I can fix rocksdb but don't know how to import it or start from dir.

#8 Updated by Karol Murawski about 2 years ago

Another error:

  -227> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x804000~1000 from file(ino 1 size 0x804000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -226> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0xc000 len 0x1000
  -225> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -224> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x804000: txn(seq 312207 len 0x347 crc 0x50bb5d6d)
  -223> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _replay 0x804000:  op_file_update  file(ino 2117 size 0x4324a2b mtime 2022-03-22T12:00:33.398173+0100 allocated 4330000 extents [1:0x29ede820000~10000,1:0x29f2fad0000~10000,1:0x29f0e7e0000~50000,1:0x29f33380000~10000,1:0x29f1a7b0000~40000,1:0x29eaf2b0000~20000,1:0x29ef8f30000~30000,1:0x29f2ea50000~80000,1:0x29f4c430000~80000,1:0x29effba0000~70000,1:0x29f8b850000~80000,1:0x29ee3b30000~80000,1:0x29f26a80000~80000,1:0x29f2ead0000~20000,1:0x29f157f0000~80000,1:0x29f2a740000~80000,1:0x29edb290000~80000,1:0x29f2eaf0000~20000,1:0x29f4c4b0000~40000,1:0x29f8b8d0000~40000,1:0x29ee3bb0000~50000,1:0x29f60500000~50000,1:0x29f26b00000~20000,1:0x29f15870000~70000,1:0x29ed5db0000~30000,1:0x29f26b20000~40000,1:0x29f2a7c0000~70000,1:0x29edb310000~a0000,1:0x29f30300000~30000,1:0x29f609b0000~1a0000,1:0x29f60650000~250000,1:0x29eddcb0000~250000,1:0x29edb7c0000~20000,1:0x29eddf00000~20000,1:0x29f790f0000~270000,1:0x29f32c80000~2c0000,1:0x29f5cef0000~80000,1:0x29f32f40000~40000,1:0x29f5cf70000~60000,1:0x29f24e60000~50000,1:0x29f5cfd0000~270000,1:0x29ed5ee0000~140000,1:0x29efc660000~d0000,1:0x29efc740000~150000,1:0x29ed6020000~3b0000,1:0x29efc260000~3b0000,1:0x29f2ff00000~80000,1:0x29efc610000~50000,1:0x29f2ff80000~2c0000,1:0x29edb7b0000~10000,1:0x29f30240000~c0000,1:0x29eeccf0000~400000,1:0x29efc730000~10000,1:0x29f2b730000~20000,1:0x29ed59b0000~400000,1:0x29eec8f0000~400000,1:0x29f2af30000~400000,1:0x29edf130000~300000,1:0x29eed0f0000~10000,1:0x29f48a80000~90000,1:0x29edf430000~170000,1:0x29edb3b0000~400000,1:0x29f24a60000~220000])
  -222> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x805000~1000 from file(ino 1 size 0x805000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -221> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0xb000 len 0x1000
  -220> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -219> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x805000: txn(seq 312208 len 0x347 crc 0x2751ad37)
  -218> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _replay 0x805000:  op_file_update  file(ino 2117 size 0x43263d3 mtime 2022-03-22T12:00:33.446932+0100 allocated 4330000 extents [1:0x29ede820000~10000,1:0x29f2fad0000~10000,1:0x29f0e7e0000~50000,1:0x29f33380000~10000,1:0x29f1a7b0000~40000,1:0x29eaf2b0000~20000,1:0x29ef8f30000~30000,1:0x29f2ea50000~80000,1:0x29f4c430000~80000,1:0x29effba0000~70000,1:0x29f8b850000~80000,1:0x29ee3b30000~80000,1:0x29f26a80000~80000,1:0x29f2ead0000~20000,1:0x29f157f0000~80000,1:0x29f2a740000~80000,1:0x29edb290000~80000,1:0x29f2eaf0000~20000,1:0x29f4c4b0000~40000,1:0x29f8b8d0000~40000,1:0x29ee3bb0000~50000,1:0x29f60500000~50000,1:0x29f26b00000~20000,1:0x29f15870000~70000,1:0x29ed5db0000~30000,1:0x29f26b20000~40000,1:0x29f2a7c0000~70000,1:0x29edb310000~a0000,1:0x29f30300000~30000,1:0x29f609b0000~1a0000,1:0x29f60650000~250000,1:0x29eddcb0000~250000,1:0x29edb7c0000~20000,1:0x29eddf00000~20000,1:0x29f790f0000~270000,1:0x29f32c80000~2c0000,1:0x29f5cef0000~80000,1:0x29f32f40000~40000,1:0x29f5cf70000~60000,1:0x29f24e60000~50000,1:0x29f5cfd0000~270000,1:0x29ed5ee0000~140000,1:0x29efc660000~d0000,1:0x29efc740000~150000,1:0x29ed6020000~3b0000,1:0x29efc260000~3b0000,1:0x29f2ff00000~80000,1:0x29efc610000~50000,1:0x29f2ff80000~2c0000,1:0x29edb7b0000~10000,1:0x29f30240000~c0000,1:0x29eeccf0000~400000,1:0x29efc730000~10000,1:0x29f2b730000~20000,1:0x29ed59b0000~400000,1:0x29eec8f0000~400000,1:0x29f2af30000~400000,1:0x29edf130000~300000,1:0x29eed0f0000~10000,1:0x29f48a80000~90000,1:0x29edf430000~170000,1:0x29edb3b0000~400000,1:0x29f24a60000~220000])
  -217> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x806000~1000 from file(ino 1 size 0x806000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -216> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0xa000 len 0x1000
  -215> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -214> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x806000: txn(seq 312209 len 0x347 crc 0xcbb6d31f)
  -213> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _replay 0x806000:  op_file_update  file(ino 2117 size 0x43267f7 mtime 2022-03-22T12:00:33.549287+0100 allocated 4330000 extents [1:0x29ede820000~10000,1:0x29f2fad0000~10000,1:0x29f0e7e0000~50000,1:0x29f33380000~10000,1:0x29f1a7b0000~40000,1:0x29eaf2b0000~20000,1:0x29ef8f30000~30000,1:0x29f2ea50000~80000,1:0x29f4c430000~80000,1:0x29effba0000~70000,1:0x29f8b850000~80000,1:0x29ee3b30000~80000,1:0x29f26a80000~80000,1:0x29f2ead0000~20000,1:0x29f157f0000~80000,1:0x29f2a740000~80000,1:0x29edb290000~80000,1:0x29f2eaf0000~20000,1:0x29f4c4b0000~40000,1:0x29f8b8d0000~40000,1:0x29ee3bb0000~50000,1:0x29f60500000~50000,1:0x29f26b00000~20000,1:0x29f15870000~70000,1:0x29ed5db0000~30000,1:0x29f26b20000~40000,1:0x29f2a7c0000~70000,1:0x29edb310000~a0000,1:0x29f30300000~30000,1:0x29f609b0000~1a0000,1:0x29f60650000~250000,1:0x29eddcb0000~250000,1:0x29edb7c0000~20000,1:0x29eddf00000~20000,1:0x29f790f0000~270000,1:0x29f32c80000~2c0000,1:0x29f5cef0000~80000,1:0x29f32f40000~40000,1:0x29f5cf70000~60000,1:0x29f24e60000~50000,1:0x29f5cfd0000~270000,1:0x29ed5ee0000~140000,1:0x29efc660000~d0000,1:0x29efc740000~150000,1:0x29ed6020000~3b0000,1:0x29efc260000~3b0000,1:0x29f2ff00000~80000,1:0x29efc610000~50000,1:0x29f2ff80000~2c0000,1:0x29edb7b0000~10000,1:0x29f30240000~c0000,1:0x29eeccf0000~400000,1:0x29efc730000~10000,1:0x29f2b730000~20000,1:0x29ed59b0000~400000,1:0x29eec8f0000~400000,1:0x29f2af30000~400000,1:0x29edf130000~300000,1:0x29eed0f0000~10000,1:0x29f48a80000~90000,1:0x29edf430000~170000,1:0x29edb3b0000~400000,1:0x29f24a60000~220000])
  -212> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x807000~1000 from file(ino 1 size 0x807000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -211> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0x9000 len 0x1000
  -210> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -209> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x807000: txn(seq 312210 len 0x347 crc 0x48646517)
  -208> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _replay 0x807000:  op_file_update  file(ino 2117 size 0x4326df3 mtime 2022-03-22T12:00:33.550381+0100 allocated 4330000 extents [1:0x29ede820000~10000,1:0x29f2fad0000~10000,1:0x29f0e7e0000~50000,1:0x29f33380000~10000,1:0x29f1a7b0000~40000,1:0x29eaf2b0000~20000,1:0x29ef8f30000~30000,1:0x29f2ea50000~80000,1:0x29f4c430000~80000,1:0x29effba0000~70000,1:0x29f8b850000~80000,1:0x29ee3b30000~80000,1:0x29f26a80000~80000,1:0x29f2ead0000~20000,1:0x29f157f0000~80000,1:0x29f2a740000~80000,1:0x29edb290000~80000,1:0x29f2eaf0000~20000,1:0x29f4c4b0000~40000,1:0x29f8b8d0000~40000,1:0x29ee3bb0000~50000,1:0x29f60500000~50000,1:0x29f26b00000~20000,1:0x29f15870000~70000,1:0x29ed5db0000~30000,1:0x29f26b20000~40000,1:0x29f2a7c0000~70000,1:0x29edb310000~a0000,1:0x29f30300000~30000,1:0x29f609b0000~1a0000,1:0x29f60650000~250000,1:0x29eddcb0000~250000,1:0x29edb7c0000~20000,1:0x29eddf00000~20000,1:0x29f790f0000~270000,1:0x29f32c80000~2c0000,1:0x29f5cef0000~80000,1:0x29f32f40000~40000,1:0x29f5cf70000~60000,1:0x29f24e60000~50000,1:0x29f5cfd0000~270000,1:0x29ed5ee0000~140000,1:0x29efc660000~d0000,1:0x29efc740000~150000,1:0x29ed6020000~3b0000,1:0x29efc260000~3b0000,1:0x29f2ff00000~80000,1:0x29efc610000~50000,1:0x29f2ff80000~2c0000,1:0x29edb7b0000~10000,1:0x29f30240000~c0000,1:0x29eeccf0000~400000,1:0x29efc730000~10000,1:0x29f2b730000~20000,1:0x29ed59b0000~400000,1:0x29eec8f0000~400000,1:0x29f2af30000~400000,1:0x29edf130000~300000,1:0x29eed0f0000~10000,1:0x29f48a80000~90000,1:0x29edf430000~170000,1:0x29edb3b0000~400000,1:0x29f24a60000~220000])
  -207> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x808000~1000 from file(ino 1 size 0x808000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -206> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0x8000 len 0x1000
  -205> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -204> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x808000: txn(seq 312211 len 0x347 crc 0x2793a60e)
  -203> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _replay 0x808000:  op_file_update  file(ino 2117 size 0x4327217 mtime 2022-03-22T12:00:33.737527+0100 allocated 4330000 extents [1:0x29ede820000~10000,1:0x29f2fad0000~10000,1:0x29f0e7e0000~50000,1:0x29f33380000~10000,1:0x29f1a7b0000~40000,1:0x29eaf2b0000~20000,1:0x29ef8f30000~30000,1:0x29f2ea50000~80000,1:0x29f4c430000~80000,1:0x29effba0000~70000,1:0x29f8b850000~80000,1:0x29ee3b30000~80000,1:0x29f26a80000~80000,1:0x29f2ead0000~20000,1:0x29f157f0000~80000,1:0x29f2a740000~80000,1:0x29edb290000~80000,1:0x29f2eaf0000~20000,1:0x29f4c4b0000~40000,1:0x29f8b8d0000~40000,1:0x29ee3bb0000~50000,1:0x29f60500000~50000,1:0x29f26b00000~20000,1:0x29f15870000~70000,1:0x29ed5db0000~30000,1:0x29f26b20000~40000,1:0x29f2a7c0000~70000,1:0x29edb310000~a0000,1:0x29f30300000~30000,1:0x29f609b0000~1a0000,1:0x29f60650000~250000,1:0x29eddcb0000~250000,1:0x29edb7c0000~20000,1:0x29eddf00000~20000,1:0x29f790f0000~270000,1:0x29f32c80000~2c0000,1:0x29f5cef0000~80000,1:0x29f32f40000~40000,1:0x29f5cf70000~60000,1:0x29f24e60000~50000,1:0x29f5cfd0000~270000,1:0x29ed5ee0000~140000,1:0x29efc660000~d0000,1:0x29efc740000~150000,1:0x29ed6020000~3b0000,1:0x29efc260000~3b0000,1:0x29f2ff00000~80000,1:0x29efc610000~50000,1:0x29f2ff80000~2c0000,1:0x29edb7b0000~10000,1:0x29f30240000~c0000,1:0x29eeccf0000~400000,1:0x29efc730000~10000,1:0x29f2b730000~20000,1:0x29ed59b0000~400000,1:0x29eec8f0000~400000,1:0x29f2af30000~400000,1:0x29edf130000~300000,1:0x29eed0f0000~10000,1:0x29f48a80000~90000,1:0x29edf430000~170000,1:0x29edb3b0000~400000,1:0x29f24a60000~220000])
  -202> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _read h 0x55a7e1ec5a00 0x809000~1000 from file(ino 1 size 0x809000 mtime 0.000000 allocated c10000 extents [1:0x29f48a70000~10000,1:0x29f48670000~400000,1:0x29eded30000~400000,1:0x29f2b330000~400000])
  -201> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read left 0x7000 len 0x1000
  -200> 2022-03-25T16:55:51.996+0100 7f077e090d80 20 bluefs _read got 4096
  -199> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay 0x809000: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 67a4e321-cda4-4bf8-bc3b-6fb3da1637fe
  -198> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay log file size was 0x809000
  -197> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay block_unused_too_granular 0: []
  -196> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay block_unused_too_granular 1: []
  -195> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay block_unused_too_granular 2: []
  -194> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay block_unused_too_granular 3: []
  -193> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay block_unused_too_granular 4: []
  -192> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 bluefs _replay done
  -191> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24c10000 length 0x200000
  -190> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f48270000 length 0x3e0000
  -189> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec1810000 length 0x7c0000
  -188> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f30a30000 length 0x900000
  -187> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f66910000 length 0xa00000
  -186> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ebf230000 length 0x600000
  -185> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec1fd0000 length 0xf0000
  -184> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ebf830000 length 0x500000
  -183> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec28c0000 length 0xb00000
  -182> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f14ba0000 length 0x600000
  -181> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f31330000 length 0x90000
  -180> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f151a0000 length 0xf0000
  -179> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24c00000 length 0x10000
  -178> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24bf0000 length 0x10000
  -177> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eb1100000 length 0x41d0000
  -176> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f65bc0000 length 0xd10000
  -175> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f6a790000 length 0x3200000
  -174> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eb0e40000 length 0x2c0000
  -173> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed1d10000 length 0x110000
  -172> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ecf950000 length 0x100000
  -171> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edec30000 length 0x100000
  -170> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc160000 length 0x100000
  -169> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f00910000 length 0x100000
  -168> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f11c20000 length 0x100000
  -167> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f156f0000 length 0x100000
  -166> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f1c500000 length 0x100000
  -165> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f34d90000 length 0x100000
  -164> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f39ee0000 length 0x100000
  -163> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4cef0000 length 0x100000
  -162> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f78ff0000 length 0x100000
  -161> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f7d6e0000 length 0x100000
  -160> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f8fc20000 length 0x100000
  -159> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eed100000 length 0x200000
  -158> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2fbf0000 length 0x200000
  -157> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eddf20000 length 0x900000
  -156> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f47110000 length 0x1100000
  -155> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f48b10000 length 0x1200000
  -154> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb7e0000 length 0x3c0000
  -153> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4f6d0000 length 0x2210000
  -152> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f687e0000 length 0x1fb0000
  -151> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4e570000 length 0x1160000
  -150> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4b7f0000 length 0x40000
  -149> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f26b60000 length 0xd10000
  -148> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed1e20000 length 0x34c0000
  -147> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29fdb3e0000 length 0x41d0000
  -146> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f3bba0000 length 0x710000
  -145> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f673e0000 length 0x1400000
  -144> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f21720000 length 0x1700000
  -143> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ebfd40000 length 0xfc0000
  -142> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2f5c0000 length 0x510000
  -141> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f7eb10000 length 0xf00000
  -140> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f52cc0000 length 0x1000000
  -139> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f79360000 length 0x1100000
  -138> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f3aee0000 length 0xcc0000
  -137> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f1a7f0000 length 0xf10000
  -136> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f33390000 length 0x1a00000
  -135> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f313c0000 length 0x18c0000
  -134> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f62310000 length 0x210000
  -133> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f15de0000 length 0xd00000
  -132> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2dd50000 length 0xd00000
  -131> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2c150000 length 0xe00000
  -130> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5f700000 length 0xe00000
  -129> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2ec00000 length 0x9c0000
  -128> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed56a0000 length 0x310000
  -127> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f10650000 length 0x700000
  -126> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2a830000 length 0x700000
  -125> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec20c0000 length 0x800000
  -124> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eff3a0000 length 0x800000
  -123> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f19eb0000 length 0x900000
  -122> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ef4020000 length 0x900000
  -121> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eed300000 length 0xb00000
  -120> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f0e830000 length 0x3e0000
  -119> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f608a0000 length 0x110000
  -118> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ede830000 length 0x400000
  -117> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f32f80000 length 0x400000
  -116> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eaeeb0000 length 0x400000
  -115> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f8b910000 length 0x400000
  -114> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5d240000 length 0x400000
  -113> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ee3730000 length 0x400000
  -112> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efbb60000 length 0x500000
  -111> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ebed30000 length 0x500000
  -110> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eea780000 length 0x500000
  -109> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f1f2d0000 length 0x500000
  -108> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5c9f0000 length 0x500000
  -107> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f72520000 length 0x600000
  -106> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24460000 length 0x600000
  -105> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed52e0000 length 0x3c0000
  -104> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eaf2d0000 length 0x10000
  -103> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec0d00000 length 0xb10000
  -102> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ef7230000 length 0x1c00000
  -101> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24eb0000 length 0x1ac0000
  -100> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f26970000 length 0x110000
   -99> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f88f50000 length 0x2900000
   -98> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed63d0000 length 0x17c0000
   -97> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed7b90000 length 0x2410000
   -96> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec33e0000 length 0x1dc0000
   -95> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ec51a0000 length 0x2e10000
   -94> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eb90c0000 length 0x13a0000
   -93> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eba460000 length 0x3d10000
   -92> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eaf300000 length 0x490000
   -91> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eaf790000 length 0x16b0000
   -90> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eec8b0000 length 0x40000
   -89> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edbba0000 length 0x2110000
   -88> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f63b00000 length 0x20c0000
   -87> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f04230000 length 0x810000
   -86> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ef71e0000 length 0x50000
   -85> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eda070000 length 0x210000
   -84> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2cf50000 length 0xe00000
   -83> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc890000 length 0xe00000
   -82> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2eb10000 length 0xf0000
   -81> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efd690000 length 0x1d10000
   -80> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edf5a0000 length 0x5c0000
   -79> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29fcaca0000 length 0x41d0000
   -78> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2a630000 length 0x110000
   -77> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed5de0000 length 0x100000
   -76> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ee3c00000 length 0x100000
   -75> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ef8e30000 length 0x100000
   -74> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc060000 length 0x100000
   -73> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f00a10000 length 0x100000
   -72> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f48210000 length 0x60000
   -71> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29fc6ad0000 length 0x41d0000
   -70> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed9fa0000 length 0xb0000
   -69> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29fcee70000 length 0x41d0000
   -68> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f16ae0000 length 0x2a10000
   -67> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f41a80000 length 0x17c0000
   -66> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29fd3040000 length 0x41d0000
   -65> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ede820000 length 0x10000
   -64> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2fad0000 length 0x10000
   -63> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f0e7e0000 length 0x50000
   -62> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f33380000 length 0x10000
   -61> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f1a7b0000 length 0x40000
   -60> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eaf2b0000 length 0x20000
   -59> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ef8f30000 length 0x30000
   -58> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2ea50000 length 0x80000
   -57> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4c430000 length 0x80000
   -56> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29effba0000 length 0x70000
   -55> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f8b850000 length 0x80000
   -54> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ee3b30000 length 0x80000
   -53> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f26a80000 length 0x80000
   -52> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2ead0000 length 0x20000
   -51> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f157f0000 length 0x80000
   -50> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2a740000 length 0x80000
   -49> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb290000 length 0x80000
   -48> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2eaf0000 length 0x20000
   -47> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f4c4b0000 length 0x40000
   -46> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f8b8d0000 length 0x40000
   -45> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ee3bb0000 length 0x50000
   -44> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f60500000 length 0x50000
   -43> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f26b00000 length 0x20000
   -42> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f15870000 length 0x70000
   -41> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed5db0000 length 0x30000
   -40> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f26b20000 length 0x40000
   -39> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2a7c0000 length 0x70000
   -38> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb310000 length 0xa0000
   -37> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f30300000 length 0x30000
   -36> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f609b0000 length 0x1a0000
   -35> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f60650000 length 0x250000
   -34> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eddcb0000 length 0x250000
   -33> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb7c0000 length 0x20000
   -32> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eddf00000 length 0x20000
   -31> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f790f0000 length 0x270000
   -30> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f32c80000 length 0x2c0000
   -29> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5cef0000 length 0x80000
   -28> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f32f40000 length 0x40000
   -27> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5cf70000 length 0x60000
   -26> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24e60000 length 0x50000
   -25> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f5cfd0000 length 0x270000
   -24> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed5ee0000 length 0x140000
   -23> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc660000 length 0xd0000
   -22> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc740000 length 0x150000
   -21> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed6020000 length 0x3b0000
   -20> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc260000 length 0x3b0000
   -19> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2ff00000 length 0x80000
   -18> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc610000 length 0x50000
   -17> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2ff80000 length 0x2c0000
   -16> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb7b0000 length 0x10000
   -15> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f30240000 length 0xc0000
   -14> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eeccf0000 length 0x400000
   -13> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29efc730000 length 0x10000
   -12> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2b730000 length 0x20000
   -11> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29ed59b0000 length 0x400000
   -10> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eec8f0000 length 0x400000
    -9> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f2af30000 length 0x400000
    -8> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edf130000 length 0x300000
    -7> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29eed0f0000 length 0x10000
    -6> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f48a80000 length 0x90000
    -5> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edf430000 length 0x170000
    -4> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29edb3b0000 length 0x400000
    -3> 2022-03-25T16:55:51.996+0100 7f077e090d80 10 HybridAllocator init_rm_free offset 0x29f24a60000 length 0x220000
    -2> 2022-03-25T16:55:51.996+0100 7f077e090d80 -1 HybridAllocator init_rm_free lambdaUexpected extent:  0x29f24bf0000~90000
    -1> 2022-03-25T16:55:52.004+0100 7f077e090d80 -1 ./src/os/bluestore/HybridAllocator.cc: In function 'HybridAllocator::init_rm_free(uint64_t, uint64_t)::<lambda(uint64_t, uint64_t, bool)>' thread 7f077e090d80 time 2022-03-25T16:55:52.000469+0100
./src/os/bluestore/HybridAllocator.cc: 175: FAILED ceph_assert(false)

 ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x131) [0x55a7d788a06f]
 2: (()+0x9d01fa) [0x55a7d788a1fa]
 3: (()+0x10740b5) [0x55a7d7f2e0b5]
 4: (AvlAllocator::_try_remove_from_tree(unsigned long, unsigned long, std::function<void (unsigned long, unsigned long, bool)>)+0x276) [0x55a7d7f29ba6]
 5: (HybridAllocator::init_rm_free(unsigned long, unsigned long)+0xc3) [0x55a7d7f2ec13]
 6: (BlueFS::mount()+0x235) [0x55a7d7f0d015]
 7: (BlueStore::_open_bluefs(bool)+0x88) [0x55a7d7e05ae8]
 8: (BlueStore::_open_db(bool, bool, bool)+0x5cd) [0x55a7d7e0660d]
 9: (BlueStore::_open_db_and_around(bool)+0x4a) [0x55a7d7e1795a]
 10: (BlueStore::_mount(bool, bool)+0x56a) [0x55a7d7e5054a]
 11: (OSD::init()+0x58d) [0x55a7d795b3ad]
 12: (main()+0x3430) [0x55a7d78c4130]
 13: (__libc_start_main()+0xea) [0x7f077e19ed0a]
 14: (_start()+0x2a) [0x55a7d78d775a]

     0> 2022-03-25T16:55:52.032+0100 7f077e090d80 -1 *** Caught signal (Aborted) **
 in thread 7f077e090d80 thread_name:ceph-osd

 ceph version 15.2.15 (efc1e7ef13525f02d58d97ccd62fec325323c618) octopus (stable)
 1: (()+0x14140) [0x7f077e6e8140]
 2: (gsignal()+0x141) [0x7f077e1b3ce1]
 3: (abort()+0x123) [0x7f077e19d537]
 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x55a7d788a0b9]
 5: (()+0x9d01fa) [0x55a7d788a1fa]
 6: (()+0x10740b5) [0x55a7d7f2e0b5]
 7: (AvlAllocator::_try_remove_from_tree(unsigned long, unsigned long, std::function<void (unsigned long, unsigned long, bool)>)+0x276) [0x55a7d7f29ba6]
 8: (HybridAllocator::init_rm_free(unsigned long, unsigned long)+0xc3) [0x55a7d7f2ec13]
 9: (BlueFS::mount()+0x235) [0x55a7d7f0d015]
 10: (BlueStore::_open_bluefs(bool)+0x88) [0x55a7d7e05ae8]
 11: (BlueStore::_open_db(bool, bool, bool)+0x5cd) [0x55a7d7e0660d]
 12: (BlueStore::_open_db_and_around(bool)+0x4a) [0x55a7d7e1795a]
 13: (BlueStore::_mount(bool, bool)+0x56a) [0x55a7d7e5054a]
 14: (OSD::init()+0x58d) [0x55a7d795b3ad]
 15: (main()+0x3430) [0x55a7d78c4130]
 16: (__libc_start_main()+0xea) [0x7f077e19ed0a]
 17: (_start()+0x2a) [0x55a7d78d775a]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- logging levels ---
   0/ 5 none
   0/ 0 lockdep
   0/ 0 context
   0/ 0 crush
   1/ 5 mds
   1/ 5 mds_balancer
   1/ 5 mds_locker
   1/ 5 mds_log
   1/ 5 mds_log_expire
   1/ 5 mds_migrator
   0/ 0 buffer
   0/ 0 timer
   0/ 1 filer
   0/ 1 striper
   0/ 1 objecter
   0/ 5 rados
   0/ 5 rbd
   0/ 5 rbd_mirror
   0/ 5 rbd_replay
   0/ 5 rbd_rwl
   0/ 0 journaler
   0/ 5 objectcacher
   0/ 5 immutable_obj_cache
   0/ 5 client
  20/ 0 osd
   0/ 0 optracker
   0/ 0 objclass
   0/ 0 filestore
   0/ 0 journal
   0/ 0 ms
   1/ 5 mon
   0/ 0 monc
   1/ 5 paxos
   0/ 0 tp
   0/ 0 auth
   1/ 5 crypto
   0/ 0 finisher
   1/ 1 reserver
   0/ 0 heartbeatmap
   0/ 0 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   0/ 0 asok
   0/ 0 throttle
   0/ 0 refs
   1/ 5 compressor
  10/ 0 bluestore
  20/ 0 bluefs
   1/ 3 bdev
   1/ 5 kstore
  10/ 0 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
   1/ 5 prioritycache
   0/ 5 test
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
--- pthread ID / name mapping for recent threads ---
  7f077e090d80 / ceph-osd
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-osd.89.log
--- end dump of recent events ---

#9 Updated by Karol Murawski about 2 years ago

Dan van der Ster wrote:

checksum at blob offset 0x0, got 0x6706be76, expected 0xbe005cc8

0x6706be76 is apparently the checksum of all zeroes, indicating indeed that this was a kernel or hardware issue. See https://tracker.ceph.com/issues/22464 and https://tracker.ceph.com/issues/22102
Do you understand from the timeline why this impacted so many OSDs all at the same time? Were the crashes soon after the kernel upgrade?

Now, are you able to fuse mount the disk and browse the objects? Something like `ceph-objecstore-tool /var/lib/osd/ceph-0/ --op fuse --mountpoint /mnt/` might be useful eventually if you need to replace some key corrupted objects. (E.g. the osdmap).
But in your first report even the superblock looks corrupted -- maybe you can't even mount.

Also, can you share an entire fsck log? (We don't see the errors themselves in your first message).

Hi! Almost done with recovery. Left 12 OSD with error like in attach. Tried to mount it. There is any tool to fix it? Maybe dd with binary swap?

#10 Updated by Karol Murawski almost 2 years ago

Was able to access broken OSD with few "dummy" fix in BlueFS.cc . That was enough to bring OSD up with "custom" ceph-objectstore-tool, extract PGs and import them to working OSD. After 6 days of fighting all PG are active+clean.

Also available in: Atom PDF