Bug #54019
OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
0%
Description
When upgrading to rook 1.8.3 (ceph 16.2.7) we experience issue's with the OSD initialization; basically only +/- 50% of the raw devices are actually picked up by ceph, i.e. get an OSD instance. All others are eventually ignored.
It seems that the first osd-(prepare) run fails but leaves the osd setup in an invalid state from which ceph can't recover by simply respawning rook operator or osd pods.
In the osd-prepare logging it looks like this eventually:
2022-01-24 16:50:57.929115 D | exec: Running command: udevadm info --query=property /dev/sdc1
2022-01-24 16:50:57.938006 D | exec: Running command: lsblk /dev/sdc1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:57.941195 D | exec: Running command: ceph-volume inventory --format json /dev/sdc1
2022-01-24 16:50:58.423904 I | cephosd: skipping device "sdc1": ["Has BlueStore device label"].
2022-01-24 16:50:58.423934 D | exec: Running command: udevadm info --query=property /dev/sdd1
2022-01-24 16:50:58.432716 D | exec: Running command: lsblk /dev/sdd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.436077 D | exec: Running command: ceph-volume inventory --format json /dev/sdd1
2022-01-24 16:50:58.888559 I | cephosd: skipping device "sdd1": ["Has BlueStore device label"].
2022-01-24 16:50:58.888590 D | exec: Running command: udevadm info --query=property /dev/sde1
2022-01-24 16:50:58.897610 D | exec: Running command: lsblk /dev/sde1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.901015 D | exec: Running command: ceph-volume inventory --format json /dev/sde1
2022-01-24 16:50:59.363622 I | cephosd: skipping device "sde1": ["Has BlueStore device label"].
2022-01-24 16:50:59.363646 I | cephosd: skipping 'dm' device "dm-0"
2022-01-24 16:50:59.370400 I | cephosd: configuring osd devices: {"Entries":{}}
2022-01-24 16:50:59.370426 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2022-01-24 16:50:59.370718 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
2022-01-24 16:50:59.741959 D | cephosd: {}
2022-01-24 16:50:59.742004 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2022-01-24 16:50:59.742038 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2022-01-24 16:51:00.581917 D | cephosd: {
"0f6d36fb-23ed-4888-9469-a5ab91370bd3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sdc1",
"osd_id": 1,
"osd_uuid": "0f6d36fb-23ed-4888-9469-a5ab91370bd3",
"type": "bluestore"
},
"51b3b924-f70f-4257-88cf-91d881faf5e3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sde1",
"osd_id": 4,
"osd_uuid": "51b3b924-f70f-4257-88cf-91d881faf5e3",
"type": "bluestore"
}
}
As can be seen sdd1 is just skipped/missing.
Basically the same issue is reported in https://github.com/rook/rook/issues/8023 but closed with assumption that ceph 16.2.6 provides a fix for this issue.
After testing i can confirm 16.2.7 still has this issue.
It seems our setup somehow triggers this issue:
- centos7 (kernel 3.10)
- 3 spinning disks per host
- rook 1.8.3
I can also report that by setting "bluefs_buffered_io = false" the issue doesn't occur; although we don't fully understand today the further implications of setting this option.
Operator logging is attached:
- operator.log std 16.2.7 experiencing the issue
- operator-no-buffering.log: same but with option set; doesn't show the issue.
I can easily reproduce this issue and can run additional debug statements; but since i'm fairly new to ceph i do need some guidenance what/how to debug.
History
#2 Updated by Satoru Takeuchi 8 months ago
FYI, some rook users hit this problem as well.
quoted from https://tracker.ceph.com/issues/51034#note-12
---
https://github.com/rook/rook/issues/9646#issuecomment-1021505103
```
2022-01-25 18:38:32.269143 I | op-osd: OSD orchestration status for node dev2-cmp1l is "failed"
2022-01-25 18:38:32.269193 E | op-osd: failed to provision OSD on node dev2-cmp1l. &{OSDs:[] Status:failed PvcBackedOSD:false Message:failed to configure devices: failed to initialize osd: failed to run ceph-volume raw command. Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f4a0ae55-0b96-4304-a4cc-2f2f3d13a341
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/sde1
Running command: /usr/bin/ln -s /dev/sde1 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 3
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph
stderr: 2022-01-25T18:38:30.769+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2022-01-25T18:38:30.786+0000 7f89304df080 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 6f36f879-3c86-4ae1-8b46-abf13cc438c7, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db erroring opening db:
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 �[0;31m * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0/: (5) Input/output error�[0m
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 11, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in init
self.main(self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
self.safe_prepare(self.args)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
self.prepare()
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
tmpfs,
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 68, in prepare_bluestore
db=db
File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 481, in osd_mkfs_bluestore
raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph: exit status 1}
```
Ths problem disappeared if `bluefs_buffered_io=false` was set.
https://github.com/rook/rook/issues/9646#issuecomment-1022021781
```
And i can confirm that:
- rook 1.8.3 / 16.2.7 still have this issue operator.log
- rook 1.8.3 / 16.2.7 + bluefs_buffered_io = false does not experience this issue. operator-no-buffering.log
```
And here is another report.
https://github.com/rook/rook/issues/10160
```
stderr: 2022-04-25T18:12:29.836+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 2022-04-25T18:12:30.024+0000 7f023a2a43c0 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 68ba3e0d-e42c-46b0-a51b-e8e6ab918ad6, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _open_db erroring opening db:
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (5) Input/output error
--> Was unable to complete a new OSD, will rollback changes
```
The log is very similar to the first report. He used v17.20. But it's not clear that he tried `bluefs_buffered_io=false`. I suspect that `bluefs_buffered_id=true` (it's the default) still has some problems.
---
#3 Updated by Christian Rohmann 8 months ago
I observed similar issues at random when deploying Ceph 16.2.7 via Ansible on a VM.
ceph-osd -i 4 --mkfs --osd-objectstore=bluestore --osd-uuid de84f0aa-601a-473a-ad0b-7a2d403d0588 --monmap /tmp/monmap
return code is 250 and STDERR:
2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid
2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump:
00000000 40 6a a8 52 08 5a 8a 2a 74 c2 c5 98 f9 c6 c8 12 |@j.R.Z.*t.......|
00000010 5c 45 ca 37 bb 49 87 6d 98 f7 46 07 d6 3c 48 40 |\\E.7.I.m..F..<H@|
00000020 af b9 65 99 4f be 0b 20 9e 29 d5 9b 10 b1 74 ca |..e.O.. .)....t.|
00000030 82 77 b4 c5 ab de 04 9c eb 90 7c ff ce 76 ac ed |.w........|..v..|
00000040 3e 43 93 72 da 75 de 41 ed a9 49 5a 13 7a da 72 |>C.r.u.A..IZ.z.r|
00000050 7e 8d 1c d3 5b 31 98 7d 48 ed da 00 63 e7 f1 d5 |~...[1.}H...c...|
00000060 d1 5c f1 7f 2a c5 af 7d 56 2f fb 16 d7 26 56 ce |.\\..*..}V/...&V.|
00000070 ed ec 11 04 c6 b6 52 df 18 64 49 75 65 e8 62 47 |......R..dIue.bG|
00000080 29 1f de 22 e3 b2 45 0f ec fb 7c d9 f9 9a 5a d8 |)..\"..E...|...Z.|
00000090 82 40 71 6f c3 b6 73 5f e1 ed 3f 3d fd 56 8c e8 |.@qo..s_..?=.V..|
000000a0 a8 6a bf 6c f2 bc fa ca 44 8e e6 b5 6f f0 d0 bc |.j.l....D...o...|
000000b0 8f a3 ae 35 44 d2 23 74 6b 08 d6 75 9f 74 b0 fb |...5D.#tk..u.t..|
000000c0 8a f0 07 d1 7c b2 3e fd c3 e1 a5 7c cf d3 62 a6 |....|.>....|..b.|
000000d0 5b 04 c7 6d 7d 70 d9 72 cf 1b ee ef c6 59 41 4a |[..m}p.r.....YAJ|
000000e0 de 0d de 51 d1 02 cd ae be 3f 7d fd 45 92 f8 63 |...Q.....?}.E..c|
000000f0 6d 41 19 0f c7 d3 0e cf 64 2c 59 fc 86 0a c3 ec |mA......d,Y.....|
00000100 29 5d 6f 13 6b 62 fc 80 f4 b4 ce 9b 53 81 36 ea |)]o.kb......S.6.|
00000110 5e 07 e0 61 20 d2 15 43 2a 7a 84 f1 a3 0f 9b 8d |^..a ..C*z......|
00000120 ba 42 01 94 df f3 d4 87 fc 45 5c 8a dd d2 28 26 |.B.......E\\...(&|
00000130 a7 35 50 ee 58 1e f3 22 36 1d da 76 2c 07 c9 28 |.5P.X..\"6..v,..(|
00000140 e1 cb 66 20 cb e4 dd bb 91 eb b0 4f 66 1f 89 69 |..f .......Of..i|
00000150 4a ec 47 e3 d2 f6 2f 59 18 39 29 42 14 ea f6 19 |J.G.../Y.9)B....|
00000160 45 a0 e2 44 ed 97 4d 1f d2 f2 70 b0 13 70 e3 5b |E..D..M...p..p.[|
00000170 95 bb 43 6d 9c 35 25 17 14 2d a5 30 46 e3 38 e4 |..Cm.5%..-.0F.8.|
00000180 55 63 06 15 1c 1e d3 2e fd 7f 3e 4b 18 07 ed 53 |Uc........>K...S|
00000190 0b 7e d8 dd 39 0a d2 53 23 10 ce 7a 55 53 5e cf |.~..9..S#..zUS^.|
000001a0 77 6d a8 f8 ef e5 8a 01 13 b0 ac ce 31 07 c7 19 |wm..........1...|
000001b0 69 0e 00 c6 26 88 7a e0 be f5 2c ac 2c 6b ce 8a |i...&.z...,.,k..|
000001c0 b3 82 fe fc 16 ea 46 9e eb 7e 54 f4 b1 4d 00 dc |......F..~T..M..|
000001d0 0e e9 5b 3a 8b 5a 49 f0 46 f6 31 a8 9b 39 ba 96 |..[:.ZI.F.1..9..|
000001e0 b4 ce 76 69 91 5b 74 36 e9 89 cc 67 7c 2e c7 6b |..vi.[t6...g|..k|
000001f0 2e b2 f1 b0 fc a6 c6 c1 c9 35 b5 be 23 c1 b3 94 |.........5..#...|
00000200 11 47 12 1a ea 5e 47 84 a5 92 6c 83 e3 e2 ff 8a |.G...^G...l.....|
00000210 ed ba 48 f8 41 17 d5 2e d2 64 3a 64 64 21 ef 48 |..H.A....d:dd!.H|
00000220 f8 78 b3 2a ca 0b 1e 7d 80 0b fb 1c b5 73 1b ee |.x.*...}.....s..|
00000230 0f a0 4d 6b d9 22 72 06 1c 5d 27 af f6 c3 52 a8 |..Mk.\"r..]'...R.|
00000240 47 ac bf 18 50 4a 64 29 3c 71 1b 9d 21 37 a9 f1 |G...PJd)<q..!7..|
00000250 ea 52 7a 25 48 ef 8d 45 97 65 49 8b 1c a0 c5 1d |.Rz%H..E.eI.....|
00000260 94 da dc 90 b3 fe 56 2f 5e de 0a 02 58 6b 6f e6 |......V/^...Xko.|
00000270 e3 0e 93 eb 98 9a 4d 35 4f 8b 50 53 9f f6 35 d5 |......M5O.PS..5.|
00000280 2d c6 c6 45 d4 38 1d 68 7d b7 46 e0 8a 60 1b ec |-..E.8.h}.F..`..|
00000290 28 64 22 67 8b d4 0e 7c 05 28 24 59 fb b6 e5 44 |(d\"g...|.($Y...D|
000002a0 9e f4 d2 a5 08 cb 3a 52 a5 51 c3 56 cd fb 29 0b |......:R.Q.V..).|
000002b0 8d bb 28 ab 06 c5 e8 9c b8 99 ee 42 a5 4f 1c 0c |..(........B.O..|
000002c0 5f 29 1e 2f cf b2 f6 dc 05 21 b1 68 d2 79 ae b2 |_)./.....!.h.y..|
000002d0 d9 38 5f f6 5d 08 60 44 c1 6b d8 b1 a9 41 b7 48 |.8_.].`D.k...A.H|
000002e0 44 7a 4d f0 67 ef d9 ca 2f d5 69 9b c3 bf 78 ff |DzM.g.../.i...x.|
000002f0 2e f4 a4 94 14 0d fa 7b 60 55 fb 75 fb 49 52 5f |.......{`U.u.IR_|
00000300 fc 22 ad 16 ac c2 80 e1 ab cf bf 7f 84 6b df 0e |.\"...........k..|
00000310 4c 38 56 02 1a 37 bf 07 b1 e9 ac dc a8 56 f2 bf |L8V..7.......V..|
00000320 59 f8 8c 9b 58 88 54 2f bf d5 5e 1a c4 93 ea f3 |Y...X.T/..^.....|
00000330 ab 56 f0 a5 7c 66 7c 83 dd 00 30 6f 84 37 72 6b |.V..|f|...0o.7rk|
00000340 6a 13 d8 a8 e2 49 c7 2c e8 b8 c1 9f 29 3e e0 12 |j....I.,....)>..|
00000350 63 5f 22 df bd f7 9b 25 7e e9 c2 ff 1c 86 f0 e7 |c_\"....%~.......|
00000360 a9 59 06 18 9d fa 20 eb 2f 72 31 c4 53 c2 dd 09 |.Y.... ./r1.S...|
00000370 65 3c 9d 5d 7b 51 57 28 de 8b a5 e1 4a 1f 64 2e |e<.]{QW(....J.d.|
00000380 37 49 7f bd 15 57 09 dc fb 74 40 68 00 62 f8 5d |7I...W...t@h.b.]|
00000390 99 2c 6e c7 e6 64 bc 3f 26 46 90 f8 69 95 3b 48 |.,n..d.?&F..i.;H|
000003a0 a7 45 80 09 2d e4 5f c3 89 8e a1 e9 50 c6 87 22 |.E..-._.....P..\"|
000003b0 04 b5 c6 87 a8 1d 96 ff 65 bd 93 b0 69 62 8d b7 |........e...ib..|
000003c0 5b 9e e1 34 25 b6 7f 0e e6 2f 79 b2 ae 85 fc c8 |[..4%..../y.....|
000003d0 cd 4d dd b3 86 79 89 c8 86 d3 a9 a5 5d aa 9c 0c |.M...y......]...|
000003e0 a0 c3 e1 01 cb c8 30 81 46 35 f8 6f 81 4c d3 b5 |......0.F5.o.L..|
000003f0 be a8 e3 29 96 8c bb 1c 7c 1f fb e6 51 98 b6 90 |...)....|...Q...|
00000400 c0 a2 fc 9e bb b1 52 f1 b1 42 09 58 12 22 ec 16 |......R..B.X.\"..|
00000410 5f 81 49 ee 99 0a 6a 88 e9 76 60 4d ee 58 83 d3 |_.I...j..v`M.X..|
00000420 98 5b 55 ec 59 e8 ab 1a f6 d8 4a 9c 4b e5 56 66 |.[U.Y.....J.K.Vf|
00000430 b1 e1 fd 91 59 20 ce 42 db 6b 3d 7c 6a 7e 5e 05 |....Y .B.k=|j~^.|
00000440 49 a9 35 e3 28 25 08 bf 37 10 cb 60 27 ab c9 17 |I.5.(%..7..`'...|
00000450 2b 0e e0 54 2a 5a 53 df 80 e6 ca bf 7a de 55 54 |+..T*ZS.....z.UT|
00000460 71 f8 ca 9e fc 46 b3 c3 7f e1 ae a5 cd 6b 51 17 |q....F.......kQ.|
00000470 b3 48 fa 00 b9 7a 82 f1 16 02 ab 78 1a ad 99 5f |.H...z.....x..._|
00000480 8c c9 1d 53 dc 0e ff 12 b6 ba 6f 0e fe fb f2 31 |...S......o....1|
00000490 77 e9 f5 17 3d db ab 75 3f ce 76 48 57 37 e3 b4 |w...=..u?.vHW7..|
000004a0 b0 cc ae 79 f2 50 d4 55 fc 8b 1c aa 14 38 b8 a1 |...y.P.U.....8..|
000004b0 78 7b 20 72 67 03 25 c7 e7 15 ff a8 86 f9 ec f7 |x{ rg.%.........|
000004c0 3b 2c 56 20 87 75 53 a4 51 10 d7 1e b8 ea 6d d9 |;,V .uS.Q.....m.|
000004d0 78 18 98 b0 65 e7 a2 df fa 0b f2 79 cb c4 51 98 |x...e......y..Q.|
000004e0 be d0 eb c4 72 b4 a8 48 0f aa 6e 1b dc ff cb 1d |....r..H..n.....|
000004f0 aa 34 ee 55 57 19 87 3c 69 67 f3 63 b6 61 48 6d |.4.UW..<ig.c.aHm|
00000500 37 7d 18 32 6b 74 f1 59 c5 11 7d 13 e5 4c 21 81 |7}.2kt.Y..}..L!.|
00000510 5d 79 8a cb 9a 41 7a f9 9d 0e e7 ca 38 66 f4 1a |]y...Az.....8f..|
00000520 b7 a1 cd 7e 4d b1 19 f3 5d 53 a9 8d 9e 83 ae 0b |...~M...]S......|
00000530 c3 87 cd 4a 6e fe af a0 61 95 30 ee 99 ec 34 9a |...Jn...a.0...4.|
00000540 ec ac 19 ff fe 97 69 54 22 cc 04 17 62 a8 ee 7c |......iT\"...b..||
00000550 8f a4 a0 50 90 b9 fa e7 3f 5d 7a 2b f8 d1 83 8b |...P....?]z+....|
00000560 13 51 6c c4 a1 1c b1 5c de 83 a6 aa 43 d1 1d c1 |.Ql....\\....C...|
00000570 e9 e6 60 c0 f2 2c 35 42 b7 71 a4 00 3a b2 db 36 |..`..,5B.q..:..6|
00000580 73 40 12 90 ee 73 b6 d5 84 7b 4a c2 73 1e 2e af |s@...s...{J.s...|
00000590 33 53 82 67 89 10 cb 85 5a 96 d3 af 47 d9 47 07 |3S.g....Z...G.G.|
000005a0 18 e0 ba 7a aa 0c 69 08 d5 d2 60 a1 2a 38 8e 3e |...z..i...`.*8.>|
000005b0 dd de a1 d1 24 32 63 36 55 31 d4 7c 51 1e 83 ea |....$2c6U1.|Q...|
000005c0 4d f4 f0 ef 75 0b 89 fa 17 26 fa 81 67 53 36 57 |M...u....&..gS6W|
000005d0 87 6f ee 61 0d 65 d3 3c ba b1 04 0c ae 27 c3 7c |.o.a.e.<.....'.||
000005e0 19 ce 18 e3 cd 60 37 b4 b1 f7 f4 8b c2 fa 42 e5 |.....`7.......B.|
000005f0 a6 7f f2 2f 39 c9 94 62 93 68 64 5d 8b 1a 40 9e |.../9..b.hd]..@.|
00000600 ab 75 b3 e8 88 4f 5d ea a9 8e 4b fb c5 51 ca 9c |.u...O]...K..Q..|
00000610 49 d2 fc 68 18 4a 22 ce 7d 1d 00 e7 52 61 5f 85 |I..h.J\".}...Ra_.|
00000620 ec 3c 13 74 a5 78 75 3e 14 f2 27 ed b2 5d 92 16 |.<.t.xu>..'..]..|
00000630 05 39 dc ee 8c 13 4c af 63 86 1c 3f ac 7e 81 67 |.9....L.c..?.~.g|
00000640 b0 d7 a2 f3 9f 31 73 ae 97 08 11 2b fa 99 5e c9 |.....1s....+..^.|
00000650 d1 39 4a 9e a4 bb d1 55 83 c2 f4 2f bd 7e 85 c9 |.9J....U.../.~..|
00000660 d9 34 c6 61 c6 1d 12 9c d4 e9 ec 05 4f 04 55 3a |.4.a........O.U:|
00000670 78 2f fa 33 02 87 97 e1 65 ef af 8e f0 b2 f9 3c |x/.3....e......<|
00000680 75 ca 39 cc 12 2e 58 6e bd 50 9f 46 83 4f 02 3e |u.9...Xn.P.F.O.>|
00000690 50 8d 0a 8f 11 a2 06 70 70 0f 94 91 2d c7 da c4 |P......pp...-...|
000006a0 9a 22 eb 9f 14 04 c4 d0 35 c7 c9 60 9d b0 10 c3 |.\"......5..`....|
000006b0 ab fd c7 31 a4 b7 23 78 10 17 3b de 05 c8 f1 1b |...1..#x..;.....|
000006c0 90 32 54 ff eb bc 1f e4 11 78 f9 9c 2f ab 49 75 |.2T......x../.Iu|
000006d0 9f 42 89 54 77 9a 0b 0c a7 e8 b2 18 85 cd 2a d0 |.B.Tw.........*.|
000006e0 39 a0 09 96 09 2c a5 34 ef e9 85 0b d8 ed 22 e0 |9....,.4......\".|
000006f0 7c 80 07 8b 12 ed 60 b1 e8 ec 4e 26 97 f0 50 e1 ||.....`...N&..P.|
00000700 2f 2b 08 b8 03 87 7a da 44 7d 93 16 b5 70 c7 a2 |/+....z.D}...p..|
00000710 2f 5b 34 10 5d dc f6 f4 46 8c 8e 5c 4c aa db 44 |/[4.]...F..\\L..D|
00000720 ce fa ea 39 cb 23 e8 0c f4 13 99 e4 20 c7 9c 6a |...9.#...... ..j|
00000730 9e 37 b0 e0 42 af b3 59 98 6b fd 66 61 85 6e dc |.7..B..Y.k.fa.n.|
00000740 7c 53 b4 0b 16 9f 92 07 8b 4e 8d fc 05 0b 4e 1a ||S.......N....N.|
00000750 c6 b8 e0 4f 1b 6d 4f 3a 9d 2c 7c 3e 0a 42 e4 ba |...O.mO:.,|>.B..|
00000760 3c 6f d3 f1 88 d2 e8 69 3a d0 dd c2 b2 15 17 d0 |<o.....i:.......|
00000770 89 88 8a e7 34 fa 57 07 b3 37 89 5f 39 4a 2d 02 |....4.W..7._9J-.|
00000780 58 e6 7b 91 1b 15 5a 44 6e 38 e5 4b ef 8c 42 00 |X.{...ZDn8.K..B.|
00000790 b3 31 26 d7 6a 79 b8 e6 ca 72 88 c6 e7 f8 55 4e |.1&.jy...r....UN|
000007a0 62 15 84 08 2b dc 6b 52 7f 9b 6d 7e 47 6f 35 ca |b...+.kR..m~Go5.|
000007b0 a5 4d 10 07 54 d2 86 ea 5e e0 33 36 72 93 d9 6f |.M..T...^.36r..o|
000007c0 4f 09 19 63 8c 6b 04 8f 6d a9 c6 29 09 e2 63 61 |O..c.k..m..)..ca|
000007d0 6a 74 83 e5 9d 6c c1 b3 31 20 2b 97 34 88 d5 66 |jt...l..1 +.4..f|
000007e0 7c a4 f8 fa 04 b1 4e 2e 1c 81 23 79 9a 09 92 11 ||.....N...#y....|
000007f0 94 ba 59 25 7f ea a6 02 0d c1 0b f3 f3 b6 52 88 |..Y%..........R.|
00000800 fb 00 ac e4 bc 43 6a a2 ff 9b 5f c9 72 ad fc 3d |.....Cj..._.r..=|
00000810 02 41 2a 4e 36 d5 7f 19 b3 34 f2 7a 5a d7 42 62 |.A*N6....4.zZ.Bb|
00000820 4b 6e 52 2a 63 b7 fd 1f b9 97 18 88 53 7d d4 f5 |KnR*c.......S}..|
00000830 21 04 f4 ae bd 9d ea f7 44 7b f5 49 1a bf 0c 8c |!.......D{.I....|
00000840 15 0a fd b1 35 9c 57 3b 8f 2a 71 e6 40 cf 93 c8 |....5.W;.*q.@...|
00000850 b2 80 1d 8f 19 46 b7 16 ad bf f9 59 16 f5 1b 8d |.....F.....Y....|
00000860 aa ea 1e b2 d2 34 92 c2 aa 69 d0 81 2e 50 23 db |.....4...i...P#.|
00000870 29 f4 a7 cb b2 00 1d e5 a7 12 be d9 36 35 fe 6d |)...........65.m|
00000880 b2 97 b5 0c 84 3e 7b be 46 b5 50 9e 0c 3c a0 3a |.....>{.F.P..<.:|
00000890 d3 d3 d8 e8 16 c2 3e c1 ee 79 43 06 95 ad eb a3 |......>..yC.....|
000008a0 cd e3 6f 5b f2 9f 3f 12 69 2f eb 25 2f 14 2f 3d |..o[..?.i/.%/./=|
000008b0 aa 0f f8 fa e1 0b 6c 4c 55 a5 9d 4a 87 3b 8c 2c |......lLU..J.;.,|
000008c0 65 bc 0c b8 10 6a ed 82 bc 6e 98 6e 15 82 c8 ba |e....j...n.n....|
000008d0 83 9a 43 19 88 c4 56 3d f9 56 10 bd c0 58 5e 1b |..C...V=.V...X^.|
000008e0 14 9a d2 56 56 c7 fc 66 59 ff 81 77 1d 29 fe 0c |...VV..fY..w.)..|
000008f0 82 ca 17 75 d7 65 8f 04 3a 5b c9 af cf f3 5c 18 |...u.e..:[....\\.|
00000900 f8 57 1d 33 3b 73 7a a3 f4 a8 f0 a9 e3 aa bc e5 |.W.3;sz.........|
00000910 db 15 cb 51 d8 54 b2 f0 e2 e6 7d 39 59 8a 3b ec |...Q.T....}9Y.;.|
00000920 ee 2e bf db ed d5 33 09 6e 26 d2 30 7f b9 58 be |......3.n&.0..X.|
00000930 87 06 b1 ac 8a 07 2d b8 ee d1 92 55 21 5d d2 c1 |......-....U!]..|
00000940 2b 5b 4b e7 36 0b f7 1f 8d 96 0e 76 25 01 80 0d |+[K.6......v%...|
00000950 0e 93 98 43 c4 b9 95 e4 a6 28 e3 fc b8 b9 5b e9 |...C.....(....[.|
00000960 af 18 a6 bc 4d dc 2e 83 86 14 5c 95 c2 1b e3 bf |....M.....\\.....|
00000970 17 6e 72 8b 48 8b ed 1f 7a 99 b5 4a ad 8e 60 62 |.nr.H...z..J..`b|
00000980 40 2e 66 c5 8b 54 2f 3f df ac 0d 21 01 65 ad 53 |@.f..T/?...!.e.S|
00000990 d6 30 ce ef 8e fc f4 97 94 4e f2 52 f7 39 ef ee |.0.......N.R.9..|
000009a0 b7 e7 31 01 54 6c 9f d0 6a cc 22 42 8c c3 f8 a8 |..1.Tl..j.\"B....|
000009b0 9b b9 fb 1e e5 ed a0 54 d4 6a e0 53 4e 3d f3 fc |.......T.j.SN=..|
000009c0 d2 f7 04 8c 62 01 bf 68 d4 05 5c 3a 01 7c 17 d5 |....b..h..\\:.|..|
000009d0 ea d2 4c ed 87 32 81 68 f8 4d 36 70 aa 31 86 75 |..L..2.h.M6p.1.u|
000009e0 76 f0 c6 21 f1 0e d8 4e 74 de 9e e7 74 bb f5 dd |v..!...Nt...t...|
000009f0 f5 9c 68 9f 44 c4 c6 d2 bd f9 fe 6d 63 66 5d 6e |..h.D......mcf]n|
00000a00 64 45 28 cb 5d 4c 0d 80 c7 8c 5b 83 53 77 4b 7c |dE(.]L....[.SwK||
00000a10 f1 32 c1 f9 1f c5 11 e5 39 2a e6 a9 0c de 68 91 |.2......9*....h.|
00000a20 66 6b 38 04 ed fc d7 60 6e a7 29 7e 5b 13 47 fe |fk8....`n.)~[.G.|
00000a30 a7 e0 53 23 f5 aa 2a bd 91 d7 fd 02 ae f3 29 b1 |..S#..*.......).|
00000a40 df 0b cd e0 47 c6 cf 20 03 3e 93 a0 25 bb 4c 13 |....G.. .>..%.L.|
00000a50 5c fc dc 5c 35 55 a3 f5 86 0b da 9a 1e 63 e4 de |\\..\\5U.......c..|
00000a60 bc 83 1d f3 af dc 49 e8 21 5f 30 6d 14 f2 cc 00 |......I.!_0m....|
00000a70 2d 7a 07 47 99 21 a9 98 bf 57 bb 99 f2 d4 32 38 |-z.G.!...W....28|
00000a80 a7 67 8a 5f 17 06 e1 1c 9d 5d 69 f0 76 64 3d 45 |.g._.....]i.vd=E|
00000a90 ad c0 e6 dc 7d e9 59 8d 9b 33 4c 04 a7 be 2d b5 |....}.Y..3L...-.|
00000aa0 b0 f5 26 14 86 b9 de 45 f4 d8 8a 59 27 db 38 56 |..&....E...Y'.8V|
00000ab0 7c 4b 1b 52 37 a7 9f 71 00 75 66 3a 2f 0a e3 f9 ||K.R7..q.uf:/...|
00000ac0 c7 21 dd 70 50 52 14 9f ff 7f 2f d5 21 2e c6 ba |.!.pPR..../.!...|
00000ad0 5c 3d 84 39 be ef 25 db f2 dd ec 53 63 1a 69 fd |\\=.9..%....Sc.i.|
00000ae0 be 71 59 a4 de df 11 3e ee ae c9 5d e8 75 1c cc |.qY....>...].u..|
00000af0 1c 00 db e6 42 1b ba 99 03 29 0b 83 02 12 bc 7e |....B....).....~|
00000b00 dd a8 7e 2a 68 41 71 d8 28 b1 9c cd a3 fa 6c 3a |..~*hAq.(.....l:|
00000b10 88 78 69 2d 94 6e dc 85 a6 d4 59 02 7f c2 2f 01 |.xi-.n....Y.../.|
00000b20 0f 06 03 ac ac 30 bb b6 84 43 3f 0c 4d 9a 63 2a |.....0...C?.M.c*|
00000b30 22 98 fb b4 2c a0 b8 49 87 ba 13 ed 0b 30 c0 39 |\"...,..I.....0.9|
00000b40 f5 5b 31 7d dd 54 89 a8 2d cf fd 79 b9 ef 3a 50 |.[1}.T..-..y..:P|
00000b50 85 54 fd 5a de 21 8c 73 f5 e8 38 30 db dc 23 96 |.T.Z.!.s..80..#.|
00000b60 c7 41 f1 94 d8 ed b6 85 ef fd 81 3f b0 a8 a1 6a |.A.........?...j|
00000b70 41 7c f5 8d 3c 17 64 cf 80 37 66 78 02 e4 05 e5 |A|..<.d..7fx....|
00000b80 80 00 87 c4 dc f2 37 fd ed f3 b7 a2 91 a8 90 04 |......7.........|
00000b90 85 2c ac 52 3f d0 0a 39 20 b1 50 34 03 57 80 ed |.,.R?..9 .P4.W..|
00000ba0 8b 60 87 1b f3 59 89 f8 a0 6d f8 fa 8d 70 b9 f2 |.`...Y...m...p..|
00000bb0 57 3b cc db f4 6c 75 c3 b0 f0 54 86 e4 52 50 cc |W;...lu...T..RP.|
00000bc0 97 8e fc 69 6b 26 ba e0 2e d5 10 8c 7b 84 8f 4e |...ik&......{..N|
00000bd0 1b 40 c9 d5 f4 95 e5 c2 6c 91 75 eb 0e 85 18 3d |.@......l.u....=|
00000be0 7f 57 24 de bd 05 fb bd 83 ac 08 30 40 f0 00 06 |.W$........0@...|
00000bf0 8a 98 c7 dd 3b 08 c8 84 72 d4 88 9f f9 9d 96 9e |....;...r.......|
00000c00 d3 15 05 95 db 03 fe c5 e3 7f 31 68 ab 40 29 b9 |..........1h.@).|
00000c10 ff 66 87 cc 08 cc 49 9f 5f e4 24 de 12 fd 60 d7 |.f....I._.$...`.|
00000c20 0f 3e a0 7d ca 34 91 29 1c d9 88 6d 9d fb 56 16 |.>.}.4.)...m..V.|
00000c30 bd 55 0c c7 1b f2 4a 2d eb ab 69 8f f3 fa 83 dc |.U....J-..i.....|
00000c40 ae 6a 58 9a 4c c7 a6 50 88 fd 15 d5 df 1f a9 9b |.jX.L..P........|
00000c50 91 c1 2e 79 c1 e0 1f 42 2e 48 42 db 9c be 5f 1c |...y...B.HB..._.|
00000c60 31 88 dc ad db 6d f6 27 45 bc 03 57 c6 f1 26 0b |1....m.'E..W..&.|
00000c70 f2 1f 1d 42 44 c3 53 cd 77 e5 e2 90 64 c1 1f 77 |...BD.S.w...d..w|
00000c80 ff ef 02 00 54 21 0c 6a a2 d4 4b 9c d6 3c e8 99 |....T!.j..K..<..|
00000c90 e9 64 8c 10 7e ba b7 fd e6 68 62 b9 06 3c d8 c1 |.d..~....hb..<..|
00000ca0 5e ca 51 dd 60 df 99 28 0a 16 30 58 8d 5f 58 1d |^.Q.`..(..0X._X.|
00000cb0 47 3a 49 de 5f 11 3f d0 f3 05 d4 9f 97 a4 da 31 |G:I._.?........1|
00000cc0 7e 25 73 f7 0b ef 8f e8 c4 3d 57 ad 77 6c 78 56 |~%s......=W.wlxV|
00000cd0 9c 8b a5 10 1a 32 f1 75 ca 9e 06 59 da 58 d6 3f |.....2.u...Y.X.?|
00000ce0 43 97 4d c7 b5 da 41 f8 8e b4 1f 4c 18 d9 07 c2 |C.M...A....L....|
00000cf0 68 85 a0 94 a9 38 a6 b9 cf 16 5f 36 fd 3f 2b da |h....8...._6.?+.|
00000d00 87 d6 29 95 91 71 a3 47 5e 83 8d fe ba e6 a3 56 |..)..q.G^......V|
00000d10 c4 73 c5 d2 d6 ea 0c 6f c7 c1 b8 84 68 a7 9e de |.s.....o....h...|
00000d20 92 f7 dd a0 91 ca 78 79 65 88 90 b7 7e eb 9a 60 |......xye...~..`|
00000d30 72 2b 29 e3 56 04 6b 4b e7 38 27 bb ad 93 99 13 |r+).V.kK.8'.....|
00000d40 fe df 5a e2 5f 66 41 f3 99 11 ea 68 18 83 4c 3d |..Z._fA....h..L=|
00000d50 a8 01 8a 6b a8 aa f6 3f 7f 76 06 6b ad 04 25 fe |...k...?.v.k..%.|
00000d60 b4 ab 49 04 21 a3 e3 e3 03 c2 ba 32 04 16 48 e5 |..I.!......2..H.|
00000d70 d7 8c 2f 52 e6 3f 8d b8 94 60 e8 04 e9 34 9e fd |../R.?...`...4..|
00000d80 fa 41 12 13 da 54 7d 2f ff 6a 3e de 26 e1 9b 34 |.A...T}/.j>.&..4|
00000d90 8e 26 2b 24 d0 eb b3 22 35 1a ad b2 70 04 9b d4 |.&+$...\"5...p...|
00000da0 1e 90 46 90 de 74 9e fe a7 6e a8 59 a3 b9 b8 1a |..F..t...n.Y....|
00000db0 a3 9b 65 e0 d2 64 53 4b 5e 42 20 41 9c 55 60 48 |..e..dSK^B A.U`H|
00000dc0 b2 30 f7 f6 18 2b 71 64 cd 86 11 0e 0e a3 d5 f4 |.0...+qd........|
00000dd0 b0 ca b9 b1 bd 76 cd 40 d2 ef 12 46 3c 28 ec f6 |.....v.@...F<(..|
00000de0 33 96 dc 26 33 e0 08 46 f2 68 1e 43 a7 38 16 80 |3..&3..F.h.C.8..|
00000df0 9c 6d 8b 28 c0 03 a7 99 22 61 0a 47 34 5e 5e 58 |.m.(....\"a.G4^^X|
00000e00 67 67 db de df cf ff c0 05 dc 09 84 e2 28 60 2f |gg...........(`/|
00000e10 f7 08 3d a8 8a 69 4a 0b 1c c5 44 f1 29 11 3e 71 |..=..iJ...D.).>q|
00000e20 ae f3 d1 c2 37 d1 18 79 ab 64 8f 82 bf 38 7b 1b |....7..y.d...8{.|
00000e30 52 ef 00 9a 4f 44 dc 4c 7c f9 c9 47 ba a8 6d fa |R...OD.L|..G..m.|
00000e40 1d aa f5 ff c4 c2 0d 53 27 e6 17 d0 be 6d 97 30 |.......S'....m.0|
00000e50 81 1c 9c c6 75 35 14 ff 73 d8 78 27 92 51 8e fc |....u5..s.x'.Q..|
00000e60 94 ec 64 45 19 5c 0b be b5 99 e9 b9 76 5f 70 de |..dE.\\......v_p.|
00000e70 00 b3 16 08 b0 76 06 a9 77 8a 79 c5 fc 0b f1 6c |.....v..w.y....l|
00000e80 03 c1 0f a9 e9 70 19 ba fa b2 4f e2 13 33 6a 06 |.....p....O..3j.|
00000e90 1c af 25 2c eb 94 3b 18 31 0a d0 b9 a6 13 53 7c |..%,..;.1.....S||
00000ea0 3a 00 57 23 05 41 34 2c 7c 4f e2 f3 72 0b e2 a7 |:.W#.A4,|O..r...|
00000eb0 e0 78 02 ab 7f 2c 0e 32 c3 d4 e4 b8 2b 2f 31 fb |.x...,.2....+/1.|
00000ec0 f8 92 db 7a 41 4d 98 71 b9 a3 0d eb fd 71 fc 2e |...zAM.q.....q..|
00000ed0 e8 5c a1 b5 58 f8 5a 11 20 04 da 52 39 b6 06 70 |.\\..X.Z. ..R9..p|
00000ee0 43 0d 7a 48 14 18 21 03 2c 61 37 ef 23 41 b8 41 |C.zH..!.,a7.#A.A|
00000ef0 3b e4 a0 f6 71 2a d2 06 97 b0 5a d2 81 36 b0 6f |;...q*....Z..6.o|
00000f00 00 ed db 5f 96 bd 37 bd 7c bc 33 47 37 67 2a ad |..._..7.|.3G7g*.|
00000f10 b6 81 e1 8d 5a e2 68 c2 0b 32 d3 f1 21 02 77 3b |....Z.h..2..!.w;|
00000f20 89 ee da 0d 87 8f 26 0f 73 17 19 3b b7 04 e5 38 |......&.s..;...8|
00000f30 54 c4 f3 1b 7b bc 93 9a ce 38 87 f2 3f a2 8b 38 |T...{....8..?..8|
00000f40 a9 b1 7c 51 a2 34 f0 72 43 5c 59 35 b6 fd 50 2e |..|Q.4.rC\\Y5..P.|
00000f50 db dc 29 d4 65 6a ef 23 83 7d 9d 49 b7 56 e4 c0 |..).ej.#.}.I.V..|
00000f60 4e 4d d8 06 ca e5 85 ee 41 cb 66 84 1a af 79 7f |NM......A.f...y.|
00000f70 f0 9e 7c 74 8c 4a 80 e2 e6 0a 58 fb c9 61 3f a1 |..|t.J....X..a?.|
00000f80 b7 3b 7a ef 1f 91 95 d5 51 ab 01 ef 0e 86 c4 bb |.;z.....Q.......|
00000f90 78 0e cb f3 b5 f9 30 d8 ae f0 81 11 e9 24 e6 e2 |x.....0......$..|
00000fa0 67 8a 30 9c 04 a6 a1 cb d2 73 01 81 fb 01 28 f0 |g.0......s....(.|
00000fb0 62 2d d5 91 24 9e ec f8 74 3e 26 63 46 a0 b1 c2 |b-..$...t>&cF...|
00000fc0 63 31 0d 91 e8 fb 0d 6d 43 e0 8b ce 6b 8b 77 a2 |c1.....mC...k.w.|
00000fd0 63 56 28 c8 60 8e c1 53 ae 70 3d 06 54 b5 f4 9a |cV(.`..S.p=.T...|
00000fe0 8a be 4c 43 9c f6 64 40 6e 97 2c 97 cd 74 c8 83 |..LC..d@n.,..t..|
00000ff0 b3 0b 41 74 56 fc 31 93 a5 dd c2 15 8c dd 9d 43 |..AtV.1........C|
00001000
2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound:
2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db:
2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m
2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid
2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump: ", "00000000 40 6a a8 52 08 5a 8a 2a 74 c2 c5 98 f9 c6 c8 12 |@j.R.Z.*t.......|", "00000010 5c 45 ca 37 bb 49 87 6d 98 f7 46 07 d6 3c 48 40 |\\E.7.I.m..F..<H@|", "00000020 af b9 65 99 4f be 0b 20 9e 29 d5 9b 10 b1 74 ca |..e.O.. .)....t.|", "00000030 82 77 b4 c5 ab de 04 9c eb 90 7c ff ce 76 ac ed |.w........|..v..|", "00000040 3e 43 93 72 da 75 de 41 ed a9 49 5a 13 7a da 72 |>C.r.u.A..IZ.z.r|", "00000050 7e 8d 1c d3 5b 31 98 7d 48 ed da 00 63 e7 f1 d5 |~...[1.}H...c...|", "00000060 d1 5c f1 7f 2a c5 af 7d 56 2f fb 16 d7 26 56 ce |.\\..*..}V/...&V.|", "00000070 ed ec 11 04 c6 b6 52 df 18 64 49 75 65 e8 62 47 |......R..dIue.bG|", "00000080 29 1f de 22 e3 b2 45 0f ec fb 7c d9 f9 9a 5a d8 |)..\"..E...|...Z.|", "00000090 82 40 71 6f c3 b6 73 5f e1 ed 3f 3d fd 56 8c e8 |.@qo..s_..?=.V..|", "000000a0 a8 6a bf 6c f2 bc fa ca 44 8e e6 b5 6f f0 d0 bc |.j.l....D...o...|", "000000b0 8f a3 ae 35 44 d2 23 74 6b 08 d6 75 9f 74 b0 fb |...5D.#tk..u.t..|", "000000c0 8a f0 07 d1 7c b2 3e fd c3 e1 a5 7c cf d3 62 a6 |....|.>....|..b.|", "000000d0 5b 04 c7 6d 7d 70 d9 72 cf 1b ee ef c6 59 41 4a |[..m}p.r.....YAJ|", "000000e0 de 0d de 51 d1 02 cd ae be 3f 7d fd 45 92 f8 63 |...Q.....?}.E..c|", "000000f0 6d 41 19 0f c7 d3 0e cf 64 2c 59 fc 86 0a c3 ec |mA......d,Y.....|", "00000100 29 5d 6f 13 6b 62 fc 80 f4 b4 ce 9b 53 81 36 ea |)]o.kb......S.6.|", "00000110 5e 07 e0 61 20 d2 15 43 2a 7a 84 f1 a3 0f 9b 8d |^..a ..C*z......|", "00000120 ba 42 01 94 df f3 d4 87 fc 45 5c 8a dd d2 28 26 |.B.......E\\...(&|", "00000130 a7 35 50 ee 58 1e f3 22 36 1d da 76 2c 07 c9 28 |.5P.X..\"6..v,..(|", "00000140 e1 cb 66 20 cb e4 dd bb 91 eb b0 4f 66 1f 89 69 |..f .......Of..i|", "00000150 4a ec 47 e3 d2 f6 2f 59 18 39 29 42 14 ea f6 19 |J.G.../Y.9)B....|", "00000160 45 a0 e2 44 ed 97 4d 1f d2 f2 70 b0 13 70 e3 5b |E..D..M...p..p.[|", "00000170 95 bb 43 6d 9c 35 25 17 14 2d a5 30 46 e3 38 e4 |..Cm.5%..-.0F.8.|", "00000180 55 63 06 15 1c 1e d3 2e fd 7f 3e 4b 18 07 ed 53 |Uc........>K...S|", "00000190 0b 7e d8 dd 39 0a d2 53 23 10 ce 7a 55 53 5e cf |.~..9..S#..zUS^.|", "000001a0 77 6d a8 f8 ef e5 8a 01 13 b0 ac ce 31 07 c7 19 |wm..........1...|", "000001b0 69 0e 00 c6 26 88 7a e0 be f5 2c ac 2c 6b ce 8a |i...&.z...,.,k..|", "000001c0 b3 82 fe fc 16 ea 46 9e eb 7e 54 f4 b1 4d 00 dc |......F..~T..M..|", "000001d0 0e e9 5b 3a 8b 5a 49 f0 46 f6 31 a8 9b 39 ba 96 |..[:.ZI.F.1..9..|", "000001e0 b4 ce 76 69 91 5b 74 36 e9 89 cc 67 7c 2e c7 6b |..vi.[t6...g|..k|", "000001f0 2e b2 f1 b0 fc a6 c6 c1 c9 35 b5 be 23 c1 b3 94 |.........5..#...|", "00000200 11 47 12 1a ea 5e 47 84 a5 92 6c 83 e3 e2 ff 8a |.G...^G...l.....|", "00000210 ed ba 48 f8 41 17 d5 2e d2 64 3a 64 64 21 ef 48 |..H.A....d:dd!.H|", "00000220 f8 78 b3 2a ca 0b 1e 7d 80 0b fb 1c b5 73 1b ee |.x.*...}.....s..|", "00000230 0f a0 4d 6b d9 22 72 06 1c 5d 27 af f6 c3 52 a8 |..Mk.\"r..]'...R.|", "00000240 47 ac bf 18 50 4a 64 29 3c 71 1b 9d 21 37 a9 f1 |G...PJd)<q..!7..|", "00000250 ea 52 7a 25 48 ef 8d 45 97 65 49 8b 1c a0 c5 1d |.Rz%H..E.eI.....|", "00000260 94 da dc 90 b3 fe 56 2f 5e de 0a 02 58 6b 6f e6 |......V/^...Xko.|", "00000270 e3 0e 93 eb 98 9a 4d 35 4f 8b 50 53 9f f6 35 d5 |......M5O.PS..5.|", "00000280 2d c6 c6 45 d4 38 1d 68 7d b7 46 e0 8a 60 1b ec |-..E.8.h}.F..`..|", "00000290 28 64 22 67 8b d4 0e 7c 05 28 24 59 fb b6 e5 44 |(d\"g...|.($Y...D|", "000002a0 9e f4 d2 a5 08 cb 3a 52 a5 51 c3 56 cd fb 29 0b |......:R.Q.V..).|", "000002b0 8d bb 28 ab 06 c5 e8 9c b8 99 ee 42 a5 4f 1c 0c |..(........B.O..|", "000002c0 5f 29 1e 2f cf b2 f6 dc 05 21 b1 68 d2 79 ae b2 |_)./.....!.h.y..|", "000002d0 d9 38 5f f6 5d 08 60 44 c1 6b d8 b1 a9 41 b7 48 |.8_.].`D.k...A.H|", "000002e0 44 7a 4d f0 67 ef d9 ca 2f d5 69 9b c3 bf 78 ff |DzM.g.../.i...x.|", "000002f0 2e f4 a4 94 14 0d fa 7b 60 55 fb 75 fb 49 52 5f |.......{`U.u.IR_|", "00000300 fc 22 ad 16 ac c2 80 e1 ab cf bf 7f 84 6b df 0e |.\"...........k..|", "00000310 4c 38 56 02 1a 37 bf 07 b1 e9 ac dc a8 56 f2 bf |L8V..7.......V..|", "00000320 59 f8 8c 9b 58 88 54 2f bf d5 5e 1a c4 93 ea f3 |Y...X.T/..^.....|", "00000330 ab 56 f0 a5 7c 66 7c 83 dd 00 30 6f 84 37 72 6b |.V..|f|...0o.7rk|", "00000340 6a 13 d8 a8 e2 49 c7 2c e8 b8 c1 9f 29 3e e0 12 |j....I.,....)>..|", "00000350 63 5f 22 df bd f7 9b 25 7e e9 c2 ff 1c 86 f0 e7 |c_\"....%~.......|", "00000360 a9 59 06 18 9d fa 20 eb 2f 72 31 c4 53 c2 dd 09 |.Y.... ./r1.S...|", "00000370 65 3c 9d 5d 7b 51 57 28 de 8b a5 e1 4a 1f 64 2e |e<.]{QW(....J.d.|", "00000380 37 49 7f bd 15 57 09 dc fb 74 40 68 00 62 f8 5d |7I...W...t@h.b.]|", "00000390 99 2c 6e c7 e6 64 bc 3f 26 46 90 f8 69 95 3b 48 |.,n..d.?&F..i.;H|", "000003a0 a7 45 80 09 2d e4 5f c3 89 8e a1 e9 50 c6 87 22 |.E..-._.....P..\"|", "000003b0 04 b5 c6 87 a8 1d 96 ff 65 bd 93 b0 69 62 8d b7 |........e...ib..|", "000003c0 5b 9e e1 34 25 b6 7f 0e e6 2f 79 b2 ae 85 fc c8 |[..4%..../y.....|", "000003d0 cd 4d dd b3 86 79 89 c8 86 d3 a9 a5 5d aa 9c 0c |.M...y......]...|", "000003e0 a0 c3 e1 01 cb c8 30 81 46 35 f8 6f 81 4c d3 b5 |......0.F5.o.L..|", "000003f0 be a8 e3 29 96 8c bb 1c 7c 1f fb e6 51 98 b6 90 |...)....|...Q...|", "00000400 c0 a2 fc 9e bb b1 52 f1 b1 42 09 58 12 22 ec 16 |......R..B.X.\"..|", "00000410 5f 81 49 ee 99 0a 6a 88 e9 76 60 4d ee 58 83 d3 |_.I...j..v`M.X..|", "00000420 98 5b 55 ec 59 e8 ab 1a f6 d8 4a 9c 4b e5 56 66 |.[U.Y.....J.K.Vf|", "00000430 b1 e1 fd 91 59 20 ce 42 db 6b 3d 7c 6a 7e 5e 05 |....Y .B.k=|j~^.|", "00000440 49 a9 35 e3 28 25 08 bf 37 10 cb 60 27 ab c9 17 |I.5.(%..7..`'...|", "00000450 2b 0e e0 54 2a 5a 53 df 80 e6 ca bf 7a de 55 54 |+..T*ZS.....z.UT|", "00000460 71 f8 ca 9e fc 46 b3 c3 7f e1 ae a5 cd 6b 51 17 |q....F.......kQ.|", "00000470 b3 48 fa 00 b9 7a 82 f1 16 02 ab 78 1a ad 99 5f |.H...z.....x..._|", "00000480 8c c9 1d 53 dc 0e ff 12 b6 ba 6f 0e fe fb f2 31 |...S......o....1|", "00000490 77 e9 f5 17 3d db ab 75 3f ce 76 48 57 37 e3 b4 |w...=..u?.vHW7..|", "000004a0 b0 cc ae 79 f2 50 d4 55 fc 8b 1c aa 14 38 b8 a1 |...y.P.U.....8..|", "000004b0 78 7b 20 72 67 03 25 c7 e7 15 ff a8 86 f9 ec f7 |x{ rg.%.........|", "000004c0 3b 2c 56 20 87 75 53 a4 51 10 d7 1e b8 ea 6d d9 |;,V .uS.Q.....m.|", "000004d0 78 18 98 b0 65 e7 a2 df fa 0b f2 79 cb c4 51 98 |x...e......y..Q.|", "000004e0 be d0 eb c4 72 b4 a8 48 0f aa 6e 1b dc ff cb 1d |....r..H..n.....|", "000004f0 aa 34 ee 55 57 19 87 3c 69 67 f3 63 b6 61 48 6d |.4.UW..<ig.c.aHm|", "00000500 37 7d 18 32 6b 74 f1 59 c5 11 7d 13 e5 4c 21 81 |7}.2kt.Y..}..L!.|", "00000510 5d 79 8a cb 9a 41 7a f9 9d 0e e7 ca 38 66 f4 1a |]y...Az.....8f..|", "00000520 b7 a1 cd 7e 4d b1 19 f3 5d 53 a9 8d 9e 83 ae 0b |...~M...]S......|", "00000530 c3 87 cd 4a 6e fe af a0 61 95 30 ee 99 ec 34 9a |...Jn...a.0...4.|", "00000540 ec ac 19 ff fe 97 69 54 22 cc 04 17 62 a8 ee 7c |......iT\"...b..||", "00000550 8f a4 a0 50 90 b9 fa e7 3f 5d 7a 2b f8 d1 83 8b |...P....?]z+....|", "00000560 13 51 6c c4 a1 1c b1 5c de 83 a6 aa 43 d1 1d c1 |.Ql....\\....C...|", "00000570 e9 e6 60 c0 f2 2c 35 42 b7 71 a4 00 3a b2 db 36 |..`..,5B.q..:..6|", "00000580 73 40 12 90 ee 73 b6 d5 84 7b 4a c2 73 1e 2e af |s@...s...{J.s...|", "00000590 33 53 82 67 89 10 cb 85 5a 96 d3 af 47 d9 47 07 |3S.g....Z...G.G.|", "000005a0 18 e0 ba 7a aa 0c 69 08 d5 d2 60 a1 2a 38 8e 3e |...z..i...`.*8.>|", "000005b0 dd de a1 d1 24 32 63 36 55 31 d4 7c 51 1e 83 ea |....$2c6U1.|Q...|", "000005c0 4d f4 f0 ef 75 0b 89 fa 17 26 fa 81 67 53 36 57 |M...u....&..gS6W|", "000005d0 87 6f ee 61 0d 65 d3 3c ba b1 04 0c ae 27 c3 7c |.o.a.e.<.....'.||", "000005e0 19 ce 18 e3 cd 60 37 b4 b1 f7 f4 8b c2 fa 42 e5 |.....`7.......B.|", "000005f0 a6 7f f2 2f 39 c9 94 62 93 68 64 5d 8b 1a 40 9e |.../9..b.hd]..@.|", "00000600 ab 75 b3 e8 88 4f 5d ea a9 8e 4b fb c5 51 ca 9c |.u...O]...K..Q..|", "00000610 49 d2 fc 68 18 4a 22 ce 7d 1d 00 e7 52 61 5f 85 |I..h.J\".}...Ra_.|", "00000620 ec 3c 13 74 a5 78 75 3e 14 f2 27 ed b2 5d 92 16 |.<.t.xu>..'..]..|", "00000630 05 39 dc ee 8c 13 4c af 63 86 1c 3f ac 7e 81 67 |.9....L.c..?.~.g|", "00000640 b0 d7 a2 f3 9f 31 73 ae 97 08 11 2b fa 99 5e c9 |.....1s....+..^.|", "00000650 d1 39 4a 9e a4 bb d1 55 83 c2 f4 2f bd 7e 85 c9 |.9J....U.../.~..|", "00000660 d9 34 c6 61 c6 1d 12 9c d4 e9 ec 05 4f 04 55 3a |.4.a........O.U:|", "00000670 78 2f fa 33 02 87 97 e1 65 ef af 8e f0 b2 f9 3c |x/.3....e......<|", "00000680 75 ca 39 cc 12 2e 58 6e bd 50 9f 46 83 4f 02 3e |u.9...Xn.P.F.O.>|", "00000690 50 8d 0a 8f 11 a2 06 70 70 0f 94 91 2d c7 da c4 |P......pp...-...|", "000006a0 9a 22 eb 9f 14 04 c4 d0 35 c7 c9 60 9d b0 10 c3 |.\"......5..`....|", "000006b0 ab fd c7 31 a4 b7 23 78 10 17 3b de 05 c8 f1 1b |...1..#x..;.....|", "000006c0 90 32 54 ff eb bc 1f e4 11 78 f9 9c 2f ab 49 75 |.2T......x../.Iu|", "000006d0 9f 42 89 54 77 9a 0b 0c a7 e8 b2 18 85 cd 2a d0 |.B.Tw.........*.|", "000006e0 39 a0 09 96 09 2c a5 34 ef e9 85 0b d8 ed 22 e0 |9....,.4......\".|", "000006f0 7c 80 07 8b 12 ed 60 b1 e8 ec 4e 26 97 f0 50 e1 ||.....`...N&..P.|", "00000700 2f 2b 08 b8 03 87 7a da 44 7d 93 16 b5 70 c7 a2 |/+....z.D}...p..|", "00000710 2f 5b 34 10 5d dc f6 f4 46 8c 8e 5c 4c aa db 44 |/[4.]...F..\\L..D|", "00000720 ce fa ea 39 cb 23 e8 0c f4 13 99 e4 20 c7 9c 6a |...9.#...... ..j|", "00000730 9e 37 b0 e0 42 af b3 59 98 6b fd 66 61 85 6e dc |.7..B..Y.k.fa.n.|", "00000740 7c 53 b4 0b 16 9f 92 07 8b 4e 8d fc 05 0b 4e 1a ||S.......N....N.|", "00000750 c6 b8 e0 4f 1b 6d 4f 3a 9d 2c 7c 3e 0a 42 e4 ba |...O.mO:.,|>.B..|", "00000760 3c 6f d3 f1 88 d2 e8 69 3a d0 dd c2 b2 15 17 d0 |<o.....i:.......|", "00000770 89 88 8a e7 34 fa 57 07 b3 37 89 5f 39 4a 2d 02 |....4.W..7._9J-.|", "00000780 58 e6 7b 91 1b 15 5a 44 6e 38 e5 4b ef 8c 42 00 |X.{...ZDn8.K..B.|", "00000790 b3 31 26 d7 6a 79 b8 e6 ca 72 88 c6 e7 f8 55 4e |.1&.jy...r....UN|", "000007a0 62 15 84 08 2b dc 6b 52 7f 9b 6d 7e 47 6f 35 ca |b...+.kR..m~Go5.|", "000007b0 a5 4d 10 07 54 d2 86 ea 5e e0 33 36 72 93 d9 6f |.M..T...^.36r..o|", "000007c0 4f 09 19 63 8c 6b 04 8f 6d a9 c6 29 09 e2 63 61 |O..c.k..m..)..ca|", "000007d0 6a 74 83 e5 9d 6c c1 b3 31 20 2b 97 34 88 d5 66 |jt...l..1 +.4..f|", "000007e0 7c a4 f8 fa 04 b1 4e 2e 1c 81 23 79 9a 09 92 11 ||.....N...#y....|", "000007f0 94 ba 59 25 7f ea a6 02 0d c1 0b f3 f3 b6 52 88 |..Y%..........R.|", "00000800 fb 00 ac e4 bc 43 6a a2 ff 9b 5f c9 72 ad fc 3d |.....Cj..._.r..=|", "00000810 02 41 2a 4e 36 d5 7f 19 b3 34 f2 7a 5a d7 42 62 |.A*N6....4.zZ.Bb|", "00000820 4b 6e 52 2a 63 b7 fd 1f b9 97 18 88 53 7d d4 f5 |KnR*c.......S}..|", "00000830 21 04 f4 ae bd 9d ea f7 44 7b f5 49 1a bf 0c 8c |!.......D{.I....|", "00000840 15 0a fd b1 35 9c 57 3b 8f 2a 71 e6 40 cf 93 c8 |....5.W;.*q.@...|", "00000850 b2 80 1d 8f 19 46 b7 16 ad bf f9 59 16 f5 1b 8d |.....F.....Y....|", "00000860 aa ea 1e b2 d2 34 92 c2 aa 69 d0 81 2e 50 23 db |.....4...i...P#.|", "00000870 29 f4 a7 cb b2 00 1d e5 a7 12 be d9 36 35 fe 6d |)...........65.m|", "00000880 b2 97 b5 0c 84 3e 7b be 46 b5 50 9e 0c 3c a0 3a |.....>{.F.P..<.:|", "00000890 d3 d3 d8 e8 16 c2 3e c1 ee 79 43 06 95 ad eb a3 |......>..yC.....|", "000008a0 cd e3 6f 5b f2 9f 3f 12 69 2f eb 25 2f 14 2f 3d |..o[..?.i/.%/./=|", "000008b0 aa 0f f8 fa e1 0b 6c 4c 55 a5 9d 4a 87 3b 8c 2c |......lLU..J.;.,|", "000008c0 65 bc 0c b8 10 6a ed 82 bc 6e 98 6e 15 82 c8 ba |e....j...n.n....|", "000008d0 83 9a 43 19 88 c4 56 3d f9 56 10 bd c0 58 5e 1b |..C...V=.V...X^.|", "000008e0 14 9a d2 56 56 c7 fc 66 59 ff 81 77 1d 29 fe 0c |...VV..fY..w.)..|", "000008f0 82 ca 17 75 d7 65 8f 04 3a 5b c9 af cf f3 5c 18 |...u.e..:[....\\.|", "00000900 f8 57 1d 33 3b 73 7a a3 f4 a8 f0 a9 e3 aa bc e5 |.W.3;sz.........|", "00000910 db 15 cb 51 d8 54 b2 f0 e2 e6 7d 39 59 8a 3b ec |...Q.T....}9Y.;.|", "00000920 ee 2e bf db ed d5 33 09 6e 26 d2 30 7f b9 58 be |......3.n&.0..X.|", "00000930 87 06 b1 ac 8a 07 2d b8 ee d1 92 55 21 5d d2 c1 |......-....U!]..|", "00000940 2b 5b 4b e7 36 0b f7 1f 8d 96 0e 76 25 01 80 0d |+[K.6......v%...|", "00000950 0e 93 98 43 c4 b9 95 e4 a6 28 e3 fc b8 b9 5b e9 |...C.....(....[.|", "00000960 af 18 a6 bc 4d dc 2e 83 86 14 5c 95 c2 1b e3 bf |....M.....\\.....|", "00000970 17 6e 72 8b 48 8b ed 1f 7a 99 b5 4a ad 8e 60 62 |.nr.H...z..J..`b|", "00000980 40 2e 66 c5 8b 54 2f 3f df ac 0d 21 01 65 ad 53 |@.f..T/?...!.e.S|", "00000990 d6 30 ce ef 8e fc f4 97 94 4e f2 52 f7 39 ef ee |.0.......N.R.9..|", "000009a0 b7 e7 31 01 54 6c 9f d0 6a cc 22 42 8c c3 f8 a8 |..1.Tl..j.\"B....|", "000009b0 9b b9 fb 1e e5 ed a0 54 d4 6a e0 53 4e 3d f3 fc |.......T.j.SN=..|", "000009c0 d2 f7 04 8c 62 01 bf 68 d4 05 5c 3a 01 7c 17 d5 |....b..h..\\:.|..|", "000009d0 ea d2 4c ed 87 32 81 68 f8 4d 36 70 aa 31 86 75 |..L..2.h.M6p.1.u|", "000009e0 76 f0 c6 21 f1 0e d8 4e 74 de 9e e7 74 bb f5 dd |v..!...Nt...t...|", "000009f0 f5 9c 68 9f 44 c4 c6 d2 bd f9 fe 6d 63 66 5d 6e |..h.D......mcf]n|", "00000a00 64 45 28 cb 5d 4c 0d 80 c7 8c 5b 83 53 77 4b 7c |dE(.]L....[.SwK||", "00000a10 f1 32 c1 f9 1f c5 11 e5 39 2a e6 a9 0c de 68 91 |.2......9*....h.|", "00000a20 66 6b 38 04 ed fc d7 60 6e a7 29 7e 5b 13 47 fe |fk8....`n.)~[.G.|", "00000a30 a7 e0 53 23 f5 aa 2a bd 91 d7 fd 02 ae f3 29 b1 |..S#..*.......).|", "00000a40 df 0b cd e0 47 c6 cf 20 03 3e 93 a0 25 bb 4c 13 |....G.. .>..%.L.|", "00000a50 5c fc dc 5c 35 55 a3 f5 86 0b da 9a 1e 63 e4 de |\\..\\5U.......c..|", "00000a60 bc 83 1d f3 af dc 49 e8 21 5f 30 6d 14 f2 cc 00 |......I.!_0m....|", "00000a70 2d 7a 07 47 99 21 a9 98 bf 57 bb 99 f2 d4 32 38 |-z.G.!...W....28|", "00000a80 a7 67 8a 5f 17 06 e1 1c 9d 5d 69 f0 76 64 3d 45 |.g._.....]i.vd=E|", "00000a90 ad c0 e6 dc 7d e9 59 8d 9b 33 4c 04 a7 be 2d b5 |....}.Y..3L...-.|", "00000aa0 b0 f5 26 14 86 b9 de 45 f4 d8 8a 59 27 db 38 56 |..&....E...Y'.8V|", "00000ab0 7c 4b 1b 52 37 a7 9f 71 00 75 66 3a 2f 0a e3 f9 ||K.R7..q.uf:/...|", "00000ac0 c7 21 dd 70 50 52 14 9f ff 7f 2f d5 21 2e c6 ba |.!.pPR..../.!...|", "00000ad0 5c 3d 84 39 be ef 25 db f2 dd ec 53 63 1a 69 fd |\\=.9..%....Sc.i.|", "00000ae0 be 71 59 a4 de df 11 3e ee ae c9 5d e8 75 1c cc |.qY....>...].u..|", "00000af0 1c 00 db e6 42 1b ba 99 03 29 0b 83 02 12 bc 7e |....B....).....~|", "00000b00 dd a8 7e 2a 68 41 71 d8 28 b1 9c cd a3 fa 6c 3a |..~*hAq.(.....l:|", "00000b10 88 78 69 2d 94 6e dc 85 a6 d4 59 02 7f c2 2f 01 |.xi-.n....Y.../.|", "00000b20 0f 06 03 ac ac 30 bb b6 84 43 3f 0c 4d 9a 63 2a |.....0...C?.M.c*|", "00000b30 22 98 fb b4 2c a0 b8 49 87 ba 13 ed 0b 30 c0 39 |\"...,..I.....0.9|", "00000b40 f5 5b 31 7d dd 54 89 a8 2d cf fd 79 b9 ef 3a 50 |.[1}.T..-..y..:P|", "00000b50 85 54 fd 5a de 21 8c 73 f5 e8 38 30 db dc 23 96 |.T.Z.!.s..80..#.|", "00000b60 c7 41 f1 94 d8 ed b6 85 ef fd 81 3f b0 a8 a1 6a |.A.........?...j|", "00000b70 41 7c f5 8d 3c 17 64 cf 80 37 66 78 02 e4 05 e5 |A|..<.d..7fx....|", "00000b80 80 00 87 c4 dc f2 37 fd ed f3 b7 a2 91 a8 90 04 |......7.........|", "00000b90 85 2c ac 52 3f d0 0a 39 20 b1 50 34 03 57 80 ed |.,.R?..9 .P4.W..|", "00000ba0 8b 60 87 1b f3 59 89 f8 a0 6d f8 fa 8d 70 b9 f2 |.`...Y...m...p..|", "00000bb0 57 3b cc db f4 6c 75 c3 b0 f0 54 86 e4 52 50 cc |W;...lu...T..RP.|", "00000bc0 97 8e fc 69 6b 26 ba e0 2e d5 10 8c 7b 84 8f 4e |...ik&......{..N|", "00000bd0 1b 40 c9 d5 f4 95 e5 c2 6c 91 75 eb 0e 85 18 3d |.@......l.u....=|", "00000be0 7f 57 24 de bd 05 fb bd 83 ac 08 30 40 f0 00 06 |.W$........0@...|", "00000bf0 8a 98 c7 dd 3b 08 c8 84 72 d4 88 9f f9 9d 96 9e |....;...r.......|", "00000c00 d3 15 05 95 db 03 fe c5 e3 7f 31 68 ab 40 29 b9 |..........1h.@).|", "00000c10 ff 66 87 cc 08 cc 49 9f 5f e4 24 de 12 fd 60 d7 |.f....I._.$...`.|", "00000c20 0f 3e a0 7d ca 34 91 29 1c d9 88 6d 9d fb 56 16 |.>.}.4.)...m..V.|", "00000c30 bd 55 0c c7 1b f2 4a 2d eb ab 69 8f f3 fa 83 dc |.U....J-..i.....|", "00000c40 ae 6a 58 9a 4c c7 a6 50 88 fd 15 d5 df 1f a9 9b |.jX.L..P........|", "00000c50 91 c1 2e 79 c1 e0 1f 42 2e 48 42 db 9c be 5f 1c |...y...B.HB..._.|", "00000c60 31 88 dc ad db 6d f6 27 45 bc 03 57 c6 f1 26 0b |1....m.'E..W..&.|", "00000c70 f2 1f 1d 42 44 c3 53 cd 77 e5 e2 90 64 c1 1f 77 |...BD.S.w...d..w|", "00000c80 ff ef 02 00 54 21 0c 6a a2 d4 4b 9c d6 3c e8 99 |....T!.j..K..<..|", "00000c90 e9 64 8c 10 7e ba b7 fd e6 68 62 b9 06 3c d8 c1 |.d..~....hb..<..|", "00000ca0 5e ca 51 dd 60 df 99 28 0a 16 30 58 8d 5f 58 1d |^.Q.`..(..0X._X.|", "00000cb0 47 3a 49 de 5f 11 3f d0 f3 05 d4 9f 97 a4 da 31 |G:I._.?........1|", "00000cc0 7e 25 73 f7 0b ef 8f e8 c4 3d 57 ad 77 6c 78 56 |~%s......=W.wlxV|", "00000cd0 9c 8b a5 10 1a 32 f1 75 ca 9e 06 59 da 58 d6 3f |.....2.u...Y.X.?|", "00000ce0 43 97 4d c7 b5 da 41 f8 8e b4 1f 4c 18 d9 07 c2 |C.M...A....L....|", "00000cf0 68 85 a0 94 a9 38 a6 b9 cf 16 5f 36 fd 3f 2b da |h....8...._6.?+.|", "00000d00 87 d6 29 95 91 71 a3 47 5e 83 8d fe ba e6 a3 56 |..)..q.G^......V|", "00000d10 c4 73 c5 d2 d6 ea 0c 6f c7 c1 b8 84 68 a7 9e de |.s.....o....h...|", "00000d20 92 f7 dd a0 91 ca 78 79 65 88 90 b7 7e eb 9a 60 |......xye...~..`|", "00000d30 72 2b 29 e3 56 04 6b 4b e7 38 27 bb ad 93 99 13 |r+).V.kK.8'.....|", "00000d40 fe df 5a e2 5f 66 41 f3 99 11 ea 68 18 83 4c 3d |..Z._fA....h..L=|", "00000d50 a8 01 8a 6b a8 aa f6 3f 7f 76 06 6b ad 04 25 fe |...k...?.v.k..%.|", "00000d60 b4 ab 49 04 21 a3 e3 e3 03 c2 ba 32 04 16 48 e5 |..I.!......2..H.|", "00000d70 d7 8c 2f 52 e6 3f 8d b8 94 60 e8 04 e9 34 9e fd |../R.?...`...4..|", "00000d80 fa 41 12 13 da 54 7d 2f ff 6a 3e de 26 e1 9b 34 |.A...T}/.j>.&..4|", "00000d90 8e 26 2b 24 d0 eb b3 22 35 1a ad b2 70 04 9b d4 |.&+$...\"5...p...|", "00000da0 1e 90 46 90 de 74 9e fe a7 6e a8 59 a3 b9 b8 1a |..F..t...n.Y....|", "00000db0 a3 9b 65 e0 d2 64 53 4b 5e 42 20 41 9c 55 60 48 |..e..dSK^B A.U`H|", "00000dc0 b2 30 f7 f6 18 2b 71 64 cd 86 11 0e 0e a3 d5 f4 |.0...+qd........|", "00000dd0 b0 ca b9 b1 bd 76 cd 40 d2 ef 12 46 3c 28 ec f6 |.....v.@...F<(..|", "00000de0 33 96 dc 26 33 e0 08 46 f2 68 1e 43 a7 38 16 80 |3..&3..F.h.C.8..|", "00000df0 9c 6d 8b 28 c0 03 a7 99 22 61 0a 47 34 5e 5e 58 |.m.(....\"a.G4^^X|", "00000e00 67 67 db de df cf ff c0 05 dc 09 84 e2 28 60 2f |gg...........(`/|", "00000e10 f7 08 3d a8 8a 69 4a 0b 1c c5 44 f1 29 11 3e 71 |..=..iJ...D.).>q|", "00000e20 ae f3 d1 c2 37 d1 18 79 ab 64 8f 82 bf 38 7b 1b |....7..y.d...8{.|", "00000e30 52 ef 00 9a 4f 44 dc 4c 7c f9 c9 47 ba a8 6d fa |R...OD.L|..G..m.|", "00000e40 1d aa f5 ff c4 c2 0d 53 27 e6 17 d0 be 6d 97 30 |.......S'....m.0|", "00000e50 81 1c 9c c6 75 35 14 ff 73 d8 78 27 92 51 8e fc |....u5..s.x'.Q..|", "00000e60 94 ec 64 45 19 5c 0b be b5 99 e9 b9 76 5f 70 de |..dE.\\......v_p.|", "00000e70 00 b3 16 08 b0 76 06 a9 77 8a 79 c5 fc 0b f1 6c |.....v..w.y....l|", "00000e80 03 c1 0f a9 e9 70 19 ba fa b2 4f e2 13 33 6a 06 |.....p....O..3j.|", "00000e90 1c af 25 2c eb 94 3b 18 31 0a d0 b9 a6 13 53 7c |..%,..;.1.....S||", "00000ea0 3a 00 57 23 05 41 34 2c 7c 4f e2 f3 72 0b e2 a7 |:.W#.A4,|O..r...|", "00000eb0 e0 78 02 ab 7f 2c 0e 32 c3 d4 e4 b8 2b 2f 31 fb |.x...,.2....+/1.|", "00000ec0 f8 92 db 7a 41 4d 98 71 b9 a3 0d eb fd 71 fc 2e |...zAM.q.....q..|", "00000ed0 e8 5c a1 b5 58 f8 5a 11 20 04 da 52 39 b6 06 70 |.\\..X.Z. ..R9..p|", "00000ee0 43 0d 7a 48 14 18 21 03 2c 61 37 ef 23 41 b8 41 |C.zH..!.,a7.#A.A|", "00000ef0 3b e4 a0 f6 71 2a d2 06 97 b0 5a d2 81 36 b0 6f |;...q*....Z..6.o|", "00000f00 00 ed db 5f 96 bd 37 bd 7c bc 33 47 37 67 2a ad |..._..7.|.3G7g*.|", "00000f10 b6 81 e1 8d 5a e2 68 c2 0b 32 d3 f1 21 02 77 3b |....Z.h..2..!.w;|", "00000f20 89 ee da 0d 87 8f 26 0f 73 17 19 3b b7 04 e5 38 |......&.s..;...8|", "00000f30 54 c4 f3 1b 7b bc 93 9a ce 38 87 f2 3f a2 8b 38 |T...{....8..?..8|", "00000f40 a9 b1 7c 51 a2 34 f0 72 43 5c 59 35 b6 fd 50 2e |..|Q.4.rC\\Y5..P.|", "00000f50 db dc 29 d4 65 6a ef 23 83 7d 9d 49 b7 56 e4 c0 |..).ej.#.}.I.V..|", "00000f60 4e 4d d8 06 ca e5 85 ee 41 cb 66 84 1a af 79 7f |NM......A.f...y.|", "00000f70 f0 9e 7c 74 8c 4a 80 e2 e6 0a 58 fb c9 61 3f a1 |..|t.J....X..a?.|", "00000f80 b7 3b 7a ef 1f 91 95 d5 51 ab 01 ef 0e 86 c4 bb |.;z.....Q.......|", "00000f90 78 0e cb f3 b5 f9 30 d8 ae f0 81 11 e9 24 e6 e2 |x.....0......$..|", "00000fa0 67 8a 30 9c 04 a6 a1 cb d2 73 01 81 fb 01 28 f0 |g.0......s....(.|", "00000fb0 62 2d d5 91 24 9e ec f8 74 3e 26 63 46 a0 b1 c2 |b-..$...t>&cF...|", "00000fc0 63 31 0d 91 e8 fb 0d 6d 43 e0 8b ce 6b 8b 77 a2 |c1.....mC...k.w.|", "00000fd0 63 56 28 c8 60 8e c1 53 ae 70 3d 06 54 b5 f4 9a |cV(.`..S.p=.T...|", "00000fe0 8a be 4c 43 9c f6 64 40 6e 97 2c 97 cd 74 c8 83 |..LC..d@n.,..t..|", "00000ff0 b3 0b 41 74 56 fc 31 93 a5 dd c2 15 8c dd 9d 43 |..AtV.1........C|", "00001000", "", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound: ", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db: ", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m"]
a simple retry with the exact same command then works.
#4 Updated by Igor Fedotov 2 months ago
Just made a topic for potential fix discussion at https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/