Bug #54019
OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
0%
Description
When upgrading to rook 1.8.3 (ceph 16.2.7) we experience issue's with the OSD initialization; basically only +/- 50% of the raw devices are actually picked up by ceph, i.e. get an OSD instance. All others are eventually ignored.
It seems that the first osd-(prepare) run fails but leaves the osd setup in an invalid state from which ceph can't recover by simply respawning rook operator or osd pods.
In the osd-prepare logging it looks like this eventually:
2022-01-24 16:50:57.929115 D | exec: Running command: udevadm info --query=property /dev/sdc1
2022-01-24 16:50:57.938006 D | exec: Running command: lsblk /dev/sdc1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:57.941195 D | exec: Running command: ceph-volume inventory --format json /dev/sdc1
2022-01-24 16:50:58.423904 I | cephosd: skipping device "sdc1": ["Has BlueStore device label"].
2022-01-24 16:50:58.423934 D | exec: Running command: udevadm info --query=property /dev/sdd1
2022-01-24 16:50:58.432716 D | exec: Running command: lsblk /dev/sdd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.436077 D | exec: Running command: ceph-volume inventory --format json /dev/sdd1
2022-01-24 16:50:58.888559 I | cephosd: skipping device "sdd1": ["Has BlueStore device label"].
2022-01-24 16:50:58.888590 D | exec: Running command: udevadm info --query=property /dev/sde1
2022-01-24 16:50:58.897610 D | exec: Running command: lsblk /dev/sde1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.901015 D | exec: Running command: ceph-volume inventory --format json /dev/sde1
2022-01-24 16:50:59.363622 I | cephosd: skipping device "sde1": ["Has BlueStore device label"].
2022-01-24 16:50:59.363646 I | cephosd: skipping 'dm' device "dm-0"
2022-01-24 16:50:59.370400 I | cephosd: configuring osd devices: {"Entries":{}}
2022-01-24 16:50:59.370426 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2022-01-24 16:50:59.370718 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
2022-01-24 16:50:59.741959 D | cephosd: {}
2022-01-24 16:50:59.742004 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2022-01-24 16:50:59.742038 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2022-01-24 16:51:00.581917 D | cephosd: {
"0f6d36fb-23ed-4888-9469-a5ab91370bd3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sdc1",
"osd_id": 1,
"osd_uuid": "0f6d36fb-23ed-4888-9469-a5ab91370bd3",
"type": "bluestore"
},
"51b3b924-f70f-4257-88cf-91d881faf5e3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sde1",
"osd_id": 4,
"osd_uuid": "51b3b924-f70f-4257-88cf-91d881faf5e3",
"type": "bluestore"
}
}
As can be seen sdd1 is just skipped/missing.
Basically the same issue is reported in https://github.com/rook/rook/issues/8023 but closed with assumption that ceph 16.2.6 provides a fix for this issue.
After testing i can confirm 16.2.7 still has this issue.
It seems our setup somehow triggers this issue:
- centos7 (kernel 3.10)
- 3 spinning disks per host
- rook 1.8.3
I can also report that by setting "bluefs_buffered_io = false" the issue doesn't occur; although we don't fully understand today the further implications of setting this option.
Operator logging is attached:
- operator.log std 16.2.7 experiencing the issue
- operator-no-buffering.log: same but with option set; doesn't show the issue.
I can easily reproduce this issue and can run additional debug statements; but since i'm fairly new to ceph i do need some guidenance what/how to debug.
Related issues
History
#1 Updated by Neha Ojha about 2 years ago
- Assignee set to Adam Kupczyk
#2 Updated by Satoru Takeuchi almost 2 years ago
FYI, some rook users hit this problem as well.
quoted from https://tracker.ceph.com/issues/51034#note-12
---
https://github.com/rook/rook/issues/9646#issuecomment-1021505103
```
2022-01-25 18:38:32.269143 I | op-osd: OSD orchestration status for node dev2-cmp1l is "failed"
2022-01-25 18:38:32.269193 E | op-osd: failed to provision OSD on node dev2-cmp1l. &{OSDs:[] Status:failed PvcBackedOSD:false Message:failed to configure devices: failed to initialize osd: failed to run ceph-volume raw command. Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f4a0ae55-0b96-4304-a4cc-2f2f3d13a341
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/sde1
Running command: /usr/bin/ln -s /dev/sde1 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 3
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph
stderr: 2022-01-25T18:38:30.769+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2022-01-25T18:38:30.786+0000 7f89304df080 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 6f36f879-3c86-4ae1-8b46-abf13cc438c7, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db erroring opening db:
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 �[0;31m * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0/: (5) Input/output error�[0m
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 11, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in init
self.main(self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
self.safe_prepare(self.args)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
self.prepare()
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
tmpfs,
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 68, in prepare_bluestore
db=db
File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 481, in osd_mkfs_bluestore
raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph: exit status 1}
```
Ths problem disappeared if `bluefs_buffered_io=false` was set.
https://github.com/rook/rook/issues/9646#issuecomment-1022021781
```
And i can confirm that:
- rook 1.8.3 / 16.2.7 still have this issue operator.log
- rook 1.8.3 / 16.2.7 + bluefs_buffered_io = false does not experience this issue. operator-no-buffering.log
```
And here is another report.
https://github.com/rook/rook/issues/10160
```
stderr: 2022-04-25T18:12:29.836+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 2022-04-25T18:12:30.024+0000 7f023a2a43c0 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 68ba3e0d-e42c-46b0-a51b-e8e6ab918ad6, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _open_db erroring opening db:
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (5) Input/output error
--> Was unable to complete a new OSD, will rollback changes
```
The log is very similar to the first report. He used v17.20. But it's not clear that he tried `bluefs_buffered_io=false`. I suspect that `bluefs_buffered_id=true` (it's the default) still has some problems.
---
#3 Updated by Christian Rohmann almost 2 years ago
I observed similar issues at random when deploying Ceph 16.2.7 via Ansible on a VM.
ceph-osd -i 4 --mkfs --osd-objectstore=bluestore --osd-uuid de84f0aa-601a-473a-ad0b-7a2d403d0588 --monmap /tmp/monmap
return code is 250 and STDERR:
2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump: 00000000 40 6a a8 52 08 5a 8a 2a 74 c2 c5 98 f9 c6 c8 12 |@j.R.Z.*t.......| 00000010 5c 45 ca 37 bb 49 87 6d 98 f7 46 07 d6 3c 48 40 |\\E.7.I.m..F..<H@| 00000020 af b9 65 99 4f be 0b 20 9e 29 d5 9b 10 b1 74 ca |..e.O.. .)....t.| 00000030 82 77 b4 c5 ab de 04 9c eb 90 7c ff ce 76 ac ed |.w........|..v..| 00000040 3e 43 93 72 da 75 de 41 ed a9 49 5a 13 7a da 72 |>C.r.u.A..IZ.z.r| 00000050 7e 8d 1c d3 5b 31 98 7d 48 ed da 00 63 e7 f1 d5 |~...[1.}H...c...| 00000060 d1 5c f1 7f 2a c5 af 7d 56 2f fb 16 d7 26 56 ce |.\\..*..}V/...&V.| 00000070 ed ec 11 04 c6 b6 52 df 18 64 49 75 65 e8 62 47 |......R..dIue.bG| 00000080 29 1f de 22 e3 b2 45 0f ec fb 7c d9 f9 9a 5a d8 |)..\"..E...|...Z.| 00000090 82 40 71 6f c3 b6 73 5f e1 ed 3f 3d fd 56 8c e8 |.@qo..s_..?=.V..| 000000a0 a8 6a bf 6c f2 bc fa ca 44 8e e6 b5 6f f0 d0 bc |.j.l....D...o...| 000000b0 8f a3 ae 35 44 d2 23 74 6b 08 d6 75 9f 74 b0 fb |...5D.#tk..u.t..| 000000c0 8a f0 07 d1 7c b2 3e fd c3 e1 a5 7c cf d3 62 a6 |....|.>....|..b.| 000000d0 5b 04 c7 6d 7d 70 d9 72 cf 1b ee ef c6 59 41 4a |[..m}p.r.....YAJ| 000000e0 de 0d de 51 d1 02 cd ae be 3f 7d fd 45 92 f8 63 |...Q.....?}.E..c| 000000f0 6d 41 19 0f c7 d3 0e cf 64 2c 59 fc 86 0a c3 ec |mA......d,Y.....| 00000100 29 5d 6f 13 6b 62 fc 80 f4 b4 ce 9b 53 81 36 ea |)]o.kb......S.6.| 00000110 5e 07 e0 61 20 d2 15 43 2a 7a 84 f1 a3 0f 9b 8d |^..a ..C*z......| 00000120 ba 42 01 94 df f3 d4 87 fc 45 5c 8a dd d2 28 26 |.B.......E\\...(&| 00000130 a7 35 50 ee 58 1e f3 22 36 1d da 76 2c 07 c9 28 |.5P.X..\"6..v,..(| 00000140 e1 cb 66 20 cb e4 dd bb 91 eb b0 4f 66 1f 89 69 |..f .......Of..i| 00000150 4a ec 47 e3 d2 f6 2f 59 18 39 29 42 14 ea f6 19 |J.G.../Y.9)B....| 00000160 45 a0 e2 44 ed 97 4d 1f d2 f2 70 b0 13 70 e3 5b |E..D..M...p..p.[| 00000170 95 bb 43 6d 9c 35 25 17 14 2d a5 30 46 e3 38 e4 |..Cm.5%..-.0F.8.| 00000180 55 63 06 15 1c 1e d3 2e fd 7f 3e 4b 18 07 ed 53 |Uc........>K...S| 00000190 0b 7e d8 dd 39 0a d2 53 23 10 ce 7a 55 53 5e cf |.~..9..S#..zUS^.| 000001a0 77 6d a8 f8 ef e5 8a 01 13 b0 ac ce 31 07 c7 19 |wm..........1...| 000001b0 69 0e 00 c6 26 88 7a e0 be f5 2c ac 2c 6b ce 8a |i...&.z...,.,k..| 000001c0 b3 82 fe fc 16 ea 46 9e eb 7e 54 f4 b1 4d 00 dc |......F..~T..M..| 000001d0 0e e9 5b 3a 8b 5a 49 f0 46 f6 31 a8 9b 39 ba 96 |..[:.ZI.F.1..9..| 000001e0 b4 ce 76 69 91 5b 74 36 e9 89 cc 67 7c 2e c7 6b |..vi.[t6...g|..k| 000001f0 2e b2 f1 b0 fc a6 c6 c1 c9 35 b5 be 23 c1 b3 94 |.........5..#...| 00000200 11 47 12 1a ea 5e 47 84 a5 92 6c 83 e3 e2 ff 8a |.G...^G...l.....| 00000210 ed ba 48 f8 41 17 d5 2e d2 64 3a 64 64 21 ef 48 |..H.A....d:dd!.H| 00000220 f8 78 b3 2a ca 0b 1e 7d 80 0b fb 1c b5 73 1b ee |.x.*...}.....s..| 00000230 0f a0 4d 6b d9 22 72 06 1c 5d 27 af f6 c3 52 a8 |..Mk.\"r..]'...R.| 00000240 47 ac bf 18 50 4a 64 29 3c 71 1b 9d 21 37 a9 f1 |G...PJd)<q..!7..| 00000250 ea 52 7a 25 48 ef 8d 45 97 65 49 8b 1c a0 c5 1d |.Rz%H..E.eI.....| 00000260 94 da dc 90 b3 fe 56 2f 5e de 0a 02 58 6b 6f e6 |......V/^...Xko.| 00000270 e3 0e 93 eb 98 9a 4d 35 4f 8b 50 53 9f f6 35 d5 |......M5O.PS..5.| 00000280 2d c6 c6 45 d4 38 1d 68 7d b7 46 e0 8a 60 1b ec |-..E.8.h}.F..`..| 00000290 28 64 22 67 8b d4 0e 7c 05 28 24 59 fb b6 e5 44 |(d\"g...|.($Y...D| 000002a0 9e f4 d2 a5 08 cb 3a 52 a5 51 c3 56 cd fb 29 0b |......:R.Q.V..).| 000002b0 8d bb 28 ab 06 c5 e8 9c b8 99 ee 42 a5 4f 1c 0c |..(........B.O..| 000002c0 5f 29 1e 2f cf b2 f6 dc 05 21 b1 68 d2 79 ae b2 |_)./.....!.h.y..| 000002d0 d9 38 5f f6 5d 08 60 44 c1 6b d8 b1 a9 41 b7 48 |.8_.].`D.k...A.H| 000002e0 44 7a 4d f0 67 ef d9 ca 2f d5 69 9b c3 bf 78 ff |DzM.g.../.i...x.| 000002f0 2e f4 a4 94 14 0d fa 7b 60 55 fb 75 fb 49 52 5f |.......{`U.u.IR_| 00000300 fc 22 ad 16 ac c2 80 e1 ab cf bf 7f 84 6b df 0e |.\"...........k..| 00000310 4c 38 56 02 1a 37 bf 07 b1 e9 ac dc a8 56 f2 bf |L8V..7.......V..| 00000320 59 f8 8c 9b 58 88 54 2f bf d5 5e 1a c4 93 ea f3 |Y...X.T/..^.....| 00000330 ab 56 f0 a5 7c 66 7c 83 dd 00 30 6f 84 37 72 6b |.V..|f|...0o.7rk| 00000340 6a 13 d8 a8 e2 49 c7 2c e8 b8 c1 9f 29 3e e0 12 |j....I.,....)>..| 00000350 63 5f 22 df bd f7 9b 25 7e e9 c2 ff 1c 86 f0 e7 |c_\"....%~.......| 00000360 a9 59 06 18 9d fa 20 eb 2f 72 31 c4 53 c2 dd 09 |.Y.... ./r1.S...| 00000370 65 3c 9d 5d 7b 51 57 28 de 8b a5 e1 4a 1f 64 2e |e<.]{QW(....J.d.| 00000380 37 49 7f bd 15 57 09 dc fb 74 40 68 00 62 f8 5d |7I...W...t@h.b.]| 00000390 99 2c 6e c7 e6 64 bc 3f 26 46 90 f8 69 95 3b 48 |.,n..d.?&F..i.;H| 000003a0 a7 45 80 09 2d e4 5f c3 89 8e a1 e9 50 c6 87 22 |.E..-._.....P..\"| 000003b0 04 b5 c6 87 a8 1d 96 ff 65 bd 93 b0 69 62 8d b7 |........e...ib..| 000003c0 5b 9e e1 34 25 b6 7f 0e e6 2f 79 b2 ae 85 fc c8 |[..4%..../y.....| 000003d0 cd 4d dd b3 86 79 89 c8 86 d3 a9 a5 5d aa 9c 0c |.M...y......]...| 000003e0 a0 c3 e1 01 cb c8 30 81 46 35 f8 6f 81 4c d3 b5 |......0.F5.o.L..| 000003f0 be a8 e3 29 96 8c bb 1c 7c 1f fb e6 51 98 b6 90 |...)....|...Q...| 00000400 c0 a2 fc 9e bb b1 52 f1 b1 42 09 58 12 22 ec 16 |......R..B.X.\"..| 00000410 5f 81 49 ee 99 0a 6a 88 e9 76 60 4d ee 58 83 d3 |_.I...j..v`M.X..| 00000420 98 5b 55 ec 59 e8 ab 1a f6 d8 4a 9c 4b e5 56 66 |.[U.Y.....J.K.Vf| 00000430 b1 e1 fd 91 59 20 ce 42 db 6b 3d 7c 6a 7e 5e 05 |....Y .B.k=|j~^.| 00000440 49 a9 35 e3 28 25 08 bf 37 10 cb 60 27 ab c9 17 |I.5.(%..7..`'...| 00000450 2b 0e e0 54 2a 5a 53 df 80 e6 ca bf 7a de 55 54 |+..T*ZS.....z.UT| 00000460 71 f8 ca 9e fc 46 b3 c3 7f e1 ae a5 cd 6b 51 17 |q....F.......kQ.| 00000470 b3 48 fa 00 b9 7a 82 f1 16 02 ab 78 1a ad 99 5f |.H...z.....x..._| 00000480 8c c9 1d 53 dc 0e ff 12 b6 ba 6f 0e fe fb f2 31 |...S......o....1| 00000490 77 e9 f5 17 3d db ab 75 3f ce 76 48 57 37 e3 b4 |w...=..u?.vHW7..| 000004a0 b0 cc ae 79 f2 50 d4 55 fc 8b 1c aa 14 38 b8 a1 |...y.P.U.....8..| 000004b0 78 7b 20 72 67 03 25 c7 e7 15 ff a8 86 f9 ec f7 |x{ rg.%.........| 000004c0 3b 2c 56 20 87 75 53 a4 51 10 d7 1e b8 ea 6d d9 |;,V .uS.Q.....m.| 000004d0 78 18 98 b0 65 e7 a2 df fa 0b f2 79 cb c4 51 98 |x...e......y..Q.| 000004e0 be d0 eb c4 72 b4 a8 48 0f aa 6e 1b dc ff cb 1d |....r..H..n.....| 000004f0 aa 34 ee 55 57 19 87 3c 69 67 f3 63 b6 61 48 6d |.4.UW..<ig.c.aHm| 00000500 37 7d 18 32 6b 74 f1 59 c5 11 7d 13 e5 4c 21 81 |7}.2kt.Y..}..L!.| 00000510 5d 79 8a cb 9a 41 7a f9 9d 0e e7 ca 38 66 f4 1a |]y...Az.....8f..| 00000520 b7 a1 cd 7e 4d b1 19 f3 5d 53 a9 8d 9e 83 ae 0b |...~M...]S......| 00000530 c3 87 cd 4a 6e fe af a0 61 95 30 ee 99 ec 34 9a |...Jn...a.0...4.| 00000540 ec ac 19 ff fe 97 69 54 22 cc 04 17 62 a8 ee 7c |......iT\"...b..|| 00000550 8f a4 a0 50 90 b9 fa e7 3f 5d 7a 2b f8 d1 83 8b |...P....?]z+....| 00000560 13 51 6c c4 a1 1c b1 5c de 83 a6 aa 43 d1 1d c1 |.Ql....\\....C...| 00000570 e9 e6 60 c0 f2 2c 35 42 b7 71 a4 00 3a b2 db 36 |..`..,5B.q..:..6| 00000580 73 40 12 90 ee 73 b6 d5 84 7b 4a c2 73 1e 2e af |s@...s...{J.s...| 00000590 33 53 82 67 89 10 cb 85 5a 96 d3 af 47 d9 47 07 |3S.g....Z...G.G.| 000005a0 18 e0 ba 7a aa 0c 69 08 d5 d2 60 a1 2a 38 8e 3e |...z..i...`.*8.>| 000005b0 dd de a1 d1 24 32 63 36 55 31 d4 7c 51 1e 83 ea |....$2c6U1.|Q...| 000005c0 4d f4 f0 ef 75 0b 89 fa 17 26 fa 81 67 53 36 57 |M...u....&..gS6W| 000005d0 87 6f ee 61 0d 65 d3 3c ba b1 04 0c ae 27 c3 7c |.o.a.e.<.....'.|| 000005e0 19 ce 18 e3 cd 60 37 b4 b1 f7 f4 8b c2 fa 42 e5 |.....`7.......B.| 000005f0 a6 7f f2 2f 39 c9 94 62 93 68 64 5d 8b 1a 40 9e |.../9..b.hd]..@.| 00000600 ab 75 b3 e8 88 4f 5d ea a9 8e 4b fb c5 51 ca 9c |.u...O]...K..Q..| 00000610 49 d2 fc 68 18 4a 22 ce 7d 1d 00 e7 52 61 5f 85 |I..h.J\".}...Ra_.| 00000620 ec 3c 13 74 a5 78 75 3e 14 f2 27 ed b2 5d 92 16 |.<.t.xu>..'..]..| 00000630 05 39 dc ee 8c 13 4c af 63 86 1c 3f ac 7e 81 67 |.9....L.c..?.~.g| 00000640 b0 d7 a2 f3 9f 31 73 ae 97 08 11 2b fa 99 5e c9 |.....1s....+..^.| 00000650 d1 39 4a 9e a4 bb d1 55 83 c2 f4 2f bd 7e 85 c9 |.9J....U.../.~..| 00000660 d9 34 c6 61 c6 1d 12 9c d4 e9 ec 05 4f 04 55 3a |.4.a........O.U:| 00000670 78 2f fa 33 02 87 97 e1 65 ef af 8e f0 b2 f9 3c |x/.3....e......<| 00000680 75 ca 39 cc 12 2e 58 6e bd 50 9f 46 83 4f 02 3e |u.9...Xn.P.F.O.>| 00000690 50 8d 0a 8f 11 a2 06 70 70 0f 94 91 2d c7 da c4 |P......pp...-...| 000006a0 9a 22 eb 9f 14 04 c4 d0 35 c7 c9 60 9d b0 10 c3 |.\"......5..`....| 000006b0 ab fd c7 31 a4 b7 23 78 10 17 3b de 05 c8 f1 1b |...1..#x..;.....| 000006c0 90 32 54 ff eb bc 1f e4 11 78 f9 9c 2f ab 49 75 |.2T......x../.Iu| 000006d0 9f 42 89 54 77 9a 0b 0c a7 e8 b2 18 85 cd 2a d0 |.B.Tw.........*.| 000006e0 39 a0 09 96 09 2c a5 34 ef e9 85 0b d8 ed 22 e0 |9....,.4......\".| 000006f0 7c 80 07 8b 12 ed 60 b1 e8 ec 4e 26 97 f0 50 e1 ||.....`...N&..P.| 00000700 2f 2b 08 b8 03 87 7a da 44 7d 93 16 b5 70 c7 a2 |/+....z.D}...p..| 00000710 2f 5b 34 10 5d dc f6 f4 46 8c 8e 5c 4c aa db 44 |/[4.]...F..\\L..D| 00000720 ce fa ea 39 cb 23 e8 0c f4 13 99 e4 20 c7 9c 6a |...9.#...... ..j| 00000730 9e 37 b0 e0 42 af b3 59 98 6b fd 66 61 85 6e dc |.7..B..Y.k.fa.n.| 00000740 7c 53 b4 0b 16 9f 92 07 8b 4e 8d fc 05 0b 4e 1a ||S.......N....N.| 00000750 c6 b8 e0 4f 1b 6d 4f 3a 9d 2c 7c 3e 0a 42 e4 ba |...O.mO:.,|>.B..| 00000760 3c 6f d3 f1 88 d2 e8 69 3a d0 dd c2 b2 15 17 d0 |<o.....i:.......| 00000770 89 88 8a e7 34 fa 57 07 b3 37 89 5f 39 4a 2d 02 |....4.W..7._9J-.| 00000780 58 e6 7b 91 1b 15 5a 44 6e 38 e5 4b ef 8c 42 00 |X.{...ZDn8.K..B.| 00000790 b3 31 26 d7 6a 79 b8 e6 ca 72 88 c6 e7 f8 55 4e |.1&.jy...r....UN| 000007a0 62 15 84 08 2b dc 6b 52 7f 9b 6d 7e 47 6f 35 ca |b...+.kR..m~Go5.| 000007b0 a5 4d 10 07 54 d2 86 ea 5e e0 33 36 72 93 d9 6f |.M..T...^.36r..o| 000007c0 4f 09 19 63 8c 6b 04 8f 6d a9 c6 29 09 e2 63 61 |O..c.k..m..)..ca| 000007d0 6a 74 83 e5 9d 6c c1 b3 31 20 2b 97 34 88 d5 66 |jt...l..1 +.4..f| 000007e0 7c a4 f8 fa 04 b1 4e 2e 1c 81 23 79 9a 09 92 11 ||.....N...#y....| 000007f0 94 ba 59 25 7f ea a6 02 0d c1 0b f3 f3 b6 52 88 |..Y%..........R.| 00000800 fb 00 ac e4 bc 43 6a a2 ff 9b 5f c9 72 ad fc 3d |.....Cj..._.r..=| 00000810 02 41 2a 4e 36 d5 7f 19 b3 34 f2 7a 5a d7 42 62 |.A*N6....4.zZ.Bb| 00000820 4b 6e 52 2a 63 b7 fd 1f b9 97 18 88 53 7d d4 f5 |KnR*c.......S}..| 00000830 21 04 f4 ae bd 9d ea f7 44 7b f5 49 1a bf 0c 8c |!.......D{.I....| 00000840 15 0a fd b1 35 9c 57 3b 8f 2a 71 e6 40 cf 93 c8 |....5.W;.*q.@...| 00000850 b2 80 1d 8f 19 46 b7 16 ad bf f9 59 16 f5 1b 8d |.....F.....Y....| 00000860 aa ea 1e b2 d2 34 92 c2 aa 69 d0 81 2e 50 23 db |.....4...i...P#.| 00000870 29 f4 a7 cb b2 00 1d e5 a7 12 be d9 36 35 fe 6d |)...........65.m| 00000880 b2 97 b5 0c 84 3e 7b be 46 b5 50 9e 0c 3c a0 3a |.....>{.F.P..<.:| 00000890 d3 d3 d8 e8 16 c2 3e c1 ee 79 43 06 95 ad eb a3 |......>..yC.....| 000008a0 cd e3 6f 5b f2 9f 3f 12 69 2f eb 25 2f 14 2f 3d |..o[..?.i/.%/./=| 000008b0 aa 0f f8 fa e1 0b 6c 4c 55 a5 9d 4a 87 3b 8c 2c |......lLU..J.;.,| 000008c0 65 bc 0c b8 10 6a ed 82 bc 6e 98 6e 15 82 c8 ba |e....j...n.n....| 000008d0 83 9a 43 19 88 c4 56 3d f9 56 10 bd c0 58 5e 1b |..C...V=.V...X^.| 000008e0 14 9a d2 56 56 c7 fc 66 59 ff 81 77 1d 29 fe 0c |...VV..fY..w.)..| 000008f0 82 ca 17 75 d7 65 8f 04 3a 5b c9 af cf f3 5c 18 |...u.e..:[....\\.| 00000900 f8 57 1d 33 3b 73 7a a3 f4 a8 f0 a9 e3 aa bc e5 |.W.3;sz.........| 00000910 db 15 cb 51 d8 54 b2 f0 e2 e6 7d 39 59 8a 3b ec |...Q.T....}9Y.;.| 00000920 ee 2e bf db ed d5 33 09 6e 26 d2 30 7f b9 58 be |......3.n&.0..X.| 00000930 87 06 b1 ac 8a 07 2d b8 ee d1 92 55 21 5d d2 c1 |......-....U!]..| 00000940 2b 5b 4b e7 36 0b f7 1f 8d 96 0e 76 25 01 80 0d |+[K.6......v%...| 00000950 0e 93 98 43 c4 b9 95 e4 a6 28 e3 fc b8 b9 5b e9 |...C.....(....[.| 00000960 af 18 a6 bc 4d dc 2e 83 86 14 5c 95 c2 1b e3 bf |....M.....\\.....| 00000970 17 6e 72 8b 48 8b ed 1f 7a 99 b5 4a ad 8e 60 62 |.nr.H...z..J..`b| 00000980 40 2e 66 c5 8b 54 2f 3f df ac 0d 21 01 65 ad 53 |@.f..T/?...!.e.S| 00000990 d6 30 ce ef 8e fc f4 97 94 4e f2 52 f7 39 ef ee |.0.......N.R.9..| 000009a0 b7 e7 31 01 54 6c 9f d0 6a cc 22 42 8c c3 f8 a8 |..1.Tl..j.\"B....| 000009b0 9b b9 fb 1e e5 ed a0 54 d4 6a e0 53 4e 3d f3 fc |.......T.j.SN=..| 000009c0 d2 f7 04 8c 62 01 bf 68 d4 05 5c 3a 01 7c 17 d5 |....b..h..\\:.|..| 000009d0 ea d2 4c ed 87 32 81 68 f8 4d 36 70 aa 31 86 75 |..L..2.h.M6p.1.u| 000009e0 76 f0 c6 21 f1 0e d8 4e 74 de 9e e7 74 bb f5 dd |v..!...Nt...t...| 000009f0 f5 9c 68 9f 44 c4 c6 d2 bd f9 fe 6d 63 66 5d 6e |..h.D......mcf]n| 00000a00 64 45 28 cb 5d 4c 0d 80 c7 8c 5b 83 53 77 4b 7c |dE(.]L....[.SwK|| 00000a10 f1 32 c1 f9 1f c5 11 e5 39 2a e6 a9 0c de 68 91 |.2......9*....h.| 00000a20 66 6b 38 04 ed fc d7 60 6e a7 29 7e 5b 13 47 fe |fk8....`n.)~[.G.| 00000a30 a7 e0 53 23 f5 aa 2a bd 91 d7 fd 02 ae f3 29 b1 |..S#..*.......).| 00000a40 df 0b cd e0 47 c6 cf 20 03 3e 93 a0 25 bb 4c 13 |....G.. .>..%.L.| 00000a50 5c fc dc 5c 35 55 a3 f5 86 0b da 9a 1e 63 e4 de |\\..\\5U.......c..| 00000a60 bc 83 1d f3 af dc 49 e8 21 5f 30 6d 14 f2 cc 00 |......I.!_0m....| 00000a70 2d 7a 07 47 99 21 a9 98 bf 57 bb 99 f2 d4 32 38 |-z.G.!...W....28| 00000a80 a7 67 8a 5f 17 06 e1 1c 9d 5d 69 f0 76 64 3d 45 |.g._.....]i.vd=E| 00000a90 ad c0 e6 dc 7d e9 59 8d 9b 33 4c 04 a7 be 2d b5 |....}.Y..3L...-.| 00000aa0 b0 f5 26 14 86 b9 de 45 f4 d8 8a 59 27 db 38 56 |..&....E...Y'.8V| 00000ab0 7c 4b 1b 52 37 a7 9f 71 00 75 66 3a 2f 0a e3 f9 ||K.R7..q.uf:/...| 00000ac0 c7 21 dd 70 50 52 14 9f ff 7f 2f d5 21 2e c6 ba |.!.pPR..../.!...| 00000ad0 5c 3d 84 39 be ef 25 db f2 dd ec 53 63 1a 69 fd |\\=.9..%....Sc.i.| 00000ae0 be 71 59 a4 de df 11 3e ee ae c9 5d e8 75 1c cc |.qY....>...].u..| 00000af0 1c 00 db e6 42 1b ba 99 03 29 0b 83 02 12 bc 7e |....B....).....~| 00000b00 dd a8 7e 2a 68 41 71 d8 28 b1 9c cd a3 fa 6c 3a |..~*hAq.(.....l:| 00000b10 88 78 69 2d 94 6e dc 85 a6 d4 59 02 7f c2 2f 01 |.xi-.n....Y.../.| 00000b20 0f 06 03 ac ac 30 bb b6 84 43 3f 0c 4d 9a 63 2a |.....0...C?.M.c*| 00000b30 22 98 fb b4 2c a0 b8 49 87 ba 13 ed 0b 30 c0 39 |\"...,..I.....0.9| 00000b40 f5 5b 31 7d dd 54 89 a8 2d cf fd 79 b9 ef 3a 50 |.[1}.T..-..y..:P| 00000b50 85 54 fd 5a de 21 8c 73 f5 e8 38 30 db dc 23 96 |.T.Z.!.s..80..#.| 00000b60 c7 41 f1 94 d8 ed b6 85 ef fd 81 3f b0 a8 a1 6a |.A.........?...j| 00000b70 41 7c f5 8d 3c 17 64 cf 80 37 66 78 02 e4 05 e5 |A|..<.d..7fx....| 00000b80 80 00 87 c4 dc f2 37 fd ed f3 b7 a2 91 a8 90 04 |......7.........| 00000b90 85 2c ac 52 3f d0 0a 39 20 b1 50 34 03 57 80 ed |.,.R?..9 .P4.W..| 00000ba0 8b 60 87 1b f3 59 89 f8 a0 6d f8 fa 8d 70 b9 f2 |.`...Y...m...p..| 00000bb0 57 3b cc db f4 6c 75 c3 b0 f0 54 86 e4 52 50 cc |W;...lu...T..RP.| 00000bc0 97 8e fc 69 6b 26 ba e0 2e d5 10 8c 7b 84 8f 4e |...ik&......{..N| 00000bd0 1b 40 c9 d5 f4 95 e5 c2 6c 91 75 eb 0e 85 18 3d |.@......l.u....=| 00000be0 7f 57 24 de bd 05 fb bd 83 ac 08 30 40 f0 00 06 |.W$........0@...| 00000bf0 8a 98 c7 dd 3b 08 c8 84 72 d4 88 9f f9 9d 96 9e |....;...r.......| 00000c00 d3 15 05 95 db 03 fe c5 e3 7f 31 68 ab 40 29 b9 |..........1h.@).| 00000c10 ff 66 87 cc 08 cc 49 9f 5f e4 24 de 12 fd 60 d7 |.f....I._.$...`.| 00000c20 0f 3e a0 7d ca 34 91 29 1c d9 88 6d 9d fb 56 16 |.>.}.4.)...m..V.| 00000c30 bd 55 0c c7 1b f2 4a 2d eb ab 69 8f f3 fa 83 dc |.U....J-..i.....| 00000c40 ae 6a 58 9a 4c c7 a6 50 88 fd 15 d5 df 1f a9 9b |.jX.L..P........| 00000c50 91 c1 2e 79 c1 e0 1f 42 2e 48 42 db 9c be 5f 1c |...y...B.HB..._.| 00000c60 31 88 dc ad db 6d f6 27 45 bc 03 57 c6 f1 26 0b |1....m.'E..W..&.| 00000c70 f2 1f 1d 42 44 c3 53 cd 77 e5 e2 90 64 c1 1f 77 |...BD.S.w...d..w| 00000c80 ff ef 02 00 54 21 0c 6a a2 d4 4b 9c d6 3c e8 99 |....T!.j..K..<..| 00000c90 e9 64 8c 10 7e ba b7 fd e6 68 62 b9 06 3c d8 c1 |.d..~....hb..<..| 00000ca0 5e ca 51 dd 60 df 99 28 0a 16 30 58 8d 5f 58 1d |^.Q.`..(..0X._X.| 00000cb0 47 3a 49 de 5f 11 3f d0 f3 05 d4 9f 97 a4 da 31 |G:I._.?........1| 00000cc0 7e 25 73 f7 0b ef 8f e8 c4 3d 57 ad 77 6c 78 56 |~%s......=W.wlxV| 00000cd0 9c 8b a5 10 1a 32 f1 75 ca 9e 06 59 da 58 d6 3f |.....2.u...Y.X.?| 00000ce0 43 97 4d c7 b5 da 41 f8 8e b4 1f 4c 18 d9 07 c2 |C.M...A....L....| 00000cf0 68 85 a0 94 a9 38 a6 b9 cf 16 5f 36 fd 3f 2b da |h....8...._6.?+.| 00000d00 87 d6 29 95 91 71 a3 47 5e 83 8d fe ba e6 a3 56 |..)..q.G^......V| 00000d10 c4 73 c5 d2 d6 ea 0c 6f c7 c1 b8 84 68 a7 9e de |.s.....o....h...| 00000d20 92 f7 dd a0 91 ca 78 79 65 88 90 b7 7e eb 9a 60 |......xye...~..`| 00000d30 72 2b 29 e3 56 04 6b 4b e7 38 27 bb ad 93 99 13 |r+).V.kK.8'.....| 00000d40 fe df 5a e2 5f 66 41 f3 99 11 ea 68 18 83 4c 3d |..Z._fA....h..L=| 00000d50 a8 01 8a 6b a8 aa f6 3f 7f 76 06 6b ad 04 25 fe |...k...?.v.k..%.| 00000d60 b4 ab 49 04 21 a3 e3 e3 03 c2 ba 32 04 16 48 e5 |..I.!......2..H.| 00000d70 d7 8c 2f 52 e6 3f 8d b8 94 60 e8 04 e9 34 9e fd |../R.?...`...4..| 00000d80 fa 41 12 13 da 54 7d 2f ff 6a 3e de 26 e1 9b 34 |.A...T}/.j>.&..4| 00000d90 8e 26 2b 24 d0 eb b3 22 35 1a ad b2 70 04 9b d4 |.&+$...\"5...p...| 00000da0 1e 90 46 90 de 74 9e fe a7 6e a8 59 a3 b9 b8 1a |..F..t...n.Y....| 00000db0 a3 9b 65 e0 d2 64 53 4b 5e 42 20 41 9c 55 60 48 |..e..dSK^B A.U`H| 00000dc0 b2 30 f7 f6 18 2b 71 64 cd 86 11 0e 0e a3 d5 f4 |.0...+qd........| 00000dd0 b0 ca b9 b1 bd 76 cd 40 d2 ef 12 46 3c 28 ec f6 |.....v.@...F<(..| 00000de0 33 96 dc 26 33 e0 08 46 f2 68 1e 43 a7 38 16 80 |3..&3..F.h.C.8..| 00000df0 9c 6d 8b 28 c0 03 a7 99 22 61 0a 47 34 5e 5e 58 |.m.(....\"a.G4^^X| 00000e00 67 67 db de df cf ff c0 05 dc 09 84 e2 28 60 2f |gg...........(`/| 00000e10 f7 08 3d a8 8a 69 4a 0b 1c c5 44 f1 29 11 3e 71 |..=..iJ...D.).>q| 00000e20 ae f3 d1 c2 37 d1 18 79 ab 64 8f 82 bf 38 7b 1b |....7..y.d...8{.| 00000e30 52 ef 00 9a 4f 44 dc 4c 7c f9 c9 47 ba a8 6d fa |R...OD.L|..G..m.| 00000e40 1d aa f5 ff c4 c2 0d 53 27 e6 17 d0 be 6d 97 30 |.......S'....m.0| 00000e50 81 1c 9c c6 75 35 14 ff 73 d8 78 27 92 51 8e fc |....u5..s.x'.Q..| 00000e60 94 ec 64 45 19 5c 0b be b5 99 e9 b9 76 5f 70 de |..dE.\\......v_p.| 00000e70 00 b3 16 08 b0 76 06 a9 77 8a 79 c5 fc 0b f1 6c |.....v..w.y....l| 00000e80 03 c1 0f a9 e9 70 19 ba fa b2 4f e2 13 33 6a 06 |.....p....O..3j.| 00000e90 1c af 25 2c eb 94 3b 18 31 0a d0 b9 a6 13 53 7c |..%,..;.1.....S|| 00000ea0 3a 00 57 23 05 41 34 2c 7c 4f e2 f3 72 0b e2 a7 |:.W#.A4,|O..r...| 00000eb0 e0 78 02 ab 7f 2c 0e 32 c3 d4 e4 b8 2b 2f 31 fb |.x...,.2....+/1.| 00000ec0 f8 92 db 7a 41 4d 98 71 b9 a3 0d eb fd 71 fc 2e |...zAM.q.....q..| 00000ed0 e8 5c a1 b5 58 f8 5a 11 20 04 da 52 39 b6 06 70 |.\\..X.Z. ..R9..p| 00000ee0 43 0d 7a 48 14 18 21 03 2c 61 37 ef 23 41 b8 41 |C.zH..!.,a7.#A.A| 00000ef0 3b e4 a0 f6 71 2a d2 06 97 b0 5a d2 81 36 b0 6f |;...q*....Z..6.o| 00000f00 00 ed db 5f 96 bd 37 bd 7c bc 33 47 37 67 2a ad |..._..7.|.3G7g*.| 00000f10 b6 81 e1 8d 5a e2 68 c2 0b 32 d3 f1 21 02 77 3b |....Z.h..2..!.w;| 00000f20 89 ee da 0d 87 8f 26 0f 73 17 19 3b b7 04 e5 38 |......&.s..;...8| 00000f30 54 c4 f3 1b 7b bc 93 9a ce 38 87 f2 3f a2 8b 38 |T...{....8..?..8| 00000f40 a9 b1 7c 51 a2 34 f0 72 43 5c 59 35 b6 fd 50 2e |..|Q.4.rC\\Y5..P.| 00000f50 db dc 29 d4 65 6a ef 23 83 7d 9d 49 b7 56 e4 c0 |..).ej.#.}.I.V..| 00000f60 4e 4d d8 06 ca e5 85 ee 41 cb 66 84 1a af 79 7f |NM......A.f...y.| 00000f70 f0 9e 7c 74 8c 4a 80 e2 e6 0a 58 fb c9 61 3f a1 |..|t.J....X..a?.| 00000f80 b7 3b 7a ef 1f 91 95 d5 51 ab 01 ef 0e 86 c4 bb |.;z.....Q.......| 00000f90 78 0e cb f3 b5 f9 30 d8 ae f0 81 11 e9 24 e6 e2 |x.....0......$..| 00000fa0 67 8a 30 9c 04 a6 a1 cb d2 73 01 81 fb 01 28 f0 |g.0......s....(.| 00000fb0 62 2d d5 91 24 9e ec f8 74 3e 26 63 46 a0 b1 c2 |b-..$...t>&cF...| 00000fc0 63 31 0d 91 e8 fb 0d 6d 43 e0 8b ce 6b 8b 77 a2 |c1.....mC...k.w.| 00000fd0 63 56 28 c8 60 8e c1 53 ae 70 3d 06 54 b5 f4 9a |cV(.`..S.p=.T...| 00000fe0 8a be 4c 43 9c f6 64 40 6e 97 2c 97 cd 74 c8 83 |..LC..d@n.,..t..| 00000ff0 b3 0b 41 74 56 fc 31 93 a5 dd c2 15 8c dd 9d 43 |..AtV.1........C| 00001000 2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound: 2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db: 2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error 2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m 2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump: ", "00000000 40 6a a8 52 08 5a 8a 2a 74 c2 c5 98 f9 c6 c8 12 |@j.R.Z.*t.......|", "00000010 5c 45 ca 37 bb 49 87 6d 98 f7 46 07 d6 3c 48 40 |\\E.7.I.m..F..<H@|", "00000020 af b9 65 99 4f be 0b 20 9e 29 d5 9b 10 b1 74 ca |..e.O.. .)....t.|", "00000030 82 77 b4 c5 ab de 04 9c eb 90 7c ff ce 76 ac ed |.w........|..v..|", "00000040 3e 43 93 72 da 75 de 41 ed a9 49 5a 13 7a da 72 |>C.r.u.A..IZ.z.r|", "00000050 7e 8d 1c d3 5b 31 98 7d 48 ed da 00 63 e7 f1 d5 |~...[1.}H...c...|", "00000060 d1 5c f1 7f 2a c5 af 7d 56 2f fb 16 d7 26 56 ce |.\\..*..}V/...&V.|", "00000070 ed ec 11 04 c6 b6 52 df 18 64 49 75 65 e8 62 47 |......R..dIue.bG|", "00000080 29 1f de 22 e3 b2 45 0f ec fb 7c d9 f9 9a 5a d8 |)..\"..E...|...Z.|", "00000090 82 40 71 6f c3 b6 73 5f e1 ed 3f 3d fd 56 8c e8 |.@qo..s_..?=.V..|", "000000a0 a8 6a bf 6c f2 bc fa ca 44 8e e6 b5 6f f0 d0 bc |.j.l....D...o...|", "000000b0 8f a3 ae 35 44 d2 23 74 6b 08 d6 75 9f 74 b0 fb |...5D.#tk..u.t..|", "000000c0 8a f0 07 d1 7c b2 3e fd c3 e1 a5 7c cf d3 62 a6 |....|.>....|..b.|", "000000d0 5b 04 c7 6d 7d 70 d9 72 cf 1b ee ef c6 59 41 4a |[..m}p.r.....YAJ|", "000000e0 de 0d de 51 d1 02 cd ae be 3f 7d fd 45 92 f8 63 |...Q.....?}.E..c|", "000000f0 6d 41 19 0f c7 d3 0e cf 64 2c 59 fc 86 0a c3 ec |mA......d,Y.....|", "00000100 29 5d 6f 13 6b 62 fc 80 f4 b4 ce 9b 53 81 36 ea |)]o.kb......S.6.|", "00000110 5e 07 e0 61 20 d2 15 43 2a 7a 84 f1 a3 0f 9b 8d |^..a ..C*z......|", "00000120 ba 42 01 94 df f3 d4 87 fc 45 5c 8a dd d2 28 26 |.B.......E\\...(&|", "00000130 a7 35 50 ee 58 1e f3 22 36 1d da 76 2c 07 c9 28 |.5P.X..\"6..v,..(|", "00000140 e1 cb 66 20 cb e4 dd bb 91 eb b0 4f 66 1f 89 69 |..f .......Of..i|", "00000150 4a ec 47 e3 d2 f6 2f 59 18 39 29 42 14 ea f6 19 |J.G.../Y.9)B....|", "00000160 45 a0 e2 44 ed 97 4d 1f d2 f2 70 b0 13 70 e3 5b |E..D..M...p..p.[|", "00000170 95 bb 43 6d 9c 35 25 17 14 2d a5 30 46 e3 38 e4 |..Cm.5%..-.0F.8.|", "00000180 55 63 06 15 1c 1e d3 2e fd 7f 3e 4b 18 07 ed 53 |Uc........>K...S|", "00000190 0b 7e d8 dd 39 0a d2 53 23 10 ce 7a 55 53 5e cf |.~..9..S#..zUS^.|", "000001a0 77 6d a8 f8 ef e5 8a 01 13 b0 ac ce 31 07 c7 19 |wm..........1...|", "000001b0 69 0e 00 c6 26 88 7a e0 be f5 2c ac 2c 6b ce 8a |i...&.z...,.,k..|", "000001c0 b3 82 fe fc 16 ea 46 9e eb 7e 54 f4 b1 4d 00 dc |......F..~T..M..|", "000001d0 0e e9 5b 3a 8b 5a 49 f0 46 f6 31 a8 9b 39 ba 96 |..[:.ZI.F.1..9..|", "000001e0 b4 ce 76 69 91 5b 74 36 e9 89 cc 67 7c 2e c7 6b |..vi.[t6...g|..k|", "000001f0 2e b2 f1 b0 fc a6 c6 c1 c9 35 b5 be 23 c1 b3 94 |.........5..#...|", "00000200 11 47 12 1a ea 5e 47 84 a5 92 6c 83 e3 e2 ff 8a |.G...^G...l.....|", "00000210 ed ba 48 f8 41 17 d5 2e d2 64 3a 64 64 21 ef 48 |..H.A....d:dd!.H|", "00000220 f8 78 b3 2a ca 0b 1e 7d 80 0b fb 1c b5 73 1b ee |.x.*...}.....s..|", "00000230 0f a0 4d 6b d9 22 72 06 1c 5d 27 af f6 c3 52 a8 |..Mk.\"r..]'...R.|", "00000240 47 ac bf 18 50 4a 64 29 3c 71 1b 9d 21 37 a9 f1 |G...PJd)<q..!7..|", "00000250 ea 52 7a 25 48 ef 8d 45 97 65 49 8b 1c a0 c5 1d |.Rz%H..E.eI.....|", "00000260 94 da dc 90 b3 fe 56 2f 5e de 0a 02 58 6b 6f e6 |......V/^...Xko.|", "00000270 e3 0e 93 eb 98 9a 4d 35 4f 8b 50 53 9f f6 35 d5 |......M5O.PS..5.|", "00000280 2d c6 c6 45 d4 38 1d 68 7d b7 46 e0 8a 60 1b ec |-..E.8.h}.F..`..|", "00000290 28 64 22 67 8b d4 0e 7c 05 28 24 59 fb b6 e5 44 |(d\"g...|.($Y...D|", "000002a0 9e f4 d2 a5 08 cb 3a 52 a5 51 c3 56 cd fb 29 0b |......:R.Q.V..).|", "000002b0 8d bb 28 ab 06 c5 e8 9c b8 99 ee 42 a5 4f 1c 0c |..(........B.O..|", "000002c0 5f 29 1e 2f cf b2 f6 dc 05 21 b1 68 d2 79 ae b2 |_)./.....!.h.y..|", "000002d0 d9 38 5f f6 5d 08 60 44 c1 6b d8 b1 a9 41 b7 48 |.8_.].`D.k...A.H|", "000002e0 44 7a 4d f0 67 ef d9 ca 2f d5 69 9b c3 bf 78 ff |DzM.g.../.i...x.|", "000002f0 2e f4 a4 94 14 0d fa 7b 60 55 fb 75 fb 49 52 5f |.......{`U.u.IR_|", "00000300 fc 22 ad 16 ac c2 80 e1 ab cf bf 7f 84 6b df 0e |.\"...........k..|", "00000310 4c 38 56 02 1a 37 bf 07 b1 e9 ac dc a8 56 f2 bf |L8V..7.......V..|", "00000320 59 f8 8c 9b 58 88 54 2f bf d5 5e 1a c4 93 ea f3 |Y...X.T/..^.....|", "00000330 ab 56 f0 a5 7c 66 7c 83 dd 00 30 6f 84 37 72 6b |.V..|f|...0o.7rk|", "00000340 6a 13 d8 a8 e2 49 c7 2c e8 b8 c1 9f 29 3e e0 12 |j....I.,....)>..|", "00000350 63 5f 22 df bd f7 9b 25 7e e9 c2 ff 1c 86 f0 e7 |c_\"....%~.......|", "00000360 a9 59 06 18 9d fa 20 eb 2f 72 31 c4 53 c2 dd 09 |.Y.... ./r1.S...|", "00000370 65 3c 9d 5d 7b 51 57 28 de 8b a5 e1 4a 1f 64 2e |e<.]{QW(....J.d.|", "00000380 37 49 7f bd 15 57 09 dc fb 74 40 68 00 62 f8 5d |7I...W...t@h.b.]|", "00000390 99 2c 6e c7 e6 64 bc 3f 26 46 90 f8 69 95 3b 48 |.,n..d.?&F..i.;H|", "000003a0 a7 45 80 09 2d e4 5f c3 89 8e a1 e9 50 c6 87 22 |.E..-._.....P..\"|", "000003b0 04 b5 c6 87 a8 1d 96 ff 65 bd 93 b0 69 62 8d b7 |........e...ib..|", "000003c0 5b 9e e1 34 25 b6 7f 0e e6 2f 79 b2 ae 85 fc c8 |[..4%..../y.....|", "000003d0 cd 4d dd b3 86 79 89 c8 86 d3 a9 a5 5d aa 9c 0c |.M...y......]...|", "000003e0 a0 c3 e1 01 cb c8 30 81 46 35 f8 6f 81 4c d3 b5 |......0.F5.o.L..|", "000003f0 be a8 e3 29 96 8c bb 1c 7c 1f fb e6 51 98 b6 90 |...)....|...Q...|", "00000400 c0 a2 fc 9e bb b1 52 f1 b1 42 09 58 12 22 ec 16 |......R..B.X.\"..|", "00000410 5f 81 49 ee 99 0a 6a 88 e9 76 60 4d ee 58 83 d3 |_.I...j..v`M.X..|", "00000420 98 5b 55 ec 59 e8 ab 1a f6 d8 4a 9c 4b e5 56 66 |.[U.Y.....J.K.Vf|", "00000430 b1 e1 fd 91 59 20 ce 42 db 6b 3d 7c 6a 7e 5e 05 |....Y .B.k=|j~^.|", "00000440 49 a9 35 e3 28 25 08 bf 37 10 cb 60 27 ab c9 17 |I.5.(%..7..`'...|", "00000450 2b 0e e0 54 2a 5a 53 df 80 e6 ca bf 7a de 55 54 |+..T*ZS.....z.UT|", "00000460 71 f8 ca 9e fc 46 b3 c3 7f e1 ae a5 cd 6b 51 17 |q....F.......kQ.|", "00000470 b3 48 fa 00 b9 7a 82 f1 16 02 ab 78 1a ad 99 5f |.H...z.....x..._|", "00000480 8c c9 1d 53 dc 0e ff 12 b6 ba 6f 0e fe fb f2 31 |...S......o....1|", "00000490 77 e9 f5 17 3d db ab 75 3f ce 76 48 57 37 e3 b4 |w...=..u?.vHW7..|", "000004a0 b0 cc ae 79 f2 50 d4 55 fc 8b 1c aa 14 38 b8 a1 |...y.P.U.....8..|", "000004b0 78 7b 20 72 67 03 25 c7 e7 15 ff a8 86 f9 ec f7 |x{ rg.%.........|", "000004c0 3b 2c 56 20 87 75 53 a4 51 10 d7 1e b8 ea 6d d9 |;,V .uS.Q.....m.|", "000004d0 78 18 98 b0 65 e7 a2 df fa 0b f2 79 cb c4 51 98 |x...e......y..Q.|", "000004e0 be d0 eb c4 72 b4 a8 48 0f aa 6e 1b dc ff cb 1d |....r..H..n.....|", "000004f0 aa 34 ee 55 57 19 87 3c 69 67 f3 63 b6 61 48 6d |.4.UW..<ig.c.aHm|", "00000500 37 7d 18 32 6b 74 f1 59 c5 11 7d 13 e5 4c 21 81 |7}.2kt.Y..}..L!.|", "00000510 5d 79 8a cb 9a 41 7a f9 9d 0e e7 ca 38 66 f4 1a |]y...Az.....8f..|", "00000520 b7 a1 cd 7e 4d b1 19 f3 5d 53 a9 8d 9e 83 ae 0b |...~M...]S......|", "00000530 c3 87 cd 4a 6e fe af a0 61 95 30 ee 99 ec 34 9a |...Jn...a.0...4.|", "00000540 ec ac 19 ff fe 97 69 54 22 cc 04 17 62 a8 ee 7c |......iT\"...b..||", "00000550 8f a4 a0 50 90 b9 fa e7 3f 5d 7a 2b f8 d1 83 8b |...P....?]z+....|", "00000560 13 51 6c c4 a1 1c b1 5c de 83 a6 aa 43 d1 1d c1 |.Ql....\\....C...|", "00000570 e9 e6 60 c0 f2 2c 35 42 b7 71 a4 00 3a b2 db 36 |..`..,5B.q..:..6|", "00000580 73 40 12 90 ee 73 b6 d5 84 7b 4a c2 73 1e 2e af |s@...s...{J.s...|", "00000590 33 53 82 67 89 10 cb 85 5a 96 d3 af 47 d9 47 07 |3S.g....Z...G.G.|", "000005a0 18 e0 ba 7a aa 0c 69 08 d5 d2 60 a1 2a 38 8e 3e |...z..i...`.*8.>|", "000005b0 dd de a1 d1 24 32 63 36 55 31 d4 7c 51 1e 83 ea |....$2c6U1.|Q...|", "000005c0 4d f4 f0 ef 75 0b 89 fa 17 26 fa 81 67 53 36 57 |M...u....&..gS6W|", "000005d0 87 6f ee 61 0d 65 d3 3c ba b1 04 0c ae 27 c3 7c |.o.a.e.<.....'.||", "000005e0 19 ce 18 e3 cd 60 37 b4 b1 f7 f4 8b c2 fa 42 e5 |.....`7.......B.|", "000005f0 a6 7f f2 2f 39 c9 94 62 93 68 64 5d 8b 1a 40 9e |.../9..b.hd]..@.|", "00000600 ab 75 b3 e8 88 4f 5d ea a9 8e 4b fb c5 51 ca 9c |.u...O]...K..Q..|", "00000610 49 d2 fc 68 18 4a 22 ce 7d 1d 00 e7 52 61 5f 85 |I..h.J\".}...Ra_.|", "00000620 ec 3c 13 74 a5 78 75 3e 14 f2 27 ed b2 5d 92 16 |.<.t.xu>..'..]..|", "00000630 05 39 dc ee 8c 13 4c af 63 86 1c 3f ac 7e 81 67 |.9....L.c..?.~.g|", "00000640 b0 d7 a2 f3 9f 31 73 ae 97 08 11 2b fa 99 5e c9 |.....1s....+..^.|", "00000650 d1 39 4a 9e a4 bb d1 55 83 c2 f4 2f bd 7e 85 c9 |.9J....U.../.~..|", "00000660 d9 34 c6 61 c6 1d 12 9c d4 e9 ec 05 4f 04 55 3a |.4.a........O.U:|", "00000670 78 2f fa 33 02 87 97 e1 65 ef af 8e f0 b2 f9 3c |x/.3....e......<|", "00000680 75 ca 39 cc 12 2e 58 6e bd 50 9f 46 83 4f 02 3e |u.9...Xn.P.F.O.>|", "00000690 50 8d 0a 8f 11 a2 06 70 70 0f 94 91 2d c7 da c4 |P......pp...-...|", "000006a0 9a 22 eb 9f 14 04 c4 d0 35 c7 c9 60 9d b0 10 c3 |.\"......5..`....|", "000006b0 ab fd c7 31 a4 b7 23 78 10 17 3b de 05 c8 f1 1b |...1..#x..;.....|", "000006c0 90 32 54 ff eb bc 1f e4 11 78 f9 9c 2f ab 49 75 |.2T......x../.Iu|", "000006d0 9f 42 89 54 77 9a 0b 0c a7 e8 b2 18 85 cd 2a d0 |.B.Tw.........*.|", "000006e0 39 a0 09 96 09 2c a5 34 ef e9 85 0b d8 ed 22 e0 |9....,.4......\".|", "000006f0 7c 80 07 8b 12 ed 60 b1 e8 ec 4e 26 97 f0 50 e1 ||.....`...N&..P.|", "00000700 2f 2b 08 b8 03 87 7a da 44 7d 93 16 b5 70 c7 a2 |/+....z.D}...p..|", "00000710 2f 5b 34 10 5d dc f6 f4 46 8c 8e 5c 4c aa db 44 |/[4.]...F..\\L..D|", "00000720 ce fa ea 39 cb 23 e8 0c f4 13 99 e4 20 c7 9c 6a |...9.#...... ..j|", "00000730 9e 37 b0 e0 42 af b3 59 98 6b fd 66 61 85 6e dc |.7..B..Y.k.fa.n.|", "00000740 7c 53 b4 0b 16 9f 92 07 8b 4e 8d fc 05 0b 4e 1a ||S.......N....N.|", "00000750 c6 b8 e0 4f 1b 6d 4f 3a 9d 2c 7c 3e 0a 42 e4 ba |...O.mO:.,|>.B..|", "00000760 3c 6f d3 f1 88 d2 e8 69 3a d0 dd c2 b2 15 17 d0 |<o.....i:.......|", "00000770 89 88 8a e7 34 fa 57 07 b3 37 89 5f 39 4a 2d 02 |....4.W..7._9J-.|", "00000780 58 e6 7b 91 1b 15 5a 44 6e 38 e5 4b ef 8c 42 00 |X.{...ZDn8.K..B.|", "00000790 b3 31 26 d7 6a 79 b8 e6 ca 72 88 c6 e7 f8 55 4e |.1&.jy...r....UN|", "000007a0 62 15 84 08 2b dc 6b 52 7f 9b 6d 7e 47 6f 35 ca |b...+.kR..m~Go5.|", "000007b0 a5 4d 10 07 54 d2 86 ea 5e e0 33 36 72 93 d9 6f |.M..T...^.36r..o|", "000007c0 4f 09 19 63 8c 6b 04 8f 6d a9 c6 29 09 e2 63 61 |O..c.k..m..)..ca|", "000007d0 6a 74 83 e5 9d 6c c1 b3 31 20 2b 97 34 88 d5 66 |jt...l..1 +.4..f|", "000007e0 7c a4 f8 fa 04 b1 4e 2e 1c 81 23 79 9a 09 92 11 ||.....N...#y....|", "000007f0 94 ba 59 25 7f ea a6 02 0d c1 0b f3 f3 b6 52 88 |..Y%..........R.|", "00000800 fb 00 ac e4 bc 43 6a a2 ff 9b 5f c9 72 ad fc 3d |.....Cj..._.r..=|", "00000810 02 41 2a 4e 36 d5 7f 19 b3 34 f2 7a 5a d7 42 62 |.A*N6....4.zZ.Bb|", "00000820 4b 6e 52 2a 63 b7 fd 1f b9 97 18 88 53 7d d4 f5 |KnR*c.......S}..|", "00000830 21 04 f4 ae bd 9d ea f7 44 7b f5 49 1a bf 0c 8c |!.......D{.I....|", "00000840 15 0a fd b1 35 9c 57 3b 8f 2a 71 e6 40 cf 93 c8 |....5.W;.*q.@...|", "00000850 b2 80 1d 8f 19 46 b7 16 ad bf f9 59 16 f5 1b 8d |.....F.....Y....|", "00000860 aa ea 1e b2 d2 34 92 c2 aa 69 d0 81 2e 50 23 db |.....4...i...P#.|", "00000870 29 f4 a7 cb b2 00 1d e5 a7 12 be d9 36 35 fe 6d |)...........65.m|", "00000880 b2 97 b5 0c 84 3e 7b be 46 b5 50 9e 0c 3c a0 3a |.....>{.F.P..<.:|", "00000890 d3 d3 d8 e8 16 c2 3e c1 ee 79 43 06 95 ad eb a3 |......>..yC.....|", "000008a0 cd e3 6f 5b f2 9f 3f 12 69 2f eb 25 2f 14 2f 3d |..o[..?.i/.%/./=|", "000008b0 aa 0f f8 fa e1 0b 6c 4c 55 a5 9d 4a 87 3b 8c 2c |......lLU..J.;.,|", "000008c0 65 bc 0c b8 10 6a ed 82 bc 6e 98 6e 15 82 c8 ba |e....j...n.n....|", "000008d0 83 9a 43 19 88 c4 56 3d f9 56 10 bd c0 58 5e 1b |..C...V=.V...X^.|", "000008e0 14 9a d2 56 56 c7 fc 66 59 ff 81 77 1d 29 fe 0c |...VV..fY..w.)..|", "000008f0 82 ca 17 75 d7 65 8f 04 3a 5b c9 af cf f3 5c 18 |...u.e..:[....\\.|", "00000900 f8 57 1d 33 3b 73 7a a3 f4 a8 f0 a9 e3 aa bc e5 |.W.3;sz.........|", "00000910 db 15 cb 51 d8 54 b2 f0 e2 e6 7d 39 59 8a 3b ec |...Q.T....}9Y.;.|", "00000920 ee 2e bf db ed d5 33 09 6e 26 d2 30 7f b9 58 be |......3.n&.0..X.|", "00000930 87 06 b1 ac 8a 07 2d b8 ee d1 92 55 21 5d d2 c1 |......-....U!]..|", "00000940 2b 5b 4b e7 36 0b f7 1f 8d 96 0e 76 25 01 80 0d |+[K.6......v%...|", "00000950 0e 93 98 43 c4 b9 95 e4 a6 28 e3 fc b8 b9 5b e9 |...C.....(....[.|", "00000960 af 18 a6 bc 4d dc 2e 83 86 14 5c 95 c2 1b e3 bf |....M.....\\.....|", "00000970 17 6e 72 8b 48 8b ed 1f 7a 99 b5 4a ad 8e 60 62 |.nr.H...z..J..`b|", "00000980 40 2e 66 c5 8b 54 2f 3f df ac 0d 21 01 65 ad 53 |@.f..T/?...!.e.S|", "00000990 d6 30 ce ef 8e fc f4 97 94 4e f2 52 f7 39 ef ee |.0.......N.R.9..|", "000009a0 b7 e7 31 01 54 6c 9f d0 6a cc 22 42 8c c3 f8 a8 |..1.Tl..j.\"B....|", "000009b0 9b b9 fb 1e e5 ed a0 54 d4 6a e0 53 4e 3d f3 fc |.......T.j.SN=..|", "000009c0 d2 f7 04 8c 62 01 bf 68 d4 05 5c 3a 01 7c 17 d5 |....b..h..\\:.|..|", "000009d0 ea d2 4c ed 87 32 81 68 f8 4d 36 70 aa 31 86 75 |..L..2.h.M6p.1.u|", "000009e0 76 f0 c6 21 f1 0e d8 4e 74 de 9e e7 74 bb f5 dd |v..!...Nt...t...|", "000009f0 f5 9c 68 9f 44 c4 c6 d2 bd f9 fe 6d 63 66 5d 6e |..h.D......mcf]n|", "00000a00 64 45 28 cb 5d 4c 0d 80 c7 8c 5b 83 53 77 4b 7c |dE(.]L....[.SwK||", "00000a10 f1 32 c1 f9 1f c5 11 e5 39 2a e6 a9 0c de 68 91 |.2......9*....h.|", "00000a20 66 6b 38 04 ed fc d7 60 6e a7 29 7e 5b 13 47 fe |fk8....`n.)~[.G.|", "00000a30 a7 e0 53 23 f5 aa 2a bd 91 d7 fd 02 ae f3 29 b1 |..S#..*.......).|", "00000a40 df 0b cd e0 47 c6 cf 20 03 3e 93 a0 25 bb 4c 13 |....G.. .>..%.L.|", "00000a50 5c fc dc 5c 35 55 a3 f5 86 0b da 9a 1e 63 e4 de |\\..\\5U.......c..|", "00000a60 bc 83 1d f3 af dc 49 e8 21 5f 30 6d 14 f2 cc 00 |......I.!_0m....|", "00000a70 2d 7a 07 47 99 21 a9 98 bf 57 bb 99 f2 d4 32 38 |-z.G.!...W....28|", "00000a80 a7 67 8a 5f 17 06 e1 1c 9d 5d 69 f0 76 64 3d 45 |.g._.....]i.vd=E|", "00000a90 ad c0 e6 dc 7d e9 59 8d 9b 33 4c 04 a7 be 2d b5 |....}.Y..3L...-.|", "00000aa0 b0 f5 26 14 86 b9 de 45 f4 d8 8a 59 27 db 38 56 |..&....E...Y'.8V|", "00000ab0 7c 4b 1b 52 37 a7 9f 71 00 75 66 3a 2f 0a e3 f9 ||K.R7..q.uf:/...|", "00000ac0 c7 21 dd 70 50 52 14 9f ff 7f 2f d5 21 2e c6 ba |.!.pPR..../.!...|", "00000ad0 5c 3d 84 39 be ef 25 db f2 dd ec 53 63 1a 69 fd |\\=.9..%....Sc.i.|", "00000ae0 be 71 59 a4 de df 11 3e ee ae c9 5d e8 75 1c cc |.qY....>...].u..|", "00000af0 1c 00 db e6 42 1b ba 99 03 29 0b 83 02 12 bc 7e |....B....).....~|", "00000b00 dd a8 7e 2a 68 41 71 d8 28 b1 9c cd a3 fa 6c 3a |..~*hAq.(.....l:|", "00000b10 88 78 69 2d 94 6e dc 85 a6 d4 59 02 7f c2 2f 01 |.xi-.n....Y.../.|", "00000b20 0f 06 03 ac ac 30 bb b6 84 43 3f 0c 4d 9a 63 2a |.....0...C?.M.c*|", "00000b30 22 98 fb b4 2c a0 b8 49 87 ba 13 ed 0b 30 c0 39 |\"...,..I.....0.9|", "00000b40 f5 5b 31 7d dd 54 89 a8 2d cf fd 79 b9 ef 3a 50 |.[1}.T..-..y..:P|", "00000b50 85 54 fd 5a de 21 8c 73 f5 e8 38 30 db dc 23 96 |.T.Z.!.s..80..#.|", "00000b60 c7 41 f1 94 d8 ed b6 85 ef fd 81 3f b0 a8 a1 6a |.A.........?...j|", "00000b70 41 7c f5 8d 3c 17 64 cf 80 37 66 78 02 e4 05 e5 |A|..<.d..7fx....|", "00000b80 80 00 87 c4 dc f2 37 fd ed f3 b7 a2 91 a8 90 04 |......7.........|", "00000b90 85 2c ac 52 3f d0 0a 39 20 b1 50 34 03 57 80 ed |.,.R?..9 .P4.W..|", "00000ba0 8b 60 87 1b f3 59 89 f8 a0 6d f8 fa 8d 70 b9 f2 |.`...Y...m...p..|", "00000bb0 57 3b cc db f4 6c 75 c3 b0 f0 54 86 e4 52 50 cc |W;...lu...T..RP.|", "00000bc0 97 8e fc 69 6b 26 ba e0 2e d5 10 8c 7b 84 8f 4e |...ik&......{..N|", "00000bd0 1b 40 c9 d5 f4 95 e5 c2 6c 91 75 eb 0e 85 18 3d |.@......l.u....=|", "00000be0 7f 57 24 de bd 05 fb bd 83 ac 08 30 40 f0 00 06 |.W$........0@...|", "00000bf0 8a 98 c7 dd 3b 08 c8 84 72 d4 88 9f f9 9d 96 9e |....;...r.......|", "00000c00 d3 15 05 95 db 03 fe c5 e3 7f 31 68 ab 40 29 b9 |..........1h.@).|", "00000c10 ff 66 87 cc 08 cc 49 9f 5f e4 24 de 12 fd 60 d7 |.f....I._.$...`.|", "00000c20 0f 3e a0 7d ca 34 91 29 1c d9 88 6d 9d fb 56 16 |.>.}.4.)...m..V.|", "00000c30 bd 55 0c c7 1b f2 4a 2d eb ab 69 8f f3 fa 83 dc |.U....J-..i.....|", "00000c40 ae 6a 58 9a 4c c7 a6 50 88 fd 15 d5 df 1f a9 9b |.jX.L..P........|", "00000c50 91 c1 2e 79 c1 e0 1f 42 2e 48 42 db 9c be 5f 1c |...y...B.HB..._.|", "00000c60 31 88 dc ad db 6d f6 27 45 bc 03 57 c6 f1 26 0b |1....m.'E..W..&.|", "00000c70 f2 1f 1d 42 44 c3 53 cd 77 e5 e2 90 64 c1 1f 77 |...BD.S.w...d..w|", "00000c80 ff ef 02 00 54 21 0c 6a a2 d4 4b 9c d6 3c e8 99 |....T!.j..K..<..|", "00000c90 e9 64 8c 10 7e ba b7 fd e6 68 62 b9 06 3c d8 c1 |.d..~....hb..<..|", "00000ca0 5e ca 51 dd 60 df 99 28 0a 16 30 58 8d 5f 58 1d |^.Q.`..(..0X._X.|", "00000cb0 47 3a 49 de 5f 11 3f d0 f3 05 d4 9f 97 a4 da 31 |G:I._.?........1|", "00000cc0 7e 25 73 f7 0b ef 8f e8 c4 3d 57 ad 77 6c 78 56 |~%s......=W.wlxV|", "00000cd0 9c 8b a5 10 1a 32 f1 75 ca 9e 06 59 da 58 d6 3f |.....2.u...Y.X.?|", "00000ce0 43 97 4d c7 b5 da 41 f8 8e b4 1f 4c 18 d9 07 c2 |C.M...A....L....|", "00000cf0 68 85 a0 94 a9 38 a6 b9 cf 16 5f 36 fd 3f 2b da |h....8...._6.?+.|", "00000d00 87 d6 29 95 91 71 a3 47 5e 83 8d fe ba e6 a3 56 |..)..q.G^......V|", "00000d10 c4 73 c5 d2 d6 ea 0c 6f c7 c1 b8 84 68 a7 9e de |.s.....o....h...|", "00000d20 92 f7 dd a0 91 ca 78 79 65 88 90 b7 7e eb 9a 60 |......xye...~..`|", "00000d30 72 2b 29 e3 56 04 6b 4b e7 38 27 bb ad 93 99 13 |r+).V.kK.8'.....|", "00000d40 fe df 5a e2 5f 66 41 f3 99 11 ea 68 18 83 4c 3d |..Z._fA....h..L=|", "00000d50 a8 01 8a 6b a8 aa f6 3f 7f 76 06 6b ad 04 25 fe |...k...?.v.k..%.|", "00000d60 b4 ab 49 04 21 a3 e3 e3 03 c2 ba 32 04 16 48 e5 |..I.!......2..H.|", "00000d70 d7 8c 2f 52 e6 3f 8d b8 94 60 e8 04 e9 34 9e fd |../R.?...`...4..|", "00000d80 fa 41 12 13 da 54 7d 2f ff 6a 3e de 26 e1 9b 34 |.A...T}/.j>.&..4|", "00000d90 8e 26 2b 24 d0 eb b3 22 35 1a ad b2 70 04 9b d4 |.&+$...\"5...p...|", "00000da0 1e 90 46 90 de 74 9e fe a7 6e a8 59 a3 b9 b8 1a |..F..t...n.Y....|", "00000db0 a3 9b 65 e0 d2 64 53 4b 5e 42 20 41 9c 55 60 48 |..e..dSK^B A.U`H|", "00000dc0 b2 30 f7 f6 18 2b 71 64 cd 86 11 0e 0e a3 d5 f4 |.0...+qd........|", "00000dd0 b0 ca b9 b1 bd 76 cd 40 d2 ef 12 46 3c 28 ec f6 |.....v.@...F<(..|", "00000de0 33 96 dc 26 33 e0 08 46 f2 68 1e 43 a7 38 16 80 |3..&3..F.h.C.8..|", "00000df0 9c 6d 8b 28 c0 03 a7 99 22 61 0a 47 34 5e 5e 58 |.m.(....\"a.G4^^X|", "00000e00 67 67 db de df cf ff c0 05 dc 09 84 e2 28 60 2f |gg...........(`/|", "00000e10 f7 08 3d a8 8a 69 4a 0b 1c c5 44 f1 29 11 3e 71 |..=..iJ...D.).>q|", "00000e20 ae f3 d1 c2 37 d1 18 79 ab 64 8f 82 bf 38 7b 1b |....7..y.d...8{.|", "00000e30 52 ef 00 9a 4f 44 dc 4c 7c f9 c9 47 ba a8 6d fa |R...OD.L|..G..m.|", "00000e40 1d aa f5 ff c4 c2 0d 53 27 e6 17 d0 be 6d 97 30 |.......S'....m.0|", "00000e50 81 1c 9c c6 75 35 14 ff 73 d8 78 27 92 51 8e fc |....u5..s.x'.Q..|", "00000e60 94 ec 64 45 19 5c 0b be b5 99 e9 b9 76 5f 70 de |..dE.\\......v_p.|", "00000e70 00 b3 16 08 b0 76 06 a9 77 8a 79 c5 fc 0b f1 6c |.....v..w.y....l|", "00000e80 03 c1 0f a9 e9 70 19 ba fa b2 4f e2 13 33 6a 06 |.....p....O..3j.|", "00000e90 1c af 25 2c eb 94 3b 18 31 0a d0 b9 a6 13 53 7c |..%,..;.1.....S||", "00000ea0 3a 00 57 23 05 41 34 2c 7c 4f e2 f3 72 0b e2 a7 |:.W#.A4,|O..r...|", "00000eb0 e0 78 02 ab 7f 2c 0e 32 c3 d4 e4 b8 2b 2f 31 fb |.x...,.2....+/1.|", "00000ec0 f8 92 db 7a 41 4d 98 71 b9 a3 0d eb fd 71 fc 2e |...zAM.q.....q..|", "00000ed0 e8 5c a1 b5 58 f8 5a 11 20 04 da 52 39 b6 06 70 |.\\..X.Z. ..R9..p|", "00000ee0 43 0d 7a 48 14 18 21 03 2c 61 37 ef 23 41 b8 41 |C.zH..!.,a7.#A.A|", "00000ef0 3b e4 a0 f6 71 2a d2 06 97 b0 5a d2 81 36 b0 6f |;...q*....Z..6.o|", "00000f00 00 ed db 5f 96 bd 37 bd 7c bc 33 47 37 67 2a ad |..._..7.|.3G7g*.|", "00000f10 b6 81 e1 8d 5a e2 68 c2 0b 32 d3 f1 21 02 77 3b |....Z.h..2..!.w;|", "00000f20 89 ee da 0d 87 8f 26 0f 73 17 19 3b b7 04 e5 38 |......&.s..;...8|", "00000f30 54 c4 f3 1b 7b bc 93 9a ce 38 87 f2 3f a2 8b 38 |T...{....8..?..8|", "00000f40 a9 b1 7c 51 a2 34 f0 72 43 5c 59 35 b6 fd 50 2e |..|Q.4.rC\\Y5..P.|", "00000f50 db dc 29 d4 65 6a ef 23 83 7d 9d 49 b7 56 e4 c0 |..).ej.#.}.I.V..|", "00000f60 4e 4d d8 06 ca e5 85 ee 41 cb 66 84 1a af 79 7f |NM......A.f...y.|", "00000f70 f0 9e 7c 74 8c 4a 80 e2 e6 0a 58 fb c9 61 3f a1 |..|t.J....X..a?.|", "00000f80 b7 3b 7a ef 1f 91 95 d5 51 ab 01 ef 0e 86 c4 bb |.;z.....Q.......|", "00000f90 78 0e cb f3 b5 f9 30 d8 ae f0 81 11 e9 24 e6 e2 |x.....0......$..|", "00000fa0 67 8a 30 9c 04 a6 a1 cb d2 73 01 81 fb 01 28 f0 |g.0......s....(.|", "00000fb0 62 2d d5 91 24 9e ec f8 74 3e 26 63 46 a0 b1 c2 |b-..$...t>&cF...|", "00000fc0 63 31 0d 91 e8 fb 0d 6d 43 e0 8b ce 6b 8b 77 a2 |c1.....mC...k.w.|", "00000fd0 63 56 28 c8 60 8e c1 53 ae 70 3d 06 54 b5 f4 9a |cV(.`..S.p=.T...|", "00000fe0 8a be 4c 43 9c f6 64 40 6e 97 2c 97 cd 74 c8 83 |..LC..d@n.,..t..|", "00000ff0 b3 0b 41 74 56 fc 31 93 a5 dd c2 15 8c dd 9d 43 |..AtV.1........C|", "00001000", "", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound: ", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db: ", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m"]
a simple retry with the exact same command then works.
#4 Updated by Igor Fedotov over 1 year ago
Just made a topic for potential fix discussion at https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/
#5 Updated by Yuri Weinstein about 1 year ago
#6 Updated by Christian Rohmann about 1 year ago
Thanks for fixing this ... but somehow the link to the ML (https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/) with Igors discussion seems to 404 now.
#7 Updated by Igor Fedotov about 1 year ago
Christian Rohmann wrote:
Thanks for fixing this ... but somehow the link to the ML (https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/) with Igors discussion seems to 404 now.
Here is another link: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/message/K6WSU4NCELZHDNZKC43YNUDM2WGGIHO7/
We experienced some issues with the mail-archive a few weeks ago hence the update...
#8 Updated by Igor Fedotov about 1 year ago
- Status changed from New to Pending Backport
- Backport set to quincy, pacific
#9 Updated by Igor Fedotov about 1 year ago
- Pull request ID set to 49431
#10 Updated by Igor Fedotov about 1 year ago
- Backport changed from quincy, pacific to quincy, reef
#11 Updated by Backport Bot about 1 year ago
- Copied to Backport #58633: quincy: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error added
#12 Updated by Backport Bot about 1 year ago
- Tags changed from osd to osd backport_processed
#13 Updated by Richard Hesse about 1 year ago
Is there a nightly container build for Pacific that includes this fix? This issue has crippled my previously working 16.2.10 cluster. All of the simple volumes are fine. All of the LVM based volumes are unable to start.
#14 Updated by Radoslaw Zarzynski about 1 year ago
- Copied to Backport #58952: reef: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error added
#15 Updated by Igor Fedotov 12 months ago
- Status changed from Pending Backport to Resolved