Project

General

Profile

Bug #54019

OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error

Added by Paul Bormans about 2 years ago. Updated 12 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
osd backport_processed
Backport:
quincy, reef
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

When upgrading to rook 1.8.3 (ceph 16.2.7) we experience issue's with the OSD initialization; basically only +/- 50% of the raw devices are actually picked up by ceph, i.e. get an OSD instance. All others are eventually ignored.

It seems that the first osd-(prepare) run fails but leaves the osd setup in an invalid state from which ceph can't recover by simply respawning rook operator or osd pods.

In the osd-prepare logging it looks like this eventually:

2022-01-24 16:50:57.929115 D | exec: Running command: udevadm info --query=property /dev/sdc1
2022-01-24 16:50:57.938006 D | exec: Running command: lsblk /dev/sdc1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:57.941195 D | exec: Running command: ceph-volume inventory --format json /dev/sdc1
2022-01-24 16:50:58.423904 I | cephosd: skipping device "sdc1": ["Has BlueStore device label"].
2022-01-24 16:50:58.423934 D | exec: Running command: udevadm info --query=property /dev/sdd1
2022-01-24 16:50:58.432716 D | exec: Running command: lsblk /dev/sdd1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.436077 D | exec: Running command: ceph-volume inventory --format json /dev/sdd1
2022-01-24 16:50:58.888559 I | cephosd: skipping device "sdd1": ["Has BlueStore device label"].
2022-01-24 16:50:58.888590 D | exec: Running command: udevadm info --query=property /dev/sde1
2022-01-24 16:50:58.897610 D | exec: Running command: lsblk /dev/sde1 --bytes --nodeps --pairs --paths --output SIZE,ROTA,RO,TYPE,PKNAME,NAME,KNAME
2022-01-24 16:50:58.901015 D | exec: Running command: ceph-volume inventory --format json /dev/sde1
2022-01-24 16:50:59.363622 I | cephosd: skipping device "sde1": ["Has BlueStore device label"].
2022-01-24 16:50:59.363646 I | cephosd: skipping 'dm' device "dm-0"
2022-01-24 16:50:59.370400 I | cephosd: configuring osd devices: {"Entries":{}}
2022-01-24 16:50:59.370426 I | cephosd: no new devices to configure. returning devices already configured with ceph-volume.
2022-01-24 16:50:59.370718 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log lvm list --format json
2022-01-24 16:50:59.741959 D | cephosd: {}
2022-01-24 16:50:59.742004 I | cephosd: 0 ceph-volume lvm osd devices configured on this node
2022-01-24 16:50:59.742038 D | exec: Running command: stdbuf -oL ceph-volume --log-path /tmp/ceph-log raw list --format json
2022-01-24 16:51:00.581917 D | cephosd: {
"0f6d36fb-23ed-4888-9469-a5ab91370bd3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sdc1",
"osd_id": 1,
"osd_uuid": "0f6d36fb-23ed-4888-9469-a5ab91370bd3",
"type": "bluestore"
},
"51b3b924-f70f-4257-88cf-91d881faf5e3": {
"ceph_fsid": "576c5282-5b6c-4204-80d3-80049ef7326d",
"device": "/dev/sde1",
"osd_id": 4,
"osd_uuid": "51b3b924-f70f-4257-88cf-91d881faf5e3",
"type": "bluestore"
}
}

As can be seen sdd1 is just skipped/missing.

Basically the same issue is reported in https://github.com/rook/rook/issues/8023 but closed with assumption that ceph 16.2.6 provides a fix for this issue.

After testing i can confirm 16.2.7 still has this issue.

It seems our setup somehow triggers this issue:
- centos7 (kernel 3.10)
- 3 spinning disks per host
- rook 1.8.3

I can also report that by setting "bluefs_buffered_io = false" the issue doesn't occur; although we don't fully understand today the further implications of setting this option.

Operator logging is attached:
- operator.log std 16.2.7 experiencing the issue
- operator-no-buffering.log: same but with option set; doesn't show the issue.

I can easily reproduce this issue and can run additional debug statements; but since i'm fairly new to ceph i do need some guidenance what/how to debug.

operator-no-buffering.log View (67 KB) Paul Bormans, 01/26/2022 01:25 PM

operator.log View (90.1 KB) Paul Bormans, 01/26/2022 01:25 PM


Related issues

Copied to bluestore - Backport #58633: quincy: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error Resolved
Copied to bluestore - Backport #58952: reef: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error Resolved

History

#1 Updated by Neha Ojha about 2 years ago

  • Assignee set to Adam Kupczyk

#2 Updated by Satoru Takeuchi almost 2 years ago

FYI, some rook users hit this problem as well.

quoted from https://tracker.ceph.com/issues/51034#note-12

---
https://github.com/rook/rook/issues/9646#issuecomment-1021505103
```
2022-01-25 18:38:32.269143 I | op-osd: OSD orchestration status for node dev2-cmp1l is "failed"
2022-01-25 18:38:32.269193 E | op-osd: failed to provision OSD on node dev2-cmp1l. &{OSDs:[] Status:failed PvcBackedOSD:false Message:failed to configure devices: failed to initialize osd: failed to run ceph-volume raw command. Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new f4a0ae55-0b96-4304-a4cc-2f2f3d13a341
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
Running command: /usr/bin/chown -R ceph:ceph /dev/sde1
Running command: /usr/bin/ln -s /dev/sde1 /var/lib/ceph/osd/ceph-0/block
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/osd/ceph-0/activate.monmap
stderr: got monmap epoch 3
Running command: /usr/bin/ceph-authtool /var/lib/ceph/osd/ceph-0/keyring --create-keyring --name osd.0 --add-key AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==
stdout: creating /var/lib/ceph/osd/ceph-0/keyring
added entity osd.0 auth(key=AQClQ/Bhr58XLRAACuf8ROY8okq1izdbu4HZaQ==)
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/keyring
Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0/
Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph
stderr: 2022-01-25T18:38:30.769+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _read_fsid unparsable uuid
stderr: 2022-01-25T18:38:30.786+0000 7f89304df080 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 6f36f879-3c86-4ae1-8b46-abf13cc438c7, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-01-25T18:38:31.281+0000 7f89304df080 -1 bluestore(/var/lib/ceph/osd/ceph-0/) _open_db erroring opening db:
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-01-25T18:38:31.783+0000 7f89304df080 -1 �[0;31m * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-0/: (5) Input/output error�[0m
--> Was unable to complete a new OSD, will rollback changes
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.0 --yes-i-really-mean-it
stderr: purged osd.0
Traceback (most recent call last):
File "/usr/sbin/ceph-volume", line 11, in <module>
load_entry_point('ceph-volume==1.0.0', 'console_scripts', 'ceph-volume')()
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 40, in init
self.main(self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
return f(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
terminal.dispatch(self.mapper, subcommand_args)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/main.py", line 32, in main
terminal.dispatch(self.mapper, self.argv)
File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
instance.main()
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 169, in main
self.safe_prepare(self.args)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 91, in safe_prepare
self.prepare()
File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
return func(*a, **kw)
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 134, in prepare
tmpfs,
File "/usr/lib/python3.6/site-packages/ceph_volume/devices/raw/prepare.py", line 68, in prepare_bluestore
db=db
File "/usr/lib/python3.6/site-packages/ceph_volume/util/prepare.py", line 481, in osd_mkfs_bluestore
raise RuntimeError('Command failed with exit code %s: %s' % (returncode, ' '.join(command)))
RuntimeError: Command failed with exit code 250: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 0 --monmap /var/lib/ceph/osd/ceph-0/activate.monmap --keyfile - --osd-data /var/lib/ceph/osd/ceph-0/ --osd-uuid f4a0ae55-0b96-4304-a4cc-2f2f3d13a341 --setuser ceph --setgroup ceph: exit status 1}

```

Ths problem disappeared if `bluefs_buffered_io=false` was set.

https://github.com/rook/rook/issues/9646#issuecomment-1022021781
```
And i can confirm that:

- rook 1.8.3 / 16.2.7 still have this issue operator.log
- rook 1.8.3 / 16.2.7 + bluefs_buffered_io = false does not experience this issue. operator-no-buffering.log
```

And here is another report.

https://github.com/rook/rook/issues/10160
```
stderr: 2022-04-25T18:12:29.836+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _read_fsid unparsable uuid
stderr: 2022-04-25T18:12:30.024+0000 7f023a2a43c0 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid 68ba3e0d-e42c-46b0-a51b-e8e6ab918ad6, block dump:
stderr: 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr:
stderr: 00000ff0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
stderr: 00001000
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 rocksdb: verify_sharding unable to list column families: NotFound:
stderr: 2022-04-25T18:12:30.664+0000 7f023a2a43c0 -1 bluestore(/var/lib/ceph/osd/ceph-1/) _open_db erroring opening db:
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
stderr: 2022-04-25T18:12:31.224+0000 7f023a2a43c0 -1 * ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-1/: (5) Input/output error
--> Was unable to complete a new OSD, will rollback changes
```

The log is very similar to the first report. He used v17.20. But it's not clear that he tried `bluefs_buffered_io=false`. I suspect that `bluefs_buffered_id=true` (it's the default) still has some problems.
---

#3 Updated by Christian Rohmann almost 2 years ago

I observed similar issues at random when deploying Ceph 16.2.7 via Ansible on a VM.

ceph-osd -i 4 --mkfs --osd-objectstore=bluestore --osd-uuid de84f0aa-601a-473a-ad0b-7a2d403d0588 --monmap /tmp/monmap

return code is 250 and STDERR:

2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 
2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump: 
00000000  40 6a a8 52 08 5a 8a 2a  74 c2 c5 98 f9 c6 c8 12  |@j.R.Z.*t.......|
00000010  5c 45 ca 37 bb 49 87 6d  98 f7 46 07 d6 3c 48 40  |\\E.7.I.m..F..<H@|
00000020  af b9 65 99 4f be 0b 20  9e 29 d5 9b 10 b1 74 ca  |..e.O.. .)....t.|
00000030  82 77 b4 c5 ab de 04 9c  eb 90 7c ff ce 76 ac ed  |.w........|..v..|
00000040  3e 43 93 72 da 75 de 41  ed a9 49 5a 13 7a da 72  |>C.r.u.A..IZ.z.r|
00000050  7e 8d 1c d3 5b 31 98 7d  48 ed da 00 63 e7 f1 d5  |~...[1.}H...c...|
00000060  d1 5c f1 7f 2a c5 af 7d  56 2f fb 16 d7 26 56 ce  |.\\..*..}V/...&V.|
00000070  ed ec 11 04 c6 b6 52 df  18 64 49 75 65 e8 62 47  |......R..dIue.bG|
00000080  29 1f de 22 e3 b2 45 0f  ec fb 7c d9 f9 9a 5a d8  |)..\"..E...|...Z.|
00000090  82 40 71 6f c3 b6 73 5f  e1 ed 3f 3d fd 56 8c e8  |.@qo..s_..?=.V..|
000000a0  a8 6a bf 6c f2 bc fa ca  44 8e e6 b5 6f f0 d0 bc  |.j.l....D...o...|
000000b0  8f a3 ae 35 44 d2 23 74  6b 08 d6 75 9f 74 b0 fb  |...5D.#tk..u.t..|
000000c0  8a f0 07 d1 7c b2 3e fd  c3 e1 a5 7c cf d3 62 a6  |....|.>....|..b.|
000000d0  5b 04 c7 6d 7d 70 d9 72  cf 1b ee ef c6 59 41 4a  |[..m}p.r.....YAJ|
000000e0  de 0d de 51 d1 02 cd ae  be 3f 7d fd 45 92 f8 63  |...Q.....?}.E..c|
000000f0  6d 41 19 0f c7 d3 0e cf  64 2c 59 fc 86 0a c3 ec  |mA......d,Y.....|
00000100  29 5d 6f 13 6b 62 fc 80  f4 b4 ce 9b 53 81 36 ea  |)]o.kb......S.6.|
00000110  5e 07 e0 61 20 d2 15 43  2a 7a 84 f1 a3 0f 9b 8d  |^..a ..C*z......|
00000120  ba 42 01 94 df f3 d4 87  fc 45 5c 8a dd d2 28 26  |.B.......E\\...(&|
00000130  a7 35 50 ee 58 1e f3 22  36 1d da 76 2c 07 c9 28  |.5P.X..\"6..v,..(|
00000140  e1 cb 66 20 cb e4 dd bb  91 eb b0 4f 66 1f 89 69  |..f .......Of..i|
00000150  4a ec 47 e3 d2 f6 2f 59  18 39 29 42 14 ea f6 19  |J.G.../Y.9)B....|
00000160  45 a0 e2 44 ed 97 4d 1f  d2 f2 70 b0 13 70 e3 5b  |E..D..M...p..p.[|
00000170  95 bb 43 6d 9c 35 25 17  14 2d a5 30 46 e3 38 e4  |..Cm.5%..-.0F.8.|
00000180  55 63 06 15 1c 1e d3 2e  fd 7f 3e 4b 18 07 ed 53  |Uc........>K...S|
00000190  0b 7e d8 dd 39 0a d2 53  23 10 ce 7a 55 53 5e cf  |.~..9..S#..zUS^.|
000001a0  77 6d a8 f8 ef e5 8a 01  13 b0 ac ce 31 07 c7 19  |wm..........1...|
000001b0  69 0e 00 c6 26 88 7a e0  be f5 2c ac 2c 6b ce 8a  |i...&.z...,.,k..|
000001c0  b3 82 fe fc 16 ea 46 9e  eb 7e 54 f4 b1 4d 00 dc  |......F..~T..M..|
000001d0  0e e9 5b 3a 8b 5a 49 f0  46 f6 31 a8 9b 39 ba 96  |..[:.ZI.F.1..9..|
000001e0  b4 ce 76 69 91 5b 74 36  e9 89 cc 67 7c 2e c7 6b  |..vi.[t6...g|..k|
000001f0  2e b2 f1 b0 fc a6 c6 c1  c9 35 b5 be 23 c1 b3 94  |.........5..#...|
00000200  11 47 12 1a ea 5e 47 84  a5 92 6c 83 e3 e2 ff 8a  |.G...^G...l.....|
00000210  ed ba 48 f8 41 17 d5 2e  d2 64 3a 64 64 21 ef 48  |..H.A....d:dd!.H|
00000220  f8 78 b3 2a ca 0b 1e 7d  80 0b fb 1c b5 73 1b ee  |.x.*...}.....s..|
00000230  0f a0 4d 6b d9 22 72 06  1c 5d 27 af f6 c3 52 a8  |..Mk.\"r..]'...R.|
00000240  47 ac bf 18 50 4a 64 29  3c 71 1b 9d 21 37 a9 f1  |G...PJd)<q..!7..|
00000250  ea 52 7a 25 48 ef 8d 45  97 65 49 8b 1c a0 c5 1d  |.Rz%H..E.eI.....|
00000260  94 da dc 90 b3 fe 56 2f  5e de 0a 02 58 6b 6f e6  |......V/^...Xko.|
00000270  e3 0e 93 eb 98 9a 4d 35  4f 8b 50 53 9f f6 35 d5  |......M5O.PS..5.|
00000280  2d c6 c6 45 d4 38 1d 68  7d b7 46 e0 8a 60 1b ec  |-..E.8.h}.F..`..|
00000290  28 64 22 67 8b d4 0e 7c  05 28 24 59 fb b6 e5 44  |(d\"g...|.($Y...D|
000002a0  9e f4 d2 a5 08 cb 3a 52  a5 51 c3 56 cd fb 29 0b  |......:R.Q.V..).|
000002b0  8d bb 28 ab 06 c5 e8 9c  b8 99 ee 42 a5 4f 1c 0c  |..(........B.O..|
000002c0  5f 29 1e 2f cf b2 f6 dc  05 21 b1 68 d2 79 ae b2  |_)./.....!.h.y..|
000002d0  d9 38 5f f6 5d 08 60 44  c1 6b d8 b1 a9 41 b7 48  |.8_.].`D.k...A.H|
000002e0  44 7a 4d f0 67 ef d9 ca  2f d5 69 9b c3 bf 78 ff  |DzM.g.../.i...x.|
000002f0  2e f4 a4 94 14 0d fa 7b  60 55 fb 75 fb 49 52 5f  |.......{`U.u.IR_|
00000300  fc 22 ad 16 ac c2 80 e1  ab cf bf 7f 84 6b df 0e  |.\"...........k..|
00000310  4c 38 56 02 1a 37 bf 07  b1 e9 ac dc a8 56 f2 bf  |L8V..7.......V..|
00000320  59 f8 8c 9b 58 88 54 2f  bf d5 5e 1a c4 93 ea f3  |Y...X.T/..^.....|
00000330  ab 56 f0 a5 7c 66 7c 83  dd 00 30 6f 84 37 72 6b  |.V..|f|...0o.7rk|
00000340  6a 13 d8 a8 e2 49 c7 2c  e8 b8 c1 9f 29 3e e0 12  |j....I.,....)>..|
00000350  63 5f 22 df bd f7 9b 25  7e e9 c2 ff 1c 86 f0 e7  |c_\"....%~.......|
00000360  a9 59 06 18 9d fa 20 eb  2f 72 31 c4 53 c2 dd 09  |.Y.... ./r1.S...|
00000370  65 3c 9d 5d 7b 51 57 28  de 8b a5 e1 4a 1f 64 2e  |e<.]{QW(....J.d.|
00000380  37 49 7f bd 15 57 09 dc  fb 74 40 68 00 62 f8 5d  |7I...W...t@h.b.]|
00000390  99 2c 6e c7 e6 64 bc 3f  26 46 90 f8 69 95 3b 48  |.,n..d.?&F..i.;H|
000003a0  a7 45 80 09 2d e4 5f c3  89 8e a1 e9 50 c6 87 22  |.E..-._.....P..\"|
000003b0  04 b5 c6 87 a8 1d 96 ff  65 bd 93 b0 69 62 8d b7  |........e...ib..|
000003c0  5b 9e e1 34 25 b6 7f 0e  e6 2f 79 b2 ae 85 fc c8  |[..4%..../y.....|
000003d0  cd 4d dd b3 86 79 89 c8  86 d3 a9 a5 5d aa 9c 0c  |.M...y......]...|
000003e0  a0 c3 e1 01 cb c8 30 81  46 35 f8 6f 81 4c d3 b5  |......0.F5.o.L..|
000003f0  be a8 e3 29 96 8c bb 1c  7c 1f fb e6 51 98 b6 90  |...)....|...Q...|
00000400  c0 a2 fc 9e bb b1 52 f1  b1 42 09 58 12 22 ec 16  |......R..B.X.\"..|
00000410  5f 81 49 ee 99 0a 6a 88  e9 76 60 4d ee 58 83 d3  |_.I...j..v`M.X..|
00000420  98 5b 55 ec 59 e8 ab 1a  f6 d8 4a 9c 4b e5 56 66  |.[U.Y.....J.K.Vf|
00000430  b1 e1 fd 91 59 20 ce 42  db 6b 3d 7c 6a 7e 5e 05  |....Y .B.k=|j~^.|
00000440  49 a9 35 e3 28 25 08 bf  37 10 cb 60 27 ab c9 17  |I.5.(%..7..`'...|
00000450  2b 0e e0 54 2a 5a 53 df  80 e6 ca bf 7a de 55 54  |+..T*ZS.....z.UT|
00000460  71 f8 ca 9e fc 46 b3 c3  7f e1 ae a5 cd 6b 51 17  |q....F.......kQ.|
00000470  b3 48 fa 00 b9 7a 82 f1  16 02 ab 78 1a ad 99 5f  |.H...z.....x..._|
00000480  8c c9 1d 53 dc 0e ff 12  b6 ba 6f 0e fe fb f2 31  |...S......o....1|
00000490  77 e9 f5 17 3d db ab 75  3f ce 76 48 57 37 e3 b4  |w...=..u?.vHW7..|
000004a0  b0 cc ae 79 f2 50 d4 55  fc 8b 1c aa 14 38 b8 a1  |...y.P.U.....8..|
000004b0  78 7b 20 72 67 03 25 c7  e7 15 ff a8 86 f9 ec f7  |x{ rg.%.........|
000004c0  3b 2c 56 20 87 75 53 a4  51 10 d7 1e b8 ea 6d d9  |;,V .uS.Q.....m.|
000004d0  78 18 98 b0 65 e7 a2 df  fa 0b f2 79 cb c4 51 98  |x...e......y..Q.|
000004e0  be d0 eb c4 72 b4 a8 48  0f aa 6e 1b dc ff cb 1d  |....r..H..n.....|
000004f0  aa 34 ee 55 57 19 87 3c  69 67 f3 63 b6 61 48 6d  |.4.UW..<ig.c.aHm|
00000500  37 7d 18 32 6b 74 f1 59  c5 11 7d 13 e5 4c 21 81  |7}.2kt.Y..}..L!.|
00000510  5d 79 8a cb 9a 41 7a f9  9d 0e e7 ca 38 66 f4 1a  |]y...Az.....8f..|
00000520  b7 a1 cd 7e 4d b1 19 f3  5d 53 a9 8d 9e 83 ae 0b  |...~M...]S......|
00000530  c3 87 cd 4a 6e fe af a0  61 95 30 ee 99 ec 34 9a  |...Jn...a.0...4.|
00000540  ec ac 19 ff fe 97 69 54  22 cc 04 17 62 a8 ee 7c  |......iT\"...b..||
00000550  8f a4 a0 50 90 b9 fa e7  3f 5d 7a 2b f8 d1 83 8b  |...P....?]z+....|
00000560  13 51 6c c4 a1 1c b1 5c  de 83 a6 aa 43 d1 1d c1  |.Ql....\\....C...|
00000570  e9 e6 60 c0 f2 2c 35 42  b7 71 a4 00 3a b2 db 36  |..`..,5B.q..:..6|
00000580  73 40 12 90 ee 73 b6 d5  84 7b 4a c2 73 1e 2e af  |s@...s...{J.s...|
00000590  33 53 82 67 89 10 cb 85  5a 96 d3 af 47 d9 47 07  |3S.g....Z...G.G.|
000005a0  18 e0 ba 7a aa 0c 69 08  d5 d2 60 a1 2a 38 8e 3e  |...z..i...`.*8.>|
000005b0  dd de a1 d1 24 32 63 36  55 31 d4 7c 51 1e 83 ea  |....$2c6U1.|Q...|
000005c0  4d f4 f0 ef 75 0b 89 fa  17 26 fa 81 67 53 36 57  |M...u....&..gS6W|
000005d0  87 6f ee 61 0d 65 d3 3c  ba b1 04 0c ae 27 c3 7c  |.o.a.e.<.....'.||
000005e0  19 ce 18 e3 cd 60 37 b4  b1 f7 f4 8b c2 fa 42 e5  |.....`7.......B.|
000005f0  a6 7f f2 2f 39 c9 94 62  93 68 64 5d 8b 1a 40 9e  |.../9..b.hd]..@.|
00000600  ab 75 b3 e8 88 4f 5d ea  a9 8e 4b fb c5 51 ca 9c  |.u...O]...K..Q..|
00000610  49 d2 fc 68 18 4a 22 ce  7d 1d 00 e7 52 61 5f 85  |I..h.J\".}...Ra_.|
00000620  ec 3c 13 74 a5 78 75 3e  14 f2 27 ed b2 5d 92 16  |.<.t.xu>..'..]..|
00000630  05 39 dc ee 8c 13 4c af  63 86 1c 3f ac 7e 81 67  |.9....L.c..?.~.g|
00000640  b0 d7 a2 f3 9f 31 73 ae  97 08 11 2b fa 99 5e c9  |.....1s....+..^.|
00000650  d1 39 4a 9e a4 bb d1 55  83 c2 f4 2f bd 7e 85 c9  |.9J....U.../.~..|
00000660  d9 34 c6 61 c6 1d 12 9c  d4 e9 ec 05 4f 04 55 3a  |.4.a........O.U:|
00000670  78 2f fa 33 02 87 97 e1  65 ef af 8e f0 b2 f9 3c  |x/.3....e......<|
00000680  75 ca 39 cc 12 2e 58 6e  bd 50 9f 46 83 4f 02 3e  |u.9...Xn.P.F.O.>|
00000690  50 8d 0a 8f 11 a2 06 70  70 0f 94 91 2d c7 da c4  |P......pp...-...|
000006a0  9a 22 eb 9f 14 04 c4 d0  35 c7 c9 60 9d b0 10 c3  |.\"......5..`....|
000006b0  ab fd c7 31 a4 b7 23 78  10 17 3b de 05 c8 f1 1b  |...1..#x..;.....|
000006c0  90 32 54 ff eb bc 1f e4  11 78 f9 9c 2f ab 49 75  |.2T......x../.Iu|
000006d0  9f 42 89 54 77 9a 0b 0c  a7 e8 b2 18 85 cd 2a d0  |.B.Tw.........*.|
000006e0  39 a0 09 96 09 2c a5 34  ef e9 85 0b d8 ed 22 e0  |9....,.4......\".|
000006f0  7c 80 07 8b 12 ed 60 b1  e8 ec 4e 26 97 f0 50 e1  ||.....`...N&..P.|
00000700  2f 2b 08 b8 03 87 7a da  44 7d 93 16 b5 70 c7 a2  |/+....z.D}...p..|
00000710  2f 5b 34 10 5d dc f6 f4  46 8c 8e 5c 4c aa db 44  |/[4.]...F..\\L..D|
00000720  ce fa ea 39 cb 23 e8 0c  f4 13 99 e4 20 c7 9c 6a  |...9.#...... ..j|
00000730  9e 37 b0 e0 42 af b3 59  98 6b fd 66 61 85 6e dc  |.7..B..Y.k.fa.n.|
00000740  7c 53 b4 0b 16 9f 92 07  8b 4e 8d fc 05 0b 4e 1a  ||S.......N....N.|
00000750  c6 b8 e0 4f 1b 6d 4f 3a  9d 2c 7c 3e 0a 42 e4 ba  |...O.mO:.,|>.B..|
00000760  3c 6f d3 f1 88 d2 e8 69  3a d0 dd c2 b2 15 17 d0  |<o.....i:.......|
00000770  89 88 8a e7 34 fa 57 07  b3 37 89 5f 39 4a 2d 02  |....4.W..7._9J-.|
00000780  58 e6 7b 91 1b 15 5a 44  6e 38 e5 4b ef 8c 42 00  |X.{...ZDn8.K..B.|
00000790  b3 31 26 d7 6a 79 b8 e6  ca 72 88 c6 e7 f8 55 4e  |.1&.jy...r....UN|
000007a0  62 15 84 08 2b dc 6b 52  7f 9b 6d 7e 47 6f 35 ca  |b...+.kR..m~Go5.|
000007b0  a5 4d 10 07 54 d2 86 ea  5e e0 33 36 72 93 d9 6f  |.M..T...^.36r..o|
000007c0  4f 09 19 63 8c 6b 04 8f  6d a9 c6 29 09 e2 63 61  |O..c.k..m..)..ca|
000007d0  6a 74 83 e5 9d 6c c1 b3  31 20 2b 97 34 88 d5 66  |jt...l..1 +.4..f|
000007e0  7c a4 f8 fa 04 b1 4e 2e  1c 81 23 79 9a 09 92 11  ||.....N...#y....|
000007f0  94 ba 59 25 7f ea a6 02  0d c1 0b f3 f3 b6 52 88  |..Y%..........R.|
00000800  fb 00 ac e4 bc 43 6a a2  ff 9b 5f c9 72 ad fc 3d  |.....Cj..._.r..=|
00000810  02 41 2a 4e 36 d5 7f 19  b3 34 f2 7a 5a d7 42 62  |.A*N6....4.zZ.Bb|
00000820  4b 6e 52 2a 63 b7 fd 1f  b9 97 18 88 53 7d d4 f5  |KnR*c.......S}..|
00000830  21 04 f4 ae bd 9d ea f7  44 7b f5 49 1a bf 0c 8c  |!.......D{.I....|
00000840  15 0a fd b1 35 9c 57 3b  8f 2a 71 e6 40 cf 93 c8  |....5.W;.*q.@...|
00000850  b2 80 1d 8f 19 46 b7 16  ad bf f9 59 16 f5 1b 8d  |.....F.....Y....|
00000860  aa ea 1e b2 d2 34 92 c2  aa 69 d0 81 2e 50 23 db  |.....4...i...P#.|
00000870  29 f4 a7 cb b2 00 1d e5  a7 12 be d9 36 35 fe 6d  |)...........65.m|
00000880  b2 97 b5 0c 84 3e 7b be  46 b5 50 9e 0c 3c a0 3a  |.....>{.F.P..<.:|
00000890  d3 d3 d8 e8 16 c2 3e c1  ee 79 43 06 95 ad eb a3  |......>..yC.....|
000008a0  cd e3 6f 5b f2 9f 3f 12  69 2f eb 25 2f 14 2f 3d  |..o[..?.i/.%/./=|
000008b0  aa 0f f8 fa e1 0b 6c 4c  55 a5 9d 4a 87 3b 8c 2c  |......lLU..J.;.,|
000008c0  65 bc 0c b8 10 6a ed 82  bc 6e 98 6e 15 82 c8 ba  |e....j...n.n....|
000008d0  83 9a 43 19 88 c4 56 3d  f9 56 10 bd c0 58 5e 1b  |..C...V=.V...X^.|
000008e0  14 9a d2 56 56 c7 fc 66  59 ff 81 77 1d 29 fe 0c  |...VV..fY..w.)..|
000008f0  82 ca 17 75 d7 65 8f 04  3a 5b c9 af cf f3 5c 18  |...u.e..:[....\\.|
00000900  f8 57 1d 33 3b 73 7a a3  f4 a8 f0 a9 e3 aa bc e5  |.W.3;sz.........|
00000910  db 15 cb 51 d8 54 b2 f0  e2 e6 7d 39 59 8a 3b ec  |...Q.T....}9Y.;.|
00000920  ee 2e bf db ed d5 33 09  6e 26 d2 30 7f b9 58 be  |......3.n&.0..X.|
00000930  87 06 b1 ac 8a 07 2d b8  ee d1 92 55 21 5d d2 c1  |......-....U!]..|
00000940  2b 5b 4b e7 36 0b f7 1f  8d 96 0e 76 25 01 80 0d  |+[K.6......v%...|
00000950  0e 93 98 43 c4 b9 95 e4  a6 28 e3 fc b8 b9 5b e9  |...C.....(....[.|
00000960  af 18 a6 bc 4d dc 2e 83  86 14 5c 95 c2 1b e3 bf  |....M.....\\.....|
00000970  17 6e 72 8b 48 8b ed 1f  7a 99 b5 4a ad 8e 60 62  |.nr.H...z..J..`b|
00000980  40 2e 66 c5 8b 54 2f 3f  df ac 0d 21 01 65 ad 53  |@.f..T/?...!.e.S|
00000990  d6 30 ce ef 8e fc f4 97  94 4e f2 52 f7 39 ef ee  |.0.......N.R.9..|
000009a0  b7 e7 31 01 54 6c 9f d0  6a cc 22 42 8c c3 f8 a8  |..1.Tl..j.\"B....|
000009b0  9b b9 fb 1e e5 ed a0 54  d4 6a e0 53 4e 3d f3 fc  |.......T.j.SN=..|
000009c0  d2 f7 04 8c 62 01 bf 68  d4 05 5c 3a 01 7c 17 d5  |....b..h..\\:.|..|
000009d0  ea d2 4c ed 87 32 81 68  f8 4d 36 70 aa 31 86 75  |..L..2.h.M6p.1.u|
000009e0  76 f0 c6 21 f1 0e d8 4e  74 de 9e e7 74 bb f5 dd  |v..!...Nt...t...|
000009f0  f5 9c 68 9f 44 c4 c6 d2  bd f9 fe 6d 63 66 5d 6e  |..h.D......mcf]n|
00000a00  64 45 28 cb 5d 4c 0d 80  c7 8c 5b 83 53 77 4b 7c  |dE(.]L....[.SwK||
00000a10  f1 32 c1 f9 1f c5 11 e5  39 2a e6 a9 0c de 68 91  |.2......9*....h.|
00000a20  66 6b 38 04 ed fc d7 60  6e a7 29 7e 5b 13 47 fe  |fk8....`n.)~[.G.|
00000a30  a7 e0 53 23 f5 aa 2a bd  91 d7 fd 02 ae f3 29 b1  |..S#..*.......).|
00000a40  df 0b cd e0 47 c6 cf 20  03 3e 93 a0 25 bb 4c 13  |....G.. .>..%.L.|
00000a50  5c fc dc 5c 35 55 a3 f5  86 0b da 9a 1e 63 e4 de  |\\..\\5U.......c..|
00000a60  bc 83 1d f3 af dc 49 e8  21 5f 30 6d 14 f2 cc 00  |......I.!_0m....|
00000a70  2d 7a 07 47 99 21 a9 98  bf 57 bb 99 f2 d4 32 38  |-z.G.!...W....28|
00000a80  a7 67 8a 5f 17 06 e1 1c  9d 5d 69 f0 76 64 3d 45  |.g._.....]i.vd=E|
00000a90  ad c0 e6 dc 7d e9 59 8d  9b 33 4c 04 a7 be 2d b5  |....}.Y..3L...-.|
00000aa0  b0 f5 26 14 86 b9 de 45  f4 d8 8a 59 27 db 38 56  |..&....E...Y'.8V|
00000ab0  7c 4b 1b 52 37 a7 9f 71  00 75 66 3a 2f 0a e3 f9  ||K.R7..q.uf:/...|
00000ac0  c7 21 dd 70 50 52 14 9f  ff 7f 2f d5 21 2e c6 ba  |.!.pPR..../.!...|
00000ad0  5c 3d 84 39 be ef 25 db  f2 dd ec 53 63 1a 69 fd  |\\=.9..%....Sc.i.|
00000ae0  be 71 59 a4 de df 11 3e  ee ae c9 5d e8 75 1c cc  |.qY....>...].u..|
00000af0  1c 00 db e6 42 1b ba 99  03 29 0b 83 02 12 bc 7e  |....B....).....~|
00000b00  dd a8 7e 2a 68 41 71 d8  28 b1 9c cd a3 fa 6c 3a  |..~*hAq.(.....l:|
00000b10  88 78 69 2d 94 6e dc 85  a6 d4 59 02 7f c2 2f 01  |.xi-.n....Y.../.|
00000b20  0f 06 03 ac ac 30 bb b6  84 43 3f 0c 4d 9a 63 2a  |.....0...C?.M.c*|
00000b30  22 98 fb b4 2c a0 b8 49  87 ba 13 ed 0b 30 c0 39  |\"...,..I.....0.9|
00000b40  f5 5b 31 7d dd 54 89 a8  2d cf fd 79 b9 ef 3a 50  |.[1}.T..-..y..:P|
00000b50  85 54 fd 5a de 21 8c 73  f5 e8 38 30 db dc 23 96  |.T.Z.!.s..80..#.|
00000b60  c7 41 f1 94 d8 ed b6 85  ef fd 81 3f b0 a8 a1 6a  |.A.........?...j|
00000b70  41 7c f5 8d 3c 17 64 cf  80 37 66 78 02 e4 05 e5  |A|..<.d..7fx....|
00000b80  80 00 87 c4 dc f2 37 fd  ed f3 b7 a2 91 a8 90 04  |......7.........|
00000b90  85 2c ac 52 3f d0 0a 39  20 b1 50 34 03 57 80 ed  |.,.R?..9 .P4.W..|
00000ba0  8b 60 87 1b f3 59 89 f8  a0 6d f8 fa 8d 70 b9 f2  |.`...Y...m...p..|
00000bb0  57 3b cc db f4 6c 75 c3  b0 f0 54 86 e4 52 50 cc  |W;...lu...T..RP.|
00000bc0  97 8e fc 69 6b 26 ba e0  2e d5 10 8c 7b 84 8f 4e  |...ik&......{..N|
00000bd0  1b 40 c9 d5 f4 95 e5 c2  6c 91 75 eb 0e 85 18 3d  |.@......l.u....=|
00000be0  7f 57 24 de bd 05 fb bd  83 ac 08 30 40 f0 00 06  |.W$........0@...|
00000bf0  8a 98 c7 dd 3b 08 c8 84  72 d4 88 9f f9 9d 96 9e  |....;...r.......|
00000c00  d3 15 05 95 db 03 fe c5  e3 7f 31 68 ab 40 29 b9  |..........1h.@).|
00000c10  ff 66 87 cc 08 cc 49 9f  5f e4 24 de 12 fd 60 d7  |.f....I._.$...`.|
00000c20  0f 3e a0 7d ca 34 91 29  1c d9 88 6d 9d fb 56 16  |.>.}.4.)...m..V.|
00000c30  bd 55 0c c7 1b f2 4a 2d  eb ab 69 8f f3 fa 83 dc  |.U....J-..i.....|
00000c40  ae 6a 58 9a 4c c7 a6 50  88 fd 15 d5 df 1f a9 9b  |.jX.L..P........|
00000c50  91 c1 2e 79 c1 e0 1f 42  2e 48 42 db 9c be 5f 1c  |...y...B.HB..._.|
00000c60  31 88 dc ad db 6d f6 27  45 bc 03 57 c6 f1 26 0b  |1....m.'E..W..&.|
00000c70  f2 1f 1d 42 44 c3 53 cd  77 e5 e2 90 64 c1 1f 77  |...BD.S.w...d..w|
00000c80  ff ef 02 00 54 21 0c 6a  a2 d4 4b 9c d6 3c e8 99  |....T!.j..K..<..|
00000c90  e9 64 8c 10 7e ba b7 fd  e6 68 62 b9 06 3c d8 c1  |.d..~....hb..<..|
00000ca0  5e ca 51 dd 60 df 99 28  0a 16 30 58 8d 5f 58 1d  |^.Q.`..(..0X._X.|
00000cb0  47 3a 49 de 5f 11 3f d0  f3 05 d4 9f 97 a4 da 31  |G:I._.?........1|
00000cc0  7e 25 73 f7 0b ef 8f e8  c4 3d 57 ad 77 6c 78 56  |~%s......=W.wlxV|
00000cd0  9c 8b a5 10 1a 32 f1 75  ca 9e 06 59 da 58 d6 3f  |.....2.u...Y.X.?|
00000ce0  43 97 4d c7 b5 da 41 f8  8e b4 1f 4c 18 d9 07 c2  |C.M...A....L....|
00000cf0  68 85 a0 94 a9 38 a6 b9  cf 16 5f 36 fd 3f 2b da  |h....8...._6.?+.|
00000d00  87 d6 29 95 91 71 a3 47  5e 83 8d fe ba e6 a3 56  |..)..q.G^......V|
00000d10  c4 73 c5 d2 d6 ea 0c 6f  c7 c1 b8 84 68 a7 9e de  |.s.....o....h...|
00000d20  92 f7 dd a0 91 ca 78 79  65 88 90 b7 7e eb 9a 60  |......xye...~..`|
00000d30  72 2b 29 e3 56 04 6b 4b  e7 38 27 bb ad 93 99 13  |r+).V.kK.8'.....|
00000d40  fe df 5a e2 5f 66 41 f3  99 11 ea 68 18 83 4c 3d  |..Z._fA....h..L=|
00000d50  a8 01 8a 6b a8 aa f6 3f  7f 76 06 6b ad 04 25 fe  |...k...?.v.k..%.|
00000d60  b4 ab 49 04 21 a3 e3 e3  03 c2 ba 32 04 16 48 e5  |..I.!......2..H.|
00000d70  d7 8c 2f 52 e6 3f 8d b8  94 60 e8 04 e9 34 9e fd  |../R.?...`...4..|
00000d80  fa 41 12 13 da 54 7d 2f  ff 6a 3e de 26 e1 9b 34  |.A...T}/.j>.&..4|
00000d90  8e 26 2b 24 d0 eb b3 22  35 1a ad b2 70 04 9b d4  |.&+$...\"5...p...|
00000da0  1e 90 46 90 de 74 9e fe  a7 6e a8 59 a3 b9 b8 1a  |..F..t...n.Y....|
00000db0  a3 9b 65 e0 d2 64 53 4b  5e 42 20 41 9c 55 60 48  |..e..dSK^B A.U`H|
00000dc0  b2 30 f7 f6 18 2b 71 64  cd 86 11 0e 0e a3 d5 f4  |.0...+qd........|
00000dd0  b0 ca b9 b1 bd 76 cd 40  d2 ef 12 46 3c 28 ec f6  |.....v.@...F<(..|
00000de0  33 96 dc 26 33 e0 08 46  f2 68 1e 43 a7 38 16 80  |3..&3..F.h.C.8..|
00000df0  9c 6d 8b 28 c0 03 a7 99  22 61 0a 47 34 5e 5e 58  |.m.(....\"a.G4^^X|
00000e00  67 67 db de df cf ff c0  05 dc 09 84 e2 28 60 2f  |gg...........(`/|
00000e10  f7 08 3d a8 8a 69 4a 0b  1c c5 44 f1 29 11 3e 71  |..=..iJ...D.).>q|
00000e20  ae f3 d1 c2 37 d1 18 79  ab 64 8f 82 bf 38 7b 1b  |....7..y.d...8{.|
00000e30  52 ef 00 9a 4f 44 dc 4c  7c f9 c9 47 ba a8 6d fa  |R...OD.L|..G..m.|
00000e40  1d aa f5 ff c4 c2 0d 53  27 e6 17 d0 be 6d 97 30  |.......S'....m.0|
00000e50  81 1c 9c c6 75 35 14 ff  73 d8 78 27 92 51 8e fc  |....u5..s.x'.Q..|
00000e60  94 ec 64 45 19 5c 0b be  b5 99 e9 b9 76 5f 70 de  |..dE.\\......v_p.|
00000e70  00 b3 16 08 b0 76 06 a9  77 8a 79 c5 fc 0b f1 6c  |.....v..w.y....l|
00000e80  03 c1 0f a9 e9 70 19 ba  fa b2 4f e2 13 33 6a 06  |.....p....O..3j.|
00000e90  1c af 25 2c eb 94 3b 18  31 0a d0 b9 a6 13 53 7c  |..%,..;.1.....S||
00000ea0  3a 00 57 23 05 41 34 2c  7c 4f e2 f3 72 0b e2 a7  |:.W#.A4,|O..r...|
00000eb0  e0 78 02 ab 7f 2c 0e 32  c3 d4 e4 b8 2b 2f 31 fb  |.x...,.2....+/1.|
00000ec0  f8 92 db 7a 41 4d 98 71  b9 a3 0d eb fd 71 fc 2e  |...zAM.q.....q..|
00000ed0  e8 5c a1 b5 58 f8 5a 11  20 04 da 52 39 b6 06 70  |.\\..X.Z. ..R9..p|
00000ee0  43 0d 7a 48 14 18 21 03  2c 61 37 ef 23 41 b8 41  |C.zH..!.,a7.#A.A|
00000ef0  3b e4 a0 f6 71 2a d2 06  97 b0 5a d2 81 36 b0 6f  |;...q*....Z..6.o|
00000f00  00 ed db 5f 96 bd 37 bd  7c bc 33 47 37 67 2a ad  |..._..7.|.3G7g*.|
00000f10  b6 81 e1 8d 5a e2 68 c2  0b 32 d3 f1 21 02 77 3b  |....Z.h..2..!.w;|
00000f20  89 ee da 0d 87 8f 26 0f  73 17 19 3b b7 04 e5 38  |......&.s..;...8|
00000f30  54 c4 f3 1b 7b bc 93 9a  ce 38 87 f2 3f a2 8b 38  |T...{....8..?..8|
00000f40  a9 b1 7c 51 a2 34 f0 72  43 5c 59 35 b6 fd 50 2e  |..|Q.4.rC\\Y5..P.|
00000f50  db dc 29 d4 65 6a ef 23  83 7d 9d 49 b7 56 e4 c0  |..).ej.#.}.I.V..|
00000f60  4e 4d d8 06 ca e5 85 ee  41 cb 66 84 1a af 79 7f  |NM......A.f...y.|
00000f70  f0 9e 7c 74 8c 4a 80 e2  e6 0a 58 fb c9 61 3f a1  |..|t.J....X..a?.|
00000f80  b7 3b 7a ef 1f 91 95 d5  51 ab 01 ef 0e 86 c4 bb  |.;z.....Q.......|
00000f90  78 0e cb f3 b5 f9 30 d8  ae f0 81 11 e9 24 e6 e2  |x.....0......$..|
00000fa0  67 8a 30 9c 04 a6 a1 cb  d2 73 01 81 fb 01 28 f0  |g.0......s....(.|
00000fb0  62 2d d5 91 24 9e ec f8  74 3e 26 63 46 a0 b1 c2  |b-..$...t>&cF...|
00000fc0  63 31 0d 91 e8 fb 0d 6d  43 e0 8b ce 6b 8b 77 a2  |c1.....mC...k.w.|
00000fd0  63 56 28 c8 60 8e c1 53  ae 70 3d 06 54 b5 f4 9a  |cV(.`..S.p=.T...|
00000fe0  8a be 4c 43 9c f6 64 40  6e 97 2c 97 cd 74 c8 83  |..LC..d@n.,..t..|
00000ff0  b3 0b 41 74 56 fc 31 93  a5 dd c2 15 8c dd 9d 43  |..AtV.1........C|
00001000

2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound: 
2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db: 
2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m
2022-05-09T11:47:07.220+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid
2022-05-09T11:47:07.260+0000 7f5402bfff00 -1 bluefs _replay 0x0: stop: uuid 8a2a74c2-c598-f9c6-c812-5c45ca37bb49 != super.uuid 586cfac7-f434-4c0b-9f55-ac6c10650cc5, block dump: ", "00000000  40 6a a8 52 08 5a 8a 2a  74 c2 c5 98 f9 c6 c8 12  |@j.R.Z.*t.......|", "00000010  5c 45 ca 37 bb 49 87 6d  98 f7 46 07 d6 3c 48 40  |\\E.7.I.m..F..<H@|", "00000020  af b9 65 99 4f be 0b 20  9e 29 d5 9b 10 b1 74 ca  |..e.O.. .)....t.|", "00000030  82 77 b4 c5 ab de 04 9c  eb 90 7c ff ce 76 ac ed  |.w........|..v..|", "00000040  3e 43 93 72 da 75 de 41  ed a9 49 5a 13 7a da 72  |>C.r.u.A..IZ.z.r|", "00000050  7e 8d 1c d3 5b 31 98 7d  48 ed da 00 63 e7 f1 d5  |~...[1.}H...c...|", "00000060  d1 5c f1 7f 2a c5 af 7d  56 2f fb 16 d7 26 56 ce  |.\\..*..}V/...&V.|", "00000070  ed ec 11 04 c6 b6 52 df  18 64 49 75 65 e8 62 47  |......R..dIue.bG|", "00000080  29 1f de 22 e3 b2 45 0f  ec fb 7c d9 f9 9a 5a d8  |)..\"..E...|...Z.|", "00000090  82 40 71 6f c3 b6 73 5f  e1 ed 3f 3d fd 56 8c e8  |.@qo..s_..?=.V..|", "000000a0  a8 6a bf 6c f2 bc fa ca  44 8e e6 b5 6f f0 d0 bc  |.j.l....D...o...|", "000000b0  8f a3 ae 35 44 d2 23 74  6b 08 d6 75 9f 74 b0 fb  |...5D.#tk..u.t..|", "000000c0  8a f0 07 d1 7c b2 3e fd  c3 e1 a5 7c cf d3 62 a6  |....|.>....|..b.|", "000000d0  5b 04 c7 6d 7d 70 d9 72  cf 1b ee ef c6 59 41 4a  |[..m}p.r.....YAJ|", "000000e0  de 0d de 51 d1 02 cd ae  be 3f 7d fd 45 92 f8 63  |...Q.....?}.E..c|", "000000f0  6d 41 19 0f c7 d3 0e cf  64 2c 59 fc 86 0a c3 ec  |mA......d,Y.....|", "00000100  29 5d 6f 13 6b 62 fc 80  f4 b4 ce 9b 53 81 36 ea  |)]o.kb......S.6.|", "00000110  5e 07 e0 61 20 d2 15 43  2a 7a 84 f1 a3 0f 9b 8d  |^..a ..C*z......|", "00000120  ba 42 01 94 df f3 d4 87  fc 45 5c 8a dd d2 28 26  |.B.......E\\...(&|", "00000130  a7 35 50 ee 58 1e f3 22  36 1d da 76 2c 07 c9 28  |.5P.X..\"6..v,..(|", "00000140  e1 cb 66 20 cb e4 dd bb  91 eb b0 4f 66 1f 89 69  |..f .......Of..i|", "00000150  4a ec 47 e3 d2 f6 2f 59  18 39 29 42 14 ea f6 19  |J.G.../Y.9)B....|", "00000160  45 a0 e2 44 ed 97 4d 1f  d2 f2 70 b0 13 70 e3 5b  |E..D..M...p..p.[|", "00000170  95 bb 43 6d 9c 35 25 17  14 2d a5 30 46 e3 38 e4  |..Cm.5%..-.0F.8.|", "00000180  55 63 06 15 1c 1e d3 2e  fd 7f 3e 4b 18 07 ed 53  |Uc........>K...S|", "00000190  0b 7e d8 dd 39 0a d2 53  23 10 ce 7a 55 53 5e cf  |.~..9..S#..zUS^.|", "000001a0  77 6d a8 f8 ef e5 8a 01  13 b0 ac ce 31 07 c7 19  |wm..........1...|", "000001b0  69 0e 00 c6 26 88 7a e0  be f5 2c ac 2c 6b ce 8a  |i...&.z...,.,k..|", "000001c0  b3 82 fe fc 16 ea 46 9e  eb 7e 54 f4 b1 4d 00 dc  |......F..~T..M..|", "000001d0  0e e9 5b 3a 8b 5a 49 f0  46 f6 31 a8 9b 39 ba 96  |..[:.ZI.F.1..9..|", "000001e0  b4 ce 76 69 91 5b 74 36  e9 89 cc 67 7c 2e c7 6b  |..vi.[t6...g|..k|", "000001f0  2e b2 f1 b0 fc a6 c6 c1  c9 35 b5 be 23 c1 b3 94  |.........5..#...|", "00000200  11 47 12 1a ea 5e 47 84  a5 92 6c 83 e3 e2 ff 8a  |.G...^G...l.....|", "00000210  ed ba 48 f8 41 17 d5 2e  d2 64 3a 64 64 21 ef 48  |..H.A....d:dd!.H|", "00000220  f8 78 b3 2a ca 0b 1e 7d  80 0b fb 1c b5 73 1b ee  |.x.*...}.....s..|", "00000230  0f a0 4d 6b d9 22 72 06  1c 5d 27 af f6 c3 52 a8  |..Mk.\"r..]'...R.|", "00000240  47 ac bf 18 50 4a 64 29  3c 71 1b 9d 21 37 a9 f1  |G...PJd)<q..!7..|", "00000250  ea 52 7a 25 48 ef 8d 45  97 65 49 8b 1c a0 c5 1d  |.Rz%H..E.eI.....|", "00000260  94 da dc 90 b3 fe 56 2f  5e de 0a 02 58 6b 6f e6  |......V/^...Xko.|", "00000270  e3 0e 93 eb 98 9a 4d 35  4f 8b 50 53 9f f6 35 d5  |......M5O.PS..5.|", "00000280  2d c6 c6 45 d4 38 1d 68  7d b7 46 e0 8a 60 1b ec  |-..E.8.h}.F..`..|", "00000290  28 64 22 67 8b d4 0e 7c  05 28 24 59 fb b6 e5 44  |(d\"g...|.($Y...D|", "000002a0  9e f4 d2 a5 08 cb 3a 52  a5 51 c3 56 cd fb 29 0b  |......:R.Q.V..).|", "000002b0  8d bb 28 ab 06 c5 e8 9c  b8 99 ee 42 a5 4f 1c 0c  |..(........B.O..|", "000002c0  5f 29 1e 2f cf b2 f6 dc  05 21 b1 68 d2 79 ae b2  |_)./.....!.h.y..|", "000002d0  d9 38 5f f6 5d 08 60 44  c1 6b d8 b1 a9 41 b7 48  |.8_.].`D.k...A.H|", "000002e0  44 7a 4d f0 67 ef d9 ca  2f d5 69 9b c3 bf 78 ff  |DzM.g.../.i...x.|", "000002f0  2e f4 a4 94 14 0d fa 7b  60 55 fb 75 fb 49 52 5f  |.......{`U.u.IR_|", "00000300  fc 22 ad 16 ac c2 80 e1  ab cf bf 7f 84 6b df 0e  |.\"...........k..|", "00000310  4c 38 56 02 1a 37 bf 07  b1 e9 ac dc a8 56 f2 bf  |L8V..7.......V..|", "00000320  59 f8 8c 9b 58 88 54 2f  bf d5 5e 1a c4 93 ea f3  |Y...X.T/..^.....|", "00000330  ab 56 f0 a5 7c 66 7c 83  dd 00 30 6f 84 37 72 6b  |.V..|f|...0o.7rk|", "00000340  6a 13 d8 a8 e2 49 c7 2c  e8 b8 c1 9f 29 3e e0 12  |j....I.,....)>..|", "00000350  63 5f 22 df bd f7 9b 25  7e e9 c2 ff 1c 86 f0 e7  |c_\"....%~.......|", "00000360  a9 59 06 18 9d fa 20 eb  2f 72 31 c4 53 c2 dd 09  |.Y.... ./r1.S...|", "00000370  65 3c 9d 5d 7b 51 57 28  de 8b a5 e1 4a 1f 64 2e  |e<.]{QW(....J.d.|", "00000380  37 49 7f bd 15 57 09 dc  fb 74 40 68 00 62 f8 5d  |7I...W...t@h.b.]|", "00000390  99 2c 6e c7 e6 64 bc 3f  26 46 90 f8 69 95 3b 48  |.,n..d.?&F..i.;H|", "000003a0  a7 45 80 09 2d e4 5f c3  89 8e a1 e9 50 c6 87 22  |.E..-._.....P..\"|", "000003b0  04 b5 c6 87 a8 1d 96 ff  65 bd 93 b0 69 62 8d b7  |........e...ib..|", "000003c0  5b 9e e1 34 25 b6 7f 0e  e6 2f 79 b2 ae 85 fc c8  |[..4%..../y.....|", "000003d0  cd 4d dd b3 86 79 89 c8  86 d3 a9 a5 5d aa 9c 0c  |.M...y......]...|", "000003e0  a0 c3 e1 01 cb c8 30 81  46 35 f8 6f 81 4c d3 b5  |......0.F5.o.L..|", "000003f0  be a8 e3 29 96 8c bb 1c  7c 1f fb e6 51 98 b6 90  |...)....|...Q...|", "00000400  c0 a2 fc 9e bb b1 52 f1  b1 42 09 58 12 22 ec 16  |......R..B.X.\"..|", "00000410  5f 81 49 ee 99 0a 6a 88  e9 76 60 4d ee 58 83 d3  |_.I...j..v`M.X..|", "00000420  98 5b 55 ec 59 e8 ab 1a  f6 d8 4a 9c 4b e5 56 66  |.[U.Y.....J.K.Vf|", "00000430  b1 e1 fd 91 59 20 ce 42  db 6b 3d 7c 6a 7e 5e 05  |....Y .B.k=|j~^.|", "00000440  49 a9 35 e3 28 25 08 bf  37 10 cb 60 27 ab c9 17  |I.5.(%..7..`'...|", "00000450  2b 0e e0 54 2a 5a 53 df  80 e6 ca bf 7a de 55 54  |+..T*ZS.....z.UT|", "00000460  71 f8 ca 9e fc 46 b3 c3  7f e1 ae a5 cd 6b 51 17  |q....F.......kQ.|", "00000470  b3 48 fa 00 b9 7a 82 f1  16 02 ab 78 1a ad 99 5f  |.H...z.....x..._|", "00000480  8c c9 1d 53 dc 0e ff 12  b6 ba 6f 0e fe fb f2 31  |...S......o....1|", "00000490  77 e9 f5 17 3d db ab 75  3f ce 76 48 57 37 e3 b4  |w...=..u?.vHW7..|", "000004a0  b0 cc ae 79 f2 50 d4 55  fc 8b 1c aa 14 38 b8 a1  |...y.P.U.....8..|", "000004b0  78 7b 20 72 67 03 25 c7  e7 15 ff a8 86 f9 ec f7  |x{ rg.%.........|", "000004c0  3b 2c 56 20 87 75 53 a4  51 10 d7 1e b8 ea 6d d9  |;,V .uS.Q.....m.|", "000004d0  78 18 98 b0 65 e7 a2 df  fa 0b f2 79 cb c4 51 98  |x...e......y..Q.|", "000004e0  be d0 eb c4 72 b4 a8 48  0f aa 6e 1b dc ff cb 1d  |....r..H..n.....|", "000004f0  aa 34 ee 55 57 19 87 3c  69 67 f3 63 b6 61 48 6d  |.4.UW..<ig.c.aHm|", "00000500  37 7d 18 32 6b 74 f1 59  c5 11 7d 13 e5 4c 21 81  |7}.2kt.Y..}..L!.|", "00000510  5d 79 8a cb 9a 41 7a f9  9d 0e e7 ca 38 66 f4 1a  |]y...Az.....8f..|", "00000520  b7 a1 cd 7e 4d b1 19 f3  5d 53 a9 8d 9e 83 ae 0b  |...~M...]S......|", "00000530  c3 87 cd 4a 6e fe af a0  61 95 30 ee 99 ec 34 9a  |...Jn...a.0...4.|", "00000540  ec ac 19 ff fe 97 69 54  22 cc 04 17 62 a8 ee 7c  |......iT\"...b..||", "00000550  8f a4 a0 50 90 b9 fa e7  3f 5d 7a 2b f8 d1 83 8b  |...P....?]z+....|", "00000560  13 51 6c c4 a1 1c b1 5c  de 83 a6 aa 43 d1 1d c1  |.Ql....\\....C...|", "00000570  e9 e6 60 c0 f2 2c 35 42  b7 71 a4 00 3a b2 db 36  |..`..,5B.q..:..6|", "00000580  73 40 12 90 ee 73 b6 d5  84 7b 4a c2 73 1e 2e af  |s@...s...{J.s...|", "00000590  33 53 82 67 89 10 cb 85  5a 96 d3 af 47 d9 47 07  |3S.g....Z...G.G.|", "000005a0  18 e0 ba 7a aa 0c 69 08  d5 d2 60 a1 2a 38 8e 3e  |...z..i...`.*8.>|", "000005b0  dd de a1 d1 24 32 63 36  55 31 d4 7c 51 1e 83 ea  |....$2c6U1.|Q...|", "000005c0  4d f4 f0 ef 75 0b 89 fa  17 26 fa 81 67 53 36 57  |M...u....&..gS6W|", "000005d0  87 6f ee 61 0d 65 d3 3c  ba b1 04 0c ae 27 c3 7c  |.o.a.e.<.....'.||", "000005e0  19 ce 18 e3 cd 60 37 b4  b1 f7 f4 8b c2 fa 42 e5  |.....`7.......B.|", "000005f0  a6 7f f2 2f 39 c9 94 62  93 68 64 5d 8b 1a 40 9e  |.../9..b.hd]..@.|", "00000600  ab 75 b3 e8 88 4f 5d ea  a9 8e 4b fb c5 51 ca 9c  |.u...O]...K..Q..|", "00000610  49 d2 fc 68 18 4a 22 ce  7d 1d 00 e7 52 61 5f 85  |I..h.J\".}...Ra_.|", "00000620  ec 3c 13 74 a5 78 75 3e  14 f2 27 ed b2 5d 92 16  |.<.t.xu>..'..]..|", "00000630  05 39 dc ee 8c 13 4c af  63 86 1c 3f ac 7e 81 67  |.9....L.c..?.~.g|", "00000640  b0 d7 a2 f3 9f 31 73 ae  97 08 11 2b fa 99 5e c9  |.....1s....+..^.|", "00000650  d1 39 4a 9e a4 bb d1 55  83 c2 f4 2f bd 7e 85 c9  |.9J....U.../.~..|", "00000660  d9 34 c6 61 c6 1d 12 9c  d4 e9 ec 05 4f 04 55 3a  |.4.a........O.U:|", "00000670  78 2f fa 33 02 87 97 e1  65 ef af 8e f0 b2 f9 3c  |x/.3....e......<|", "00000680  75 ca 39 cc 12 2e 58 6e  bd 50 9f 46 83 4f 02 3e  |u.9...Xn.P.F.O.>|", "00000690  50 8d 0a 8f 11 a2 06 70  70 0f 94 91 2d c7 da c4  |P......pp...-...|", "000006a0  9a 22 eb 9f 14 04 c4 d0  35 c7 c9 60 9d b0 10 c3  |.\"......5..`....|", "000006b0  ab fd c7 31 a4 b7 23 78  10 17 3b de 05 c8 f1 1b  |...1..#x..;.....|", "000006c0  90 32 54 ff eb bc 1f e4  11 78 f9 9c 2f ab 49 75  |.2T......x../.Iu|", "000006d0  9f 42 89 54 77 9a 0b 0c  a7 e8 b2 18 85 cd 2a d0  |.B.Tw.........*.|", "000006e0  39 a0 09 96 09 2c a5 34  ef e9 85 0b d8 ed 22 e0  |9....,.4......\".|", "000006f0  7c 80 07 8b 12 ed 60 b1  e8 ec 4e 26 97 f0 50 e1  ||.....`...N&..P.|", "00000700  2f 2b 08 b8 03 87 7a da  44 7d 93 16 b5 70 c7 a2  |/+....z.D}...p..|", "00000710  2f 5b 34 10 5d dc f6 f4  46 8c 8e 5c 4c aa db 44  |/[4.]...F..\\L..D|", "00000720  ce fa ea 39 cb 23 e8 0c  f4 13 99 e4 20 c7 9c 6a  |...9.#...... ..j|", "00000730  9e 37 b0 e0 42 af b3 59  98 6b fd 66 61 85 6e dc  |.7..B..Y.k.fa.n.|", "00000740  7c 53 b4 0b 16 9f 92 07  8b 4e 8d fc 05 0b 4e 1a  ||S.......N....N.|", "00000750  c6 b8 e0 4f 1b 6d 4f 3a  9d 2c 7c 3e 0a 42 e4 ba  |...O.mO:.,|>.B..|", "00000760  3c 6f d3 f1 88 d2 e8 69  3a d0 dd c2 b2 15 17 d0  |<o.....i:.......|", "00000770  89 88 8a e7 34 fa 57 07  b3 37 89 5f 39 4a 2d 02  |....4.W..7._9J-.|", "00000780  58 e6 7b 91 1b 15 5a 44  6e 38 e5 4b ef 8c 42 00  |X.{...ZDn8.K..B.|", "00000790  b3 31 26 d7 6a 79 b8 e6  ca 72 88 c6 e7 f8 55 4e  |.1&.jy...r....UN|", "000007a0  62 15 84 08 2b dc 6b 52  7f 9b 6d 7e 47 6f 35 ca  |b...+.kR..m~Go5.|", "000007b0  a5 4d 10 07 54 d2 86 ea  5e e0 33 36 72 93 d9 6f  |.M..T...^.36r..o|", "000007c0  4f 09 19 63 8c 6b 04 8f  6d a9 c6 29 09 e2 63 61  |O..c.k..m..)..ca|", "000007d0  6a 74 83 e5 9d 6c c1 b3  31 20 2b 97 34 88 d5 66  |jt...l..1 +.4..f|", "000007e0  7c a4 f8 fa 04 b1 4e 2e  1c 81 23 79 9a 09 92 11  ||.....N...#y....|", "000007f0  94 ba 59 25 7f ea a6 02  0d c1 0b f3 f3 b6 52 88  |..Y%..........R.|", "00000800  fb 00 ac e4 bc 43 6a a2  ff 9b 5f c9 72 ad fc 3d  |.....Cj..._.r..=|", "00000810  02 41 2a 4e 36 d5 7f 19  b3 34 f2 7a 5a d7 42 62  |.A*N6....4.zZ.Bb|", "00000820  4b 6e 52 2a 63 b7 fd 1f  b9 97 18 88 53 7d d4 f5  |KnR*c.......S}..|", "00000830  21 04 f4 ae bd 9d ea f7  44 7b f5 49 1a bf 0c 8c  |!.......D{.I....|", "00000840  15 0a fd b1 35 9c 57 3b  8f 2a 71 e6 40 cf 93 c8  |....5.W;.*q.@...|", "00000850  b2 80 1d 8f 19 46 b7 16  ad bf f9 59 16 f5 1b 8d  |.....F.....Y....|", "00000860  aa ea 1e b2 d2 34 92 c2  aa 69 d0 81 2e 50 23 db  |.....4...i...P#.|", "00000870  29 f4 a7 cb b2 00 1d e5  a7 12 be d9 36 35 fe 6d  |)...........65.m|", "00000880  b2 97 b5 0c 84 3e 7b be  46 b5 50 9e 0c 3c a0 3a  |.....>{.F.P..<.:|", "00000890  d3 d3 d8 e8 16 c2 3e c1  ee 79 43 06 95 ad eb a3  |......>..yC.....|", "000008a0  cd e3 6f 5b f2 9f 3f 12  69 2f eb 25 2f 14 2f 3d  |..o[..?.i/.%/./=|", "000008b0  aa 0f f8 fa e1 0b 6c 4c  55 a5 9d 4a 87 3b 8c 2c  |......lLU..J.;.,|", "000008c0  65 bc 0c b8 10 6a ed 82  bc 6e 98 6e 15 82 c8 ba  |e....j...n.n....|", "000008d0  83 9a 43 19 88 c4 56 3d  f9 56 10 bd c0 58 5e 1b  |..C...V=.V...X^.|", "000008e0  14 9a d2 56 56 c7 fc 66  59 ff 81 77 1d 29 fe 0c  |...VV..fY..w.)..|", "000008f0  82 ca 17 75 d7 65 8f 04  3a 5b c9 af cf f3 5c 18  |...u.e..:[....\\.|", "00000900  f8 57 1d 33 3b 73 7a a3  f4 a8 f0 a9 e3 aa bc e5  |.W.3;sz.........|", "00000910  db 15 cb 51 d8 54 b2 f0  e2 e6 7d 39 59 8a 3b ec  |...Q.T....}9Y.;.|", "00000920  ee 2e bf db ed d5 33 09  6e 26 d2 30 7f b9 58 be  |......3.n&.0..X.|", "00000930  87 06 b1 ac 8a 07 2d b8  ee d1 92 55 21 5d d2 c1  |......-....U!]..|", "00000940  2b 5b 4b e7 36 0b f7 1f  8d 96 0e 76 25 01 80 0d  |+[K.6......v%...|", "00000950  0e 93 98 43 c4 b9 95 e4  a6 28 e3 fc b8 b9 5b e9  |...C.....(....[.|", "00000960  af 18 a6 bc 4d dc 2e 83  86 14 5c 95 c2 1b e3 bf  |....M.....\\.....|", "00000970  17 6e 72 8b 48 8b ed 1f  7a 99 b5 4a ad 8e 60 62  |.nr.H...z..J..`b|", "00000980  40 2e 66 c5 8b 54 2f 3f  df ac 0d 21 01 65 ad 53  |@.f..T/?...!.e.S|", "00000990  d6 30 ce ef 8e fc f4 97  94 4e f2 52 f7 39 ef ee  |.0.......N.R.9..|", "000009a0  b7 e7 31 01 54 6c 9f d0  6a cc 22 42 8c c3 f8 a8  |..1.Tl..j.\"B....|", "000009b0  9b b9 fb 1e e5 ed a0 54  d4 6a e0 53 4e 3d f3 fc  |.......T.j.SN=..|", "000009c0  d2 f7 04 8c 62 01 bf 68  d4 05 5c 3a 01 7c 17 d5  |....b..h..\\:.|..|", "000009d0  ea d2 4c ed 87 32 81 68  f8 4d 36 70 aa 31 86 75  |..L..2.h.M6p.1.u|", "000009e0  76 f0 c6 21 f1 0e d8 4e  74 de 9e e7 74 bb f5 dd  |v..!...Nt...t...|", "000009f0  f5 9c 68 9f 44 c4 c6 d2  bd f9 fe 6d 63 66 5d 6e  |..h.D......mcf]n|", "00000a00  64 45 28 cb 5d 4c 0d 80  c7 8c 5b 83 53 77 4b 7c  |dE(.]L....[.SwK||", "00000a10  f1 32 c1 f9 1f c5 11 e5  39 2a e6 a9 0c de 68 91  |.2......9*....h.|", "00000a20  66 6b 38 04 ed fc d7 60  6e a7 29 7e 5b 13 47 fe  |fk8....`n.)~[.G.|", "00000a30  a7 e0 53 23 f5 aa 2a bd  91 d7 fd 02 ae f3 29 b1  |..S#..*.......).|", "00000a40  df 0b cd e0 47 c6 cf 20  03 3e 93 a0 25 bb 4c 13  |....G.. .>..%.L.|", "00000a50  5c fc dc 5c 35 55 a3 f5  86 0b da 9a 1e 63 e4 de  |\\..\\5U.......c..|", "00000a60  bc 83 1d f3 af dc 49 e8  21 5f 30 6d 14 f2 cc 00  |......I.!_0m....|", "00000a70  2d 7a 07 47 99 21 a9 98  bf 57 bb 99 f2 d4 32 38  |-z.G.!...W....28|", "00000a80  a7 67 8a 5f 17 06 e1 1c  9d 5d 69 f0 76 64 3d 45  |.g._.....]i.vd=E|", "00000a90  ad c0 e6 dc 7d e9 59 8d  9b 33 4c 04 a7 be 2d b5  |....}.Y..3L...-.|", "00000aa0  b0 f5 26 14 86 b9 de 45  f4 d8 8a 59 27 db 38 56  |..&....E...Y'.8V|", "00000ab0  7c 4b 1b 52 37 a7 9f 71  00 75 66 3a 2f 0a e3 f9  ||K.R7..q.uf:/...|", "00000ac0  c7 21 dd 70 50 52 14 9f  ff 7f 2f d5 21 2e c6 ba  |.!.pPR..../.!...|", "00000ad0  5c 3d 84 39 be ef 25 db  f2 dd ec 53 63 1a 69 fd  |\\=.9..%....Sc.i.|", "00000ae0  be 71 59 a4 de df 11 3e  ee ae c9 5d e8 75 1c cc  |.qY....>...].u..|", "00000af0  1c 00 db e6 42 1b ba 99  03 29 0b 83 02 12 bc 7e  |....B....).....~|", "00000b00  dd a8 7e 2a 68 41 71 d8  28 b1 9c cd a3 fa 6c 3a  |..~*hAq.(.....l:|", "00000b10  88 78 69 2d 94 6e dc 85  a6 d4 59 02 7f c2 2f 01  |.xi-.n....Y.../.|", "00000b20  0f 06 03 ac ac 30 bb b6  84 43 3f 0c 4d 9a 63 2a  |.....0...C?.M.c*|", "00000b30  22 98 fb b4 2c a0 b8 49  87 ba 13 ed 0b 30 c0 39  |\"...,..I.....0.9|", "00000b40  f5 5b 31 7d dd 54 89 a8  2d cf fd 79 b9 ef 3a 50  |.[1}.T..-..y..:P|", "00000b50  85 54 fd 5a de 21 8c 73  f5 e8 38 30 db dc 23 96  |.T.Z.!.s..80..#.|", "00000b60  c7 41 f1 94 d8 ed b6 85  ef fd 81 3f b0 a8 a1 6a  |.A.........?...j|", "00000b70  41 7c f5 8d 3c 17 64 cf  80 37 66 78 02 e4 05 e5  |A|..<.d..7fx....|", "00000b80  80 00 87 c4 dc f2 37 fd  ed f3 b7 a2 91 a8 90 04  |......7.........|", "00000b90  85 2c ac 52 3f d0 0a 39  20 b1 50 34 03 57 80 ed  |.,.R?..9 .P4.W..|", "00000ba0  8b 60 87 1b f3 59 89 f8  a0 6d f8 fa 8d 70 b9 f2  |.`...Y...m...p..|", "00000bb0  57 3b cc db f4 6c 75 c3  b0 f0 54 86 e4 52 50 cc  |W;...lu...T..RP.|", "00000bc0  97 8e fc 69 6b 26 ba e0  2e d5 10 8c 7b 84 8f 4e  |...ik&......{..N|", "00000bd0  1b 40 c9 d5 f4 95 e5 c2  6c 91 75 eb 0e 85 18 3d  |.@......l.u....=|", "00000be0  7f 57 24 de bd 05 fb bd  83 ac 08 30 40 f0 00 06  |.W$........0@...|", "00000bf0  8a 98 c7 dd 3b 08 c8 84  72 d4 88 9f f9 9d 96 9e  |....;...r.......|", "00000c00  d3 15 05 95 db 03 fe c5  e3 7f 31 68 ab 40 29 b9  |..........1h.@).|", "00000c10  ff 66 87 cc 08 cc 49 9f  5f e4 24 de 12 fd 60 d7  |.f....I._.$...`.|", "00000c20  0f 3e a0 7d ca 34 91 29  1c d9 88 6d 9d fb 56 16  |.>.}.4.)...m..V.|", "00000c30  bd 55 0c c7 1b f2 4a 2d  eb ab 69 8f f3 fa 83 dc  |.U....J-..i.....|", "00000c40  ae 6a 58 9a 4c c7 a6 50  88 fd 15 d5 df 1f a9 9b  |.jX.L..P........|", "00000c50  91 c1 2e 79 c1 e0 1f 42  2e 48 42 db 9c be 5f 1c  |...y...B.HB..._.|", "00000c60  31 88 dc ad db 6d f6 27  45 bc 03 57 c6 f1 26 0b  |1....m.'E..W..&.|", "00000c70  f2 1f 1d 42 44 c3 53 cd  77 e5 e2 90 64 c1 1f 77  |...BD.S.w...d..w|", "00000c80  ff ef 02 00 54 21 0c 6a  a2 d4 4b 9c d6 3c e8 99  |....T!.j..K..<..|", "00000c90  e9 64 8c 10 7e ba b7 fd  e6 68 62 b9 06 3c d8 c1  |.d..~....hb..<..|", "00000ca0  5e ca 51 dd 60 df 99 28  0a 16 30 58 8d 5f 58 1d  |^.Q.`..(..0X._X.|", "00000cb0  47 3a 49 de 5f 11 3f d0  f3 05 d4 9f 97 a4 da 31  |G:I._.?........1|", "00000cc0  7e 25 73 f7 0b ef 8f e8  c4 3d 57 ad 77 6c 78 56  |~%s......=W.wlxV|", "00000cd0  9c 8b a5 10 1a 32 f1 75  ca 9e 06 59 da 58 d6 3f  |.....2.u...Y.X.?|", "00000ce0  43 97 4d c7 b5 da 41 f8  8e b4 1f 4c 18 d9 07 c2  |C.M...A....L....|", "00000cf0  68 85 a0 94 a9 38 a6 b9  cf 16 5f 36 fd 3f 2b da  |h....8...._6.?+.|", "00000d00  87 d6 29 95 91 71 a3 47  5e 83 8d fe ba e6 a3 56  |..)..q.G^......V|", "00000d10  c4 73 c5 d2 d6 ea 0c 6f  c7 c1 b8 84 68 a7 9e de  |.s.....o....h...|", "00000d20  92 f7 dd a0 91 ca 78 79  65 88 90 b7 7e eb 9a 60  |......xye...~..`|", "00000d30  72 2b 29 e3 56 04 6b 4b  e7 38 27 bb ad 93 99 13  |r+).V.kK.8'.....|", "00000d40  fe df 5a e2 5f 66 41 f3  99 11 ea 68 18 83 4c 3d  |..Z._fA....h..L=|", "00000d50  a8 01 8a 6b a8 aa f6 3f  7f 76 06 6b ad 04 25 fe  |...k...?.v.k..%.|", "00000d60  b4 ab 49 04 21 a3 e3 e3  03 c2 ba 32 04 16 48 e5  |..I.!......2..H.|", "00000d70  d7 8c 2f 52 e6 3f 8d b8  94 60 e8 04 e9 34 9e fd  |../R.?...`...4..|", "00000d80  fa 41 12 13 da 54 7d 2f  ff 6a 3e de 26 e1 9b 34  |.A...T}/.j>.&..4|", "00000d90  8e 26 2b 24 d0 eb b3 22  35 1a ad b2 70 04 9b d4  |.&+$...\"5...p...|", "00000da0  1e 90 46 90 de 74 9e fe  a7 6e a8 59 a3 b9 b8 1a  |..F..t...n.Y....|", "00000db0  a3 9b 65 e0 d2 64 53 4b  5e 42 20 41 9c 55 60 48  |..e..dSK^B A.U`H|", "00000dc0  b2 30 f7 f6 18 2b 71 64  cd 86 11 0e 0e a3 d5 f4  |.0...+qd........|", "00000dd0  b0 ca b9 b1 bd 76 cd 40  d2 ef 12 46 3c 28 ec f6  |.....v.@...F<(..|", "00000de0  33 96 dc 26 33 e0 08 46  f2 68 1e 43 a7 38 16 80  |3..&3..F.h.C.8..|", "00000df0  9c 6d 8b 28 c0 03 a7 99  22 61 0a 47 34 5e 5e 58  |.m.(....\"a.G4^^X|", "00000e00  67 67 db de df cf ff c0  05 dc 09 84 e2 28 60 2f  |gg...........(`/|", "00000e10  f7 08 3d a8 8a 69 4a 0b  1c c5 44 f1 29 11 3e 71  |..=..iJ...D.).>q|", "00000e20  ae f3 d1 c2 37 d1 18 79  ab 64 8f 82 bf 38 7b 1b  |....7..y.d...8{.|", "00000e30  52 ef 00 9a 4f 44 dc 4c  7c f9 c9 47 ba a8 6d fa  |R...OD.L|..G..m.|", "00000e40  1d aa f5 ff c4 c2 0d 53  27 e6 17 d0 be 6d 97 30  |.......S'....m.0|", "00000e50  81 1c 9c c6 75 35 14 ff  73 d8 78 27 92 51 8e fc  |....u5..s.x'.Q..|", "00000e60  94 ec 64 45 19 5c 0b be  b5 99 e9 b9 76 5f 70 de  |..dE.\\......v_p.|", "00000e70  00 b3 16 08 b0 76 06 a9  77 8a 79 c5 fc 0b f1 6c  |.....v..w.y....l|", "00000e80  03 c1 0f a9 e9 70 19 ba  fa b2 4f e2 13 33 6a 06  |.....p....O..3j.|", "00000e90  1c af 25 2c eb 94 3b 18  31 0a d0 b9 a6 13 53 7c  |..%,..;.1.....S||", "00000ea0  3a 00 57 23 05 41 34 2c  7c 4f e2 f3 72 0b e2 a7  |:.W#.A4,|O..r...|", "00000eb0  e0 78 02 ab 7f 2c 0e 32  c3 d4 e4 b8 2b 2f 31 fb  |.x...,.2....+/1.|", "00000ec0  f8 92 db 7a 41 4d 98 71  b9 a3 0d eb fd 71 fc 2e  |...zAM.q.....q..|", "00000ed0  e8 5c a1 b5 58 f8 5a 11  20 04 da 52 39 b6 06 70  |.\\..X.Z. ..R9..p|", "00000ee0  43 0d 7a 48 14 18 21 03  2c 61 37 ef 23 41 b8 41  |C.zH..!.,a7.#A.A|", "00000ef0  3b e4 a0 f6 71 2a d2 06  97 b0 5a d2 81 36 b0 6f  |;...q*....Z..6.o|", "00000f00  00 ed db 5f 96 bd 37 bd  7c bc 33 47 37 67 2a ad  |..._..7.|.3G7g*.|", "00000f10  b6 81 e1 8d 5a e2 68 c2  0b 32 d3 f1 21 02 77 3b  |....Z.h..2..!.w;|", "00000f20  89 ee da 0d 87 8f 26 0f  73 17 19 3b b7 04 e5 38  |......&.s..;...8|", "00000f30  54 c4 f3 1b 7b bc 93 9a  ce 38 87 f2 3f a2 8b 38  |T...{....8..?..8|", "00000f40  a9 b1 7c 51 a2 34 f0 72  43 5c 59 35 b6 fd 50 2e  |..|Q.4.rC\\Y5..P.|", "00000f50  db dc 29 d4 65 6a ef 23  83 7d 9d 49 b7 56 e4 c0  |..).ej.#.}.I.V..|", "00000f60  4e 4d d8 06 ca e5 85 ee  41 cb 66 84 1a af 79 7f  |NM......A.f...y.|", "00000f70  f0 9e 7c 74 8c 4a 80 e2  e6 0a 58 fb c9 61 3f a1  |..|t.J....X..a?.|", "00000f80  b7 3b 7a ef 1f 91 95 d5  51 ab 01 ef 0e 86 c4 bb  |.;z.....Q.......|", "00000f90  78 0e cb f3 b5 f9 30 d8  ae f0 81 11 e9 24 e6 e2  |x.....0......$..|", "00000fa0  67 8a 30 9c 04 a6 a1 cb  d2 73 01 81 fb 01 28 f0  |g.0......s....(.|", "00000fb0  62 2d d5 91 24 9e ec f8  74 3e 26 63 46 a0 b1 c2  |b-..$...t>&cF...|", "00000fc0  63 31 0d 91 e8 fb 0d 6d  43 e0 8b ce 6b 8b 77 a2  |c1.....mC...k.w.|", "00000fd0  63 56 28 c8 60 8e c1 53  ae 70 3d 06 54 b5 f4 9a  |cV(.`..S.p=.T...|", "00000fe0  8a be 4c 43 9c f6 64 40  6e 97 2c 97 cd 74 c8 83  |..LC..d@n.,..t..|", "00000ff0  b3 0b 41 74 56 fc 31 93  a5 dd c2 15 8c dd 9d 43  |..AtV.1........C|", "00001000", "", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 rocksdb: verify_sharding unable to list column families: NotFound: ", "2022-05-09T11:47:08.772+0000 7f5402bfff00 -1 bluestore(/var/lib/ceph/osd/ceph-4) _open_db erroring opening db: ", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error", "2022-05-09T11:47:09.796+0000 7f5402bfff00 -1 \u001b[0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-4: (5) Input/output error\u001b[0m"]

a simple retry with the exact same command then works.

#6 Updated by Christian Rohmann about 1 year ago

Thanks for fixing this ... but somehow the link to the ML (https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/) with Igors discussion seems to 404 now.

#7 Updated by Igor Fedotov about 1 year ago

Christian Rohmann wrote:

Thanks for fixing this ... but somehow the link to the ML (https://lists.ceph.io/hyperkitty/list/dev@ceph.io/thread/CHVBMPENHOQXHSMW3YNRO3L2HA2R7GCZ/) with Igors discussion seems to 404 now.

Here is another link: https://lists.ceph.io/hyperkitty/list/dev@ceph.io/message/K6WSU4NCELZHDNZKC43YNUDM2WGGIHO7/

We experienced some issues with the mail-archive a few weeks ago hence the update...

#8 Updated by Igor Fedotov about 1 year ago

  • Status changed from New to Pending Backport
  • Backport set to quincy, pacific

#9 Updated by Igor Fedotov about 1 year ago

  • Pull request ID set to 49431

#10 Updated by Igor Fedotov about 1 year ago

  • Backport changed from quincy, pacific to quincy, reef

#11 Updated by Backport Bot about 1 year ago

  • Copied to Backport #58633: quincy: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error added

#12 Updated by Backport Bot about 1 year ago

  • Tags changed from osd to osd backport_processed

#13 Updated by Richard Hesse about 1 year ago

Is there a nightly container build for Pacific that includes this fix? This issue has crippled my previously working 16.2.10 cluster. All of the simple volumes are fine. All of the LVM based volumes are unable to start.

#14 Updated by Radoslaw Zarzynski about 1 year ago

  • Copied to Backport #58952: reef: OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error added

#15 Updated by Igor Fedotov 12 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF