Ceph » Orchestrator

Failure that is reported

Command failed on smithi152 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 29c90c2a-baf1-11ec-8c38-001a4aab830c -- ceph orch daemon add osd smithi152:vg_nvme/lv_4'

Failure that shows up in the logs

2022-04-13T06:19:15.901 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr Running command: /usr/bin/ceph-osd --cluster ceph --osd-objectstore bluestore --mkfs -i 3 --monmap /var/lib/ceph/osd/ceph-3/activate.monmap --keyfile - --osdspec-affinity None --osd-data /var/lib/ceph/osd/ceph-3/ --osd-uuid 15fcaad2-921d-48d6-902e-b6a81e5524bb --setuser ceph --setgroup ceph
2022-04-13T06:19:15.901 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 2022-04-13T06:19:13.976+0000 7f24ba29c080 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _read_fsid unparsable uuid
2022-04-13T06:19:15.901 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 2022-04-13T06:19:14.337+0000 7f24ba29c080 -1 bluefs _replay 0x0: stop: uuid 00000000-0000-0000-0000-000000000000 != super.uuid cc4f1c6e-a26e-4aa1-ab2e-c44a5d12e79a, block dump:
2022-04-13T06:19:15.901 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 00000000  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2022-04-13T06:19:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: *
2022-04-13T06:19:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 00000ff0  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
2022-04-13T06:19:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 00001000
2022-04-13T06:19:15.902 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 2022-04-13T06:19:14.751+0000 7f24ba29c080 -1 bluestore(/var/lib/ceph/osd/ceph-3/) _open_db erroring opening db:
2022-04-13T06:19:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 2022-04-13T06:19:15.259+0000 7f24ba29c080 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2022-04-13T06:19:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr  stderr: 2022-04-13T06:19:15.259+0000 7f24ba29c080 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/osd/ceph-3/: (5) Input/output error[0m
2022-04-13T06:19:15.903 INFO:teuthology.orchestra.run.smithi152.stderr:/bin/podman: stderr --> Was unable to complete a new OSD, will rollback changes

Possible this is an issue with specific hardware since it's an "Input/output error". Will need to track which hosts this is appearing on and see if it's only on certain hosts.