Looking at the logs for the OSD that failed:
2018-04-26 08:53:01.504628 7f8b27cb0d00 0 set uid:gid to 167:167 (ceph:ceph)
2018-04-26 08:53:01.504666 7f8b27cb0d00 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 33448
2018-04-26 08:53:01.505092 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) set_cache_shards 1
2018-04-26 08:53:01.509392 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) _set_csum csum_type crc32c
2018-04-26 08:53:01.511346 7f8b27cb0d00 1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) mkfs path /var/lib/ceph/tmp/mnt.90QTvg
2018-04-26 08:53:01.511356 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _read_bdev_label
2018-04-26 08:53:01.511866 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:01.511896 7f8b27cb0d00 1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) mkfs already created
2018-04-26 08:53:01.511900 7f8b27cb0d00 1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) _fsck repair (shallow) start
2018-04-26 08:53:01.511954 7f8b27cb0d00 1 bdev create path /var/lib/ceph/tmp/mnt.90QTvg/block type kernel
2018-04-26 08:53:01.511963 7f8b27cb0d00 1 bdev(0x55bd2cfc3e00 /var/lib/ceph/tmp/mnt.90QTvg/block) open path /var/lib/ceph/tmp/mnt.90QTvg/block
2018-04-26 08:53:01.512393 7f8b27cb0d00 1 bdev(0x55bd2cfc3e00 /var/lib/ceph/tmp/mnt.90QTvg/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) rotational
2018-04-26 08:53:01.512408 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _read_bdev_label
2018-04-26 08:53:01.512524 7f8b27cb0d00 10 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:01.512536 7f8b27cb0d00 -1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.90QTvg/block fsid 607e79c6-8970-408c-9679-64c4ce3e59bd does not match our fsid 02da9e3b-b896-441e-b46f-4e76699219b7
2018-04-26 08:53:01.512544 7f8b27cb0d00 1 bdev(0x55bd2cfc3e00 /var/lib/ceph/tmp/mnt.90QTvg/block) close
2018-04-26 08:53:01.770845 7f8b27cb0d00 -1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg) mkfs fsck found fatal error: (5) Input/output error
2018-04-26 08:53:01.770892 7f8b27cb0d00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2018-04-26 08:53:01.771184 7f8b27cb0d00 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.90QTvg: (5) Input/output error[0m
2018-04-26 08:53:03.392737 7efd11702d00 0 set uid:gid to 167:167 (ceph:ceph)
2018-04-26 08:53:03.392767 7efd11702d00 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 33524
2018-04-26 08:53:03.393076 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) set_cache_shards 1
2018-04-26 08:53:03.396358 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) _set_csum csum_type crc32c
2018-04-26 08:53:03.398031 7efd11702d00 1 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) mkfs path /var/lib/ceph/tmp/mnt.BFbBCU
2018-04-26 08:53:03.398040 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU/block) _read_bdev_label
2018-04-26 08:53:03.398440 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:03.398488 7efd11702d00 1 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) mkfs already created
2018-04-26 08:53:03.398496 7efd11702d00 1 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) _fsck repair (shallow) start
2018-04-26 08:53:03.398589 7efd11702d00 1 bdev create path /var/lib/ceph/tmp/mnt.BFbBCU/block type kernel
2018-04-26 08:53:03.398605 7efd11702d00 1 bdev(0x55a712649e00 /var/lib/ceph/tmp/mnt.BFbBCU/block) open path /var/lib/ceph/tmp/mnt.BFbBCU/block
2018-04-26 08:53:03.399337 7efd11702d00 1 bdev(0x55a712649e00 /var/lib/ceph/tmp/mnt.BFbBCU/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) rotational
2018-04-26 08:53:03.399364 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU/block) _read_bdev_label
2018-04-26 08:53:03.399562 7efd11702d00 10 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:03.399583 7efd11702d00 -1 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.BFbBCU/block fsid 607e79c6-8970-408c-9679-64c4ce3e59bd does not match our fsid 02da9e3b-b896-441e-b46f-4e76699219b7
2018-04-26 08:53:03.399598 7efd11702d00 1 bdev(0x55a712649e00 /var/lib/ceph/tmp/mnt.BFbBCU/block) close
2018-04-26 08:53:03.657847 7efd11702d00 -1 bluestore(/var/lib/ceph/tmp/mnt.BFbBCU) mkfs fsck found fatal error: (5) Input/output error
2018-04-26 08:53:03.657886 7efd11702d00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2018-04-26 08:53:03.658126 7efd11702d00 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.BFbBCU: (5) Input/output error[0m
2018-04-26 08:53:05.227931 7fe9d1706d00 0 set uid:gid to 167:167 (ceph:ceph)
2018-04-26 08:53:05.227971 7fe9d1706d00 0 ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous (stable), process (unknown), pid 33598
2018-04-26 08:53:05.228290 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) set_cache_shards 1
2018-04-26 08:53:05.232601 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) _set_csum csum_type crc32c
2018-04-26 08:53:05.234586 7fe9d1706d00 1 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) mkfs path /var/lib/ceph/tmp/mnt.svj4hW
2018-04-26 08:53:05.234600 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW/block) _read_bdev_label
2018-04-26 08:53:05.234973 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:05.235017 7fe9d1706d00 1 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) mkfs already created
2018-04-26 08:53:05.235023 7fe9d1706d00 1 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) _fsck repair (shallow) start
2018-04-26 08:53:05.235103 7fe9d1706d00 1 bdev create path /var/lib/ceph/tmp/mnt.svj4hW/block type kernel
2018-04-26 08:53:05.235122 7fe9d1706d00 1 bdev(0x5654f61f9e00 /var/lib/ceph/tmp/mnt.svj4hW/block) open path /var/lib/ceph/tmp/mnt.svj4hW/block
2018-04-26 08:53:05.235527 7fe9d1706d00 1 bdev(0x5654f61f9e00 /var/lib/ceph/tmp/mnt.svj4hW/block) open size 96636764160 (0x1680000000, 92160 MB) block_size 4096 (4096 B) rotational
2018-04-26 08:53:05.235556 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW/block) _read_bdev_label
2018-04-26 08:53:05.235712 7fe9d1706d00 10 bluestore(/var/lib/ceph/tmp/mnt.svj4hW/block) _read_bdev_label got bdev(osd_uuid 607e79c6-8970-408c-9679-64c4ce3e59bd, size 0x1680000000, btime 2018-04-12 07:25:03.997935, desc main, 7 meta)
2018-04-26 08:53:05.235732 7fe9d1706d00 -1 bluestore(/var/lib/ceph/tmp/mnt.svj4hW/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.svj4hW/block fsid 607e79c6-8970-408c-9679-64c4ce3e59bd does not match our fsid 02da9e3b-b896-441e-b46f-4e76699219b7
2018-04-26 08:53:05.235748 7fe9d1706d00 1 bdev(0x5654f61f9e00 /var/lib/ceph/tmp/mnt.svj4hW/block) close
2018-04-26 08:53:05.493646 7fe9d1706d00 -1 bluestore(/var/lib/ceph/tmp/mnt.svj4hW) mkfs fsck found fatal error: (5) Input/output error
2018-04-26 08:53:05.493696 7fe9d1706d00 -1 OSD::mkfs: ObjectStore::mkfs failed with error (5) Input/output error
2018-04-26 08:53:05.493946 7fe9d1706d00 -1 [0;31m ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.svj4hW: (5) Input/output error[0m
This line is implying that the device was used for a previous cluster, and was not cleaned up:
2018-04-26 08:53:01.512536 7f8b27cb0d00 -1 bluestore(/var/lib/ceph/tmp/mnt.90QTvg/block) _check_or_set_bdev_label bdev /var/lib/ceph/tmp/mnt.90QTvg/block fsid 607e79c6-8970-408c-9679-64c4ce3e59bd does not match our fsid 02da9e3b-b896-441e-b46f-4e76699219b7