can't reuse OSD ID if ID doesn't exist in `ceph osd tree`
ceph-volume will skip adding the auth required, the monitor will not know about the OSD and it will end up with the OSD not coming up.
Both these lines (one for filestore the other one for bluestore):
# allow re-using an id, in case a prepare failed osd_id = id_ or prepare_utils.create_id(fsid, json_secrets)
Need to change so that a call to the cluster confirms that the ID is really there so there is no need to recreate it.
`ceph osd tree` can do this, but it seems that the admin keyring is required, which would be a no go for ceph-volume since it can't guarantee that an OSD node will have that around.
The workaround here would be to make create_id check if the id_ is already there, and ask for a new one otherwise.
Failure in OSD logs looks similar to this one line:
2018-01-09 23:00:52.378256 7fc3afa53e00 -1 osd.3 0 init authentication failed: (1) Operation not permitted