Here's an update from the bug submitter.
1. The osdmaptool approach failed since there is no longer any command to inject osdmaps into mon (since several years).
2. I've instead written a patch for tools/ceph_monstore_tool.cc which works for monstores, but the osdstores remain to be fixed.
3. I've traced the issues I have with the 'osd new' command not doing what it should again. My related findings below:
A. The man page and ceph command output disagree about options ordering. Man page says 'id' first and 'uuid' later. "ceph osd help" says uuid first and id or osdname later.
B. When issuing the command as "ceph osd new -i ceph-new-secrets.json <uuid-of-osd.1> 1, and having a 'osd.1' ceph auth entry, I received first "caps do not match". After staring/tracing the source code for a while I found the reason to: The caps one needs to have on an OSD for 'osd new' are hardcoded and defined at https://github.com/ceph/ceph/blob/v12.1.4/src/mon/AuthMonitor.cc#L878-L882
C. After having updated my caps to be identical to those of the hard coding in the source code, I am still returned here https://github.com/ceph/ceph/blob/v12.1.4/src/mon/OSDMonitor.cc#L6916
But this is incorrect! One IS idempotent as per https://github.com/ceph/ceph/blob/v12.1.4/src/mon/OSDMonitor.cc#L6773-L6779 , after having run 'osd destroy osd.1 --yes-i-really-mean-it'. At least on 12.1.4 as it seems.
The OSD remains in the osdmap as per my osd dump:
root@hostname:~/sdfsdf# ceph osd dump
epoch 5965
fsid a2ff4e1f-54c0-476a-8891-45c305b1a2e9
created 2016-08-08 01:23:30.220664
modified 2017-08-17 09:32:42.061199
flags noout,noscrub,nodeep-scrub,sortbitwise,recovery_deletes
crush_version 3
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 4 'ec' erasure size 5 min_size 3 crush_rule 1 object_hash rjenkins pg_num 160 pgp_num 160 last_change 96 flags hashpspool stripe_width 4128
pool 5 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 160 pgp_num 160 last_change 121 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 8 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 160 pgp_num 160 last_change 215 flags hashpspool stripe_width 0
pool 9 'cephfs_data' erasure size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 160 pgp_num 160 last_change 216 flags hashpspool stripe_width 12288
max_osd 6
osd.1 down in weight 1 up_from 5659 up_thru 5952 down_at 5960 last_clean_interval [4927,5643) <$IPADDR>:6805/6161 <$IPADDR>:6806/6161 <$IPADDR>:6807/6161 <$IPADDR>:6808/6161 destroyed,exists d706a467-bb1b-40ce-a07d-fd8fdc0ea427
osd.2 down in weight 1 up_from 5648 up_thru 5958 down_at 5960 last_clean_interval [5005,5643) <$IPADDR>:6801/6108 <$IPADDR>:6802/6108 <$IPADDR>:6803/6108 <$IPADDR>:6804/6108 destroyed,exists e3031736-acb4-4a5e-a67f-f9b171167cbd
osd.3 down in weight 1 up_from 5653 up_thru 5954 down_at 5960 last_clean_interval [5468,5643) <$IPADDR>:6809/6211 <$IPADDR>:6810/6211 <$IPADDR>:6811/6211 <$IPADDR>:6812/6211 destroyed,exists 4288d76d-2e14-4557-8d54-216319eb3581
osd.4 down in weight 1 up_from 5947 up_thru 5950 down_at 5964 last_clean_interval [208,5643) <$IPADDR>:6813/20771 <$IPADDR>:6814/20771 <$IPADDR>:6815/20771 <$IPADDR>:6816/20771 destroyed,exists 1b6b013e-29ec-462b-bd4f-75030d0bfffc
osd.5 up in weight 1 up_from 5962 up_thru 5964 down_at 5960 last_clean_interval [5645,5959) <$IPADDR>:6817/8004 <$IPADDR>:6818/8004 <$IPADDR>:6819/8004 <$IPADDR>:6820/8004 exists,up c0d25d14-4f99-4554-b811-e8373d8b032b
root@hostname:~/sdfsdf# ceph osd destroy osd.1 --yes-i-really-mean-it
destroyed osd.1
root@hostname:~/sdfsdf# ceph osd dump
epoch 5965
fsid a2ff4e1f-54c0-476a-8891-45c305b1a2e9
created 2016-08-08 01:23:30.220664
modified 2017-08-17 09:32:42.061199
flags noout,noscrub,nodeep-scrub,sortbitwise,recovery_deletes
crush_version 3
full_ratio 0.95
backfillfull_ratio 0.9
nearfull_ratio 0.85
require_min_compat_client jewel
min_compat_client jewel
require_osd_release luminous
pool 4 'ec' erasure size 5 min_size 3 crush_rule 1 object_hash rjenkins pg_num 160 pgp_num 160 last_change 96 flags hashpspool stripe_width 4128
pool 5 'rbd' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 160 pgp_num 160 last_change 121 flags hashpspool stripe_width 0
removed_snaps [1~3]
pool 8 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 160 pgp_num 160 last_change 215 flags hashpspool stripe_width 0
pool 9 'cephfs_data' erasure size 5 min_size 4 crush_rule 1 object_hash rjenkins pg_num 160 pgp_num 160 last_change 216 flags hashpspool stripe_width 12288
max_osd 6
osd.1 down in weight 1 up_from 5659 up_thru 5952 down_at 5960 last_clean_interval [4927,5643) <$IPADDR>:6805/6161 <$IPADDR>:6806/6161 <$IPADDR>:6807/6161 <$IPADDR>:6808/6161 destroyed,exists d706a467-bb1b-40ce-a07d-fd8fdc0ea427
osd.2 down in weight 1 up_from 5648 up_thru 5958 down_at 5960 last_clean_interval [5005,5643) <$IPADDR>:6801/6108 <$IPADDR>:6802/6108 <$IPADDR>:6803/6108 <$IPADDR>:6804/6108 destroyed,exists e3031736-acb4-4a5e-a67f-f9b171167cbd
osd.3 down in weight 1 up_from 5653 up_thru 5954 down_at 5960 last_clean_interval [5468,5643) <$IPADDR>:6809/6211 <$IPADDR>:6810/6211 <$IPADDR>:6811/6211 <$IPADDR>:6812/6211 destroyed,exists 4288d76d-2e14-4557-8d54-216319eb3581
osd.4 down in weight 1 up_from 5947 up_thru 5950 down_at 5964 last_clean_interval [208,5643) <$IPADDR>:6813/20771 <$IPADDR>:6814/20771 <$IPADDR>:6815/20771 <$IPADDR>:6816/20771 destroyed,exists 1b6b013e-29ec-462b-bd4f-75030d0bfffc
osd.5 up in weight 1 up_from 5962 up_thru 5964 down_at 5960 last_clean_interval [5645,5959) <$IPADDR>:6817/8004 <$IPADDR>:6818/8004 <$IPADDR>:6819/8004 <$IPADDR>:6820/8004 exists,up c0d25d14-4f99-4554-b811-e8373d8b032b
D. Following the logic of OSDMonitor::prepare_command_osd_new -- https://github.com/ceph/ceph/blob/v12.1.4/src/mon/OSDMonitor.cc#L6714 -- it seems the following conditions has to be met to arrive at the section which issues the "recreate osd" code, at https://github.com/ceph/ceph/blob/v12.1.4/src/mon/OSDMonitor.cc#L6714
* is_recreate_destroyed = true (btw, L6811 could use the bool rather than perform the check again)
* may_be_idempotent = false which only can happen iff OSDMonitor:validate_osd_create returns >= 0 && !EEXIST. And THAT only happens if either UUID either isn't supplied (which it has to be) or if the UUID doesn't match the OLD uuid. Which it has to.
4. Trying the command "ceph osd new i ceph-new-secrets.json $(uuidgen -r) 1" ran through the code and I received a new OSD "osd.1" with state "exists,new". Thus entering this into the inplace-replace-osd procedure seems to work as expected! I now just need to supply the new OSDs UUID (/var/lib/ceph/osd/$cluster$id/fsid), a simple command misunderstanding in the end. The documentation (man/command) didn't clarify which UUID is supplied to 'osd new', but the patch above does make that a fair bit more clear.
5. I still need to complete the patch for tools/ceph_objectstore_tool.cc to fix my cluster (clear the deleted flags on the previous OSDs which now refuse to boot). Be right back on that.