Ceph

Getting errors when deploying OSD.
Need help to understand what is wrong and any solution can be done.

There are 3 servers with same specs installed.
For one reason, only able to deploy the osd on one of the servers, but failed to deploy osds on another 2 servers.

See below output for "ceph orch device ls --wide"

ssd and hdd from tncpceph01 and tncpceph02 are Available, but failed to deployed.

ssd and hdd from tncpceph03 are successfully deployed.

root@tncpceph01:~# ceph orch device ls --wide
HOST PATH TYPE TRANSPORT RPM DEVICE ID SIZE HEALTH IDENT FAULT AVAILABLE REFRESHED REJECT REASONS
tncpceph01 /dev/nvme0n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03546 1600G Good N/A N/A Yes 16m ago
tncpceph01 /dev/nvme1n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03428 1600G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdb hdd SAS 15000 ST900MP0026_5000c500f0190e73 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdc hdd SAS 15000 ST900MP0026_5000c500f018728b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdd hdd SAS 15000 ST900MP0026_5000c500f019151f 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sde hdd SAS 15000 ST900MP0026_5000c500f019106b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdf hdd SAS 15000 ST900MP0026_5000c500f018f93b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdg hdd SAS 15000 ST900MP0026_5000c500f0193257 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdh hdd SAS 15000 ST900MP0026_5000c500f018e0a3 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdi hdd SAS 15000 ST900MP0026_5000c500f019106f 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdj hdd SAS 15000 ST900MP0026_5000c500f01b839f 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdk hdd SAS 15000 ST900MP0026_5000c500f0190a47 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdl hdd SAS 15000 ST900MP0026_5000c500f018beaf 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdm hdd SAS 15000 ST900MP0026_5000c500f018cad7 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdn hdd SAS 15000 ST900MP0026_5000c500f01b952f 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdo hdd SAS 15000 ST900MP0026_5000c500f018b8df 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdp hdd SAS 15000 ST900MP0026_5000c500f018af5b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdq hdd SAS 15000 ST900MP0026_5000c500f0198937 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdr hdd SAS 15000 ST900MP0026_5000c500f01926cb 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sds hdd SAS 15000 ST900MP0026_5000c500f019823b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdt hdd SAS 15000 ST900MP0026_5000c500f018dee7 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdu hdd SAS 15000 ST900MP0026_5000c500f01910e7 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdv hdd SAS 15000 ST900MP0026_5000c500f0189fdb 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdw hdd SAS 15000 ST900MP0026_5000c500f0191f3f 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdx hdd SAS 15000 ST900MP0026_5000c500f019091b 900G Good N/A N/A Yes 16m ago
tncpceph01 /dev/sdy hdd SAS 15000 ST900MP0026_5000c500f018e337 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/nvme0n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03290 1600G Good N/A N/A Yes 16m ago
tncpceph02 /dev/nvme1n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03547 1600G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdb hdd SAS 15000 ST900MP0026_5000c500f00df96f 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdc hdd SAS 15000 ST900MP0026_5000c500f00eb7af 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdd hdd SAS 15000 ST900MP0026_5000c500f00e0347 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sde hdd SAS 15000 ST900MP0026_5000c500f00dfbc3 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdf hdd SAS 15000 ST900MP0026_5000c500f00dec1b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdg hdd SAS 15000 ST900MP0026_5000c500f00db1b3 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdh hdd SAS 15000 ST900MP0026_5000c500f00e021f 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdi hdd SAS 15000 ST900MP0026_5000c500f00e00b3 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdj hdd SAS 15000 ST900MP0026_5000c500f00e024f 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdk hdd SAS 15000 ST900MP0026_5000c500f00eb7d3 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdl hdd SAS 15000 ST900MP0026_5000c500f00dfb6b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdm hdd SAS 15000 ST900MP0026_5000c500f00dfe2b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdn hdd SAS 15000 ST900MP0026_5000c500f00dfb1b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdo hdd SAS 15000 ST900MP0026_5000c500f00e042b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdp hdd SAS 15000 ST900MP0026_5000c500f00de303 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdq hdd SAS 15000 ST900MP0026_5000c500f00eb7bb 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdr hdd SAS 15000 ST900MP0026_5000c500f00dfbc7 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sds hdd SAS 15000 ST900MP0026_5000c500f00ddc8f 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdt hdd SAS 15000 ST900MP0026_5000c500f00dfbbf 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdu hdd SAS 15000 ST900MP0026_5000c500f00de80b 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdv hdd SAS 15000 ST900MP0026_5000c500f00db56f 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdw hdd SAS 15000 ST900MP0026_5000c500f00db8ef 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdx hdd SAS 15000 ST900MP0026_5000c500f00df9fb 900G Good N/A N/A Yes 16m ago
tncpceph02 /dev/sdy hdd SAS 15000 ST900MP0026_5000c500f00de4bb 900G Good N/A N/A Yes 16m ago
tncpceph03 /dev/nvme0n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03289 1600G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/nvme1n1 ssd Unknown Unknown Dell_Ent_NVMe_AGN_MU_AIC_1.6TB_S61ANA0RB03371 1600G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdb hdd SAS 15000 ST900MP0026_5000c500f018b2b3 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdc hdd SAS 15000 ST900MP0026_5000c500f0ac00af 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdd hdd SAS 15000 ST900MP0026_5000c500f0187d8f 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sde hdd SAS 15000 ST900MP0026_5000c500f018e8ef 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdf hdd SAS 15000 ST900MP0026_5000c500f018eab7 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdg hdd SAS 15000 ST900MP0026_5000c500f018a497 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdh hdd SAS 15000 ST900MP0026_5000c500f00eb813 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdi hdd SAS 15000 ST900MP0026_5000c500f019116b 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdj hdd SAS 15000 ST900MP0026_5000c500f0070a7b 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdk hdd SAS 15000 ST900MP0026_5000c500f006cb3f 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdl hdd SAS 15000 ST900MP0026_5000c500f00eb81b 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdm hdd SAS 15000 ST900MP0026_5000c500f018aa67 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdn hdd SAS 15000 ST900MP0026_5000c500f006cbcb 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdo hdd SAS 15000 ST900MP0026_5000c500f006efbf 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdp hdd SAS 15000 ST900MP0026_5000c500f006f0bf 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdq hdd SAS 15000 ST900MP0026_5000c500f006cb07 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdr hdd SAS 15000 ST900MP0026_5000c500f005118f 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sds hdd SAS 15000 ST900MP0026_5000c500f005adb7 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdt hdd SAS 15000 ST900MP0026_5000c500f006ca63 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdu hdd SAS 15000 ST900MP0026_5000c500f00eb803 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdv hdd SAS 15000 ST900MP0026_5000c500f006ccd3 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdw hdd SAS 15000 ST900MP0026_5000c500f005141f 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdx hdd SAS 15000 ST900MP0026_5000c500f006cbeb 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
tncpceph03 /dev/sdy hdd SAS 15000 ST900MP0026_5000c500f00715c3 900G Good N/A N/A No 8m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
root@tncpceph01:~#

From the logs, I was seeing this:

Sep 27, 2023, 12:12:30 PM [ERR] Cannot decode JSON: ''
Traceback (most recent call last):
File "/usr/share/ceph/mgr/cephadm/services/osd.py", line 314, in generate_previews
concat_out: Dict[str, Any] = json.loads(' '.join(out))
File "/lib64/python3.6/json/__init__.py", line 354, in loads
return _default_decoder.decode(s)
File "/lib64/python3.6/json/decoder.py", line 339, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/lib64/python3.6/json/decoder.py", line 357, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

root@tncpceph01:~# ceph -v
ceph version 16.2.14 (238ba602515df21ea7ffc75c88db29f9e5ef12c9) pacific (stable)

root@tncpceph01:~# ceph orch ps
NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID
alertmanager.tncpceph01 tncpceph01 *:9093,9094 running (80m) 9m ago 20h 22.2M - 0.23.0 ba2b418f427c afd9503136d1
ceph-exporter.tncpceph01 tncpceph01 running (18h) 9m ago 20h 26.5M - 17.2.6 46541e70e3d2 ed51e3752403
ceph-exporter.tncpceph02 tncpceph02 running (93m) 9m ago 93m 9952k - 17.2.6 46541e70e3d2 c7a78a0a4db0
ceph-exporter.tncpceph03 tncpceph03 running (91m) 40s ago 91m 25.2M - 17.2.6 46541e70e3d2 36739f41c832
crash.tncpceph01 tncpceph01 running (18h) 9m ago 20h 14.5M - 17.2.6 46541e70e3d2 3fe174212d3b
crash.tncpceph02 tncpceph02 running (93m) 9m ago 93m 14.4M - 17.2.6 46541e70e3d2 811ad8275e3b
crash.tncpceph03 tncpceph03 running (91m) 40s ago 91m 12.7M - 17.2.6 46541e70e3d2 b29cdc3ba4c4
grafana.tncpceph01 tncpceph01 *:3000 running (18h) 9m ago 20h 139M - 8.3.5 dad864ee21e9 5f5cffe39b79
mgr.tncpceph01.fzozii tncpceph01 *:9283 running (18h) 9m ago 20h 698M - 17.2.6 46541e70e3d2 0d4a2650acad
mgr.tncpceph02.kldzof tncpceph02 *:8443,9283 running (93m) 9m ago 93m 435M - 17.2.6 46541e70e3d2 7e88c454b602
mgr.tncpceph03.lcztky tncpceph03 *:8443,9283 running (80m) 40s ago 80m 432M - 17.2.6 46541e70e3d2 5e1da97a2c01
mon.tncpceph01 tncpceph01 running (18h) 9m ago 20h 443M 2048M 17.2.6 46541e70e3d2 9dc099bc17a6
mon.tncpceph02 tncpceph02 running (93m) 9m ago 93m 148M 2048M 17.2.6 46541e70e3d2 b90f940415ae
mon.tncpceph03 tncpceph03 running (91m) 40s ago 91m 160M 2048M 17.2.6 46541e70e3d2 15e60a877944
node-exporter.tncpceph01 tncpceph01 *:9100 running (18h) 9m ago 20h 54.4M - 1.3.1 1dbe0e931976 51db7a85c3de
node-exporter.tncpceph02 tncpceph02 *:9100 running (93m) 9m ago 93m 35.0M - 1.3.1 1dbe0e931976 c616599f5877
node-exporter.tncpceph03 tncpceph03 *:9100 running (91m) 40s ago 91m 36.7M - 1.3.1 1dbe0e931976 56757c1d686a
osd.0 tncpceph03 running (71m) 40s ago 71m 83.1M 7203M 17.2.6 46541e70e3d2 105afeb830b4
osd.1 tncpceph03 running (71m) 40s ago 71m 86.0M 7203M 17.2.6 46541e70e3d2 3e36ace49f29
osd.2 tncpceph03 running (59m) 40s ago 61m 87.2M 7203M 17.2.6 46541e70e3d2 4d5816a26587
osd.3 tncpceph03 running (52m) 40s ago 52m 69.4M 7203M 17.2.6 46541e70e3d2 4482fae47e78
osd.4 tncpceph03 running (42m) 40s ago 52m 66.0M 7203M 17.2.6 46541e70e3d2 b6c89e954abb
osd.5 tncpceph03 running (47m) 40s ago 52m 70.3M 7203M 17.2.6 46541e70e3d2 9679c9b15db9
osd.6 tncpceph03 running (47m) 40s ago 49m 71.0M 7203M 17.2.6 46541e70e3d2 7827fcee4ed1
osd.7 tncpceph03 running (40m) 40s ago 46m 70.7M 7203M 17.2.6 46541e70e3d2 8900eb81e660
osd.8 tncpceph03 running (42m) 40s ago 44m 67.8M 7203M 17.2.6 46541e70e3d2 b0728779b513
osd.9 tncpceph03 running (37m) 40s ago 42m 71.8M 7203M 17.2.6 46541e70e3d2 7690bad80447
osd.10 tncpceph03 running (71m) 40s ago 71m 83.9M 7203M 17.2.6 46541e70e3d2 744a536a29e5
osd.11 tncpceph03 running (68m) 40s ago 71m 80.4M 7203M 17.2.6 46541e70e3d2 1901693da13e
osd.12 tncpceph03 running (68m) 40s ago 68m 77.8M 7203M 17.2.6 46541e70e3d2 dc566a79b7e2
osd.13 tncpceph03 running (68m) 40s ago 68m 78.3M 7203M 17.2.6 46541e70e3d2 c5c7d81b6f83
osd.14 tncpceph03 running (68m) 40s ago 68m 77.2M 7203M 17.2.6 46541e70e3d2 70cf1abe37cd
osd.15 tncpceph03 running (63m) 40s ago 68m 77.1M 7203M 17.2.6 46541e70e3d2 3696852291e7
osd.16 tncpceph03 running (61m) 40s ago 68m 85.4M 7203M 17.2.6 46541e70e3d2 6d21d35e11c4
osd.17 tncpceph03 running (65m) 40s ago 68m 75.4M 7203M 17.2.6 46541e70e3d2 24a5763ebc29
osd.18 tncpceph03 running (63m) 40s ago 65m 73.6M 7203M 17.2.6 46541e70e3d2 75c7f4e595d3
osd.19 tncpceph03 running (59m) 40s ago 63m 71.9M 7203M 17.2.6 46541e70e3d2 73b0c51f54d9
osd.20 tncpceph03 running (57m) 40s ago 59m 73.1M 7203M 17.2.6 46541e70e3d2 48b2d49fab96
osd.21 tncpceph03 running (47m) 40s ago 59m 67.3M 7203M 17.2.6 46541e70e3d2 e7540d83f550
osd.22 tncpceph03 running (52m) 40s ago 56m 71.8M 7203M 17.2.6 46541e70e3d2 f259d9a3a3ea
osd.23 tncpceph03 running (52m) 40s ago 54m 70.9M 7203M 17.2.6 46541e70e3d2 36d170a32c05
prometheus.tncpceph01 tncpceph01 *:9095 running (80m) 9m ago 20h 145M - 2.33.4 514e6a882f6e 666e7d092e90
root@tncpceph01:~#