Ceph

Hi,

Ceph version 17.2.5, freshly installed with cephadm

All OSDs are fully installed and functional but the MONs are still spamming with this INF log : Refuse to add /dev/sdq due to limit policy of <4>

HEALTH_OK with osd: 720 osds: 720 up (since 46h), 720 in (since 46h)

From MON logs :

Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: from='mgr.33843 134.158.210.31:0/2078358291' entity='mgr.cccephadm40.ubrzws'
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: from='mgr.33843 134.158.210.31:0/2078358291' entity='mgr.cccephadm40.ubrzws' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader) e3 refresh_from_paxos
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos version 137024 summary v 137023
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos latest full 136986
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos applying incremental log 137024 2022-12-01T09:28:06.209023+0000 mgr.cccephadm40.ubrzws (mgr.33843) 56440 : cephadm [INF] Refuse to add /dev/sdq due to limit policy of <4>
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: get_journald
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: Refuse to add /dev/sdq due to limit policy of <4>
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 journald: cephadm
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos applying incremental log 137024 2022-12-01T09:28:06.213416+0000 mgr.cccephadm40.ubrzws (mgr.33843) 56441 : cephadm [INF] Refuse to add /dev/sdq due to limit policy of <4>
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: get_journald
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: Refuse to add /dev/sdq due to limit policy of <4>
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 journald: cephadm
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: mon.cccephadm40@0(leader).log v137024 update_from_paxos applying incremental log 137024 2022-12-01T09:28:06.218008+0000 mgr.cccephadm40.ubrzws (mgr.33843) 56442 : cephadm [INF] Refuse to add /dev/sdq due to limit policy of <4>
Dec 01 10:28:08 cccephadm40 ceph-mon[153350]: get_journald

The OSD conf I used :

# use all HDDs (16) as data_devices with 4 SSDs assigned as dedicated db/wal devices. 
service_type: osd
service_id: osd_spec_hdd
placement:
  host_pattern: '*'
spec:
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0
    limit: 4
---
# use the 2 remaining SSDs each split in 4 OSDs per device
service_type: osd
service_id: osd_spec_ssd
placement:
  host_pattern: '*'
spec:
  data_devices:
    rotational: 0
  osds_per_device: 4

Everything looks good, I have the correct number of osd_hdd and osd_ssd. I tried to restart MONs/MGRs but the message "Refuse to add..." came back.

ceph orch ls
NAME              PORTS        RUNNING  REFRESHED  AGE  PLACEMENT           
osd.osd_spec_hdd                   480  10m ago    17h  *            
osd.osd_spec_ssd                   240  10m ago    17h  *