Ceph

Hello,

I am using EC2 and EBS to test a Quincy cluster. The EC2 OSD instances have a 10G secondary disk attached to serve as backend storage. Once the hosts are added using cephadm, CEPH detects the device, but is rejecting it since the disk has "Insufficient Space (<10 extents)".

I am using CloudFormation to create the cluster, using cephadm on the monitor to first add hosts and then use --all-available-devices option to add disks on the hosts. CEPH was installed using packages available from the CEPH SIG.

HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS
ip-10-42-2-42.eu-north-1.compute.internal /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol0e3ea31df8c3f76e7 10.7G No 28m ago Insufficient space (<10 extents) on vgs, LVM detected, locked
ip-10-42-2-216.eu-north-1.compute.internal /dev/nvme1n1 ssd Amazon_Elastic_Block_Store_vol08a2126ca9e34f994 10.7G No 9m ago Insufficient space (<10 extents) on vgs, LVM detected, locked

Looking at the PVDISPLAY output on the node, I see that there are more than 10 extents in that physical volume.

[centos@ip-10-42-2-42 ~]$ sudo pvdisplay --maps
--- Physical volume ---
PV Name /dev/nvme1n1
VG Name ceph-74a2f628-fd78-4b25-8be6-48066f428ba9
PV Size 10.00 GiB / not usable 4.00 MiB
Allocatable yes (but full)
PE Size 4.00 MiB
Total PE 2559
Free PE 0
Allocated PE 2559
PV UUID 9p8UQA-z4no-OYm8-1T1c-apU4-mdsW-mzdgEc

--- Physical Segments ---
  Physical extent 0 to 2558:
    Logical volume      /dev/ceph-74a2f628-fd78-4b25-8be6-48066f428ba9/osd-block-b74d5217-3c97-4b4c-8486-18ca3aea4db5
    Logical extents     0 to 2558