Ceph » ceph-volume

I have 3 OSD nodes with 8 HDDs Hitachi (9.1 TiB), 4 NVMes Micron_9300 (2.9 TiB), and 2 NVMes Intel Optane P4800X (375 GiB). I want to use spinning disks for the data block, 2.9 NVMes for the block.DB and the intel Optane for the block.wal.

I would expect 1 lv on every data disk, 4 lv on wal disks, and 2 lv on DB disks. The problem arises on DB disks where only 1 lv gets created.

After some debugging, I think that the problem is generated when the VG gets divided into 2. I have 763089 Total PE and the first LV was created using 381545 PE (round-up for 763089/2 ?). Thanks to that, the creation of the second LV fails: "Volume group "ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f" has insufficient free space (381544 extents): 381545 required."

I tried with a spec file and also via the ceph dashboard, both with and without a number of slots defined.

[
  {
    "service_type": "osd",
    "service_id": "dashboard-admin-1615816083969",
    "host_pattern": "*",
    "data_devices": {
      "vendor": "HGST" 
    },
    "wal_devices": {
      "model": "INTEL SSDPED1K375GA" 
    },
    "db_devices": {
      "model": "Micron_9300_MTFDHAL3T2TDR" 
    },
    "db_slots": 2,
    "wal_slots": 4
  }
]

first lv:

[2021-03-15 13:52:47,645][ceph_volume.process][INFO  ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG
-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/nvme3n1
[2021-03-15 13:52:47,648][ceph_volume.process][INFO  ] stdout NAME="nvme3n1" KNAME="nvme3n1" MAJ:MIN="259:6" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="Micron_9300_MTFDHAL3T2TDR               
" SIZE="2.9T" STATE="live" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="none" TYPE="disk" DISC-ALN="0" DISC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME
="" PARTLABEL="" 
[2021-03-15 13:52:47,648][ceph_volume.process][INFO  ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_cou
nt,vg_extent_size /dev/nvme3n1
[2021-03-15 13:52:47,723][ceph_volume.process][INFO  ] stderr Failed to find physical volume "/dev/nvme3n1".
[2021-03-15 13:52:47,724][ceph_volume.process][INFO  ] Running command: /usr/sbin/vgcreate --force --yes ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f /dev/nvme3n1
[2021-03-15 13:52:47,751][ceph_volume.process][INFO  ] stdout Physical volume "/dev/nvme3n1" successfully created.
[2021-03-15 13:52:47,753][ceph_volume.process][INFO  ] stdout Volume group "ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f" successfully created
[2021-03-15 13:52:47,775][ceph_volume.process][INFO  ] Running command: /usr/sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -S vg_name=ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f -o vg_name,
pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size
[2021-03-15 13:52:47,853][ceph_volume.process][INFO  ] stdout ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f";"1";"0";"wz--n-";"763089";"763089";"4194304
[2021-03-15 13:52:47,854][ceph_volume.api.lvm][DEBUG ] size was passed: 1.46 TB -> 381545
[2021-03-15 13:52:47,854][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvcreate --yes -l 381545 -n osd-db-2192de32-0d9b-4b49-82fb-237a5a7ccab7 ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f
[2021-03-15 13:52:47,917][ceph_volume.process][INFO  ] stdout Logical volume "osd-db-2192de32-0d9b-4b49-82fb-237a5a7ccab7" created.

second LV:

[2021-03-15 13:52:53,963][ceph_volume.process][INFO  ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE,ALIGNMENT,PHY-SEC,LOG
-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/nvme3n1
[2021-03-15 13:52:53,965][ceph_volume.process][INFO  ] stdout NAME="nvme3n1" KNAME="nvme3n1" MAJ:MIN="259:6" FSTYPE="LVM2_member" MOUNTPOINT="" LABEL="" UUID="08x6yC-nrp4-8GYz-yTgj-w2U0-StLk-Vieqhp" RO="0" RM="0
" MODEL="Micron_9300_MTFDHAL3T2TDR               " SIZE="2.9T" STATE="live" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="0" SCHED="none" TYPE="disk" DISC-ALN="0" DI
SC-GRAN="512B" DISC-MAX="2T" DISC-ZERO="0" PKNAME="" PARTLABEL="" 
[2021-03-15 13:52:53,966][ceph_volume.process][INFO  ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_cou
nt,vg_extent_size /dev/nvme3n1
[2021-03-15 13:52:54,030][ceph_volume.process][INFO  ] stdout ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f";"1";"1";"wz--n-";"763089";"381544";"4194304
[2021-03-15 13:52:54,030][ceph_volume.api.lvm][DEBUG ] size was passed: 1.46 TB -> 381545
[2021-03-15 13:52:54,031][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvcreate --yes -l 381545 -n osd-db-a147911c-2e21-4bb0-a8f9-e493a53e9070 ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f
[2021-03-15 13:52:54,077][ceph_volume.process][INFO  ] stderr Volume group "ceph-d1b2ae26-bad3-4790-90fd-5d5e7d04dc8f" has insufficient free space (381544 extents): 381545 required.
[2021-03-15 13:52:54,094][ceph_volume.devices.lvm.prepare][ERROR ] lvm prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 382, in prepare
    self.args.block_db_slots)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 181, in setup_device
    **kwargs)
  File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 949, in create_lv
    process.run(command)
  File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", line 153, in run
    raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: 5
[2021-03-15 13:52:54,095][ceph_volume.devices.lvm.prepare][INFO  ] will rollback OSD ID creation
[2021-03-15 13:52:54,095][ceph_volume.process][INFO  ] Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd purge-new osd.1 --yes-i-really-mean-it
[2021-03-15 13:52:54,362][ceph_volume.process][INFO  ] stderr purged osd.1
[2021-03-15 13:52:54,369][ceph_volume][ERROR ] exception caught by decorator
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 59, in newfunc
    return f(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/main.py", line 152, in main
    terminal.dispatch(self.mapper, subcommand_args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/main.py", line 42, in main
    terminal.dispatch(self.mapper, self.argv)
  File "/usr/lib/python3.6/site-packages/ceph_volume/terminal.py", line 194, in dispatch
    instance.main()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 415, in main
    self._execute(plan)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/batch.py", line 434, in _execute
    c.create(argparse.Namespace(**args))
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/create.py", line 26, in create
    prepare_step.safe_prepare(args)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 252, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 382, in prepare
    self.args.block_db_slots)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 181, in setup_device
    **kwargs)
  File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 949, in create_lv
    process.run(command)
  File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", line 153, in run
    raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: 5