Project

General

Profile

Bug #47758

fail to create OSDs because the requested extent is too large

Added by Kiefer Chang 19 days ago. Updated 13 days ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Community (dev)
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

This was observed in this e2e test failure: https://tracker.ceph.com/issues/47742

Some context about this e2d test:
- We need additional disks to test cephadm OSD creation feature. Because there are no sparse disks on smithi node:
- 3 sparse files are created and exported as iSCSI target luns (LIO).
- Login the target, 3 disks appear on host.
- Dashboard then ask cephadm to create 3 new OSDs with these disks.

[2020-10-06 06:55:10,428][ceph_volume.main][INFO  ] Running command: ceph-volume  lvm batch --no-auto /dev/sdf --yes --no-systemd
...

[2020-10-06 06:55:10,958][ceph_volume.process][INFO  ] Running command: /usr/bin/lsblk --nodeps -P -o NAME,KNAME,MAJ:MIN,FSTYPE,MOUNTPOINT,LABEL,UUID,RO,RM,MODEL,SIZE,STATE,OWNER,GROUP,MODE
,ALIGNMENT,PHY-SEC,LOG-SEC,ROTA,SCHED,TYPE,DISC-ALN,DISC-GRAN,DISC-MAX,DISC-ZERO,PKNAME,PARTLABEL /dev/sdf
[2020-10-06 06:55:10,961][ceph_volume.process][INFO  ] stdout NAME="sdf" KNAME="sdf" MAJ:MIN="8:80" FSTYPE="" MOUNTPOINT="" LABEL="" UUID="" RO="0" RM="0" MODEL="lun1            " SIZE="15G
" STATE="running" OWNER="root" GROUP="disk" MODE="brw-rw----" ALIGNMENT="0" PHY-SEC="512" LOG-SEC="512" ROTA="1" SCHED="mq-deadline" TYPE="disk" DISC-ALN="0" DISC-GRAN="0B" DISC-MAX="0B" DI
SC-ZERO="0" PKNAME="" PARTLABEL="" 
[2020-10-06 06:55:10,961][ceph_volume.devices.lvm.prepare][DEBUG ] data device size: 15.00 GB
[2020-10-06 06:55:10,961][ceph_volume.process][INFO  ] Running command: /usr/sbin/pvs --noheadings --readonly --units=b --nosuffix --separator=";" -o vg_name,pv_count,lv_count,vg_attr,vg_ex
tent_count,vg_free_count,vg_extent_size /dev/sdf
[2020-10-06 06:55:10,973][ceph_volume.process][INFO  ] stderr Failed to find physical volume "/dev/sdf".
[2020-10-06 06:55:10,974][ceph_volume.process][INFO  ] Running command: /usr/sbin/vgcreate --force --yes ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7 /dev/sdf
[2020-10-06 06:55:10,986][ceph_volume.process][INFO  ] stdout Physical volume "/dev/sdf" successfully created.
[2020-10-06 06:55:10,999][ceph_volume.process][INFO  ] stdout Volume group "ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7" successfully created
[2020-10-06 06:55:11,007][ceph_volume.process][INFO  ] Running command: /usr/sbin/vgs --noheadings --readonly --units=b --nosuffix --separator=";" -S vg_name=ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7 -o vg_name,pv_count,lv_count,vg_attr,vg_extent_count,vg_free_count,vg_extent_size
[2020-10-06 06:55:11,026][ceph_volume.process][INFO  ] stdout ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7";"1";"0";"wz--n-";"3838";"3838";"4194304
[2020-10-06 06:55:11,026][ceph_volume.api.lvm][DEBUG ] size was passed: 15.00 GB -> 3839
[2020-10-06 06:55:11,027][ceph_volume.process][INFO  ] Running command: /usr/sbin/lvcreate --yes -l 3839 -n osd-block-21867cd0-f8ad-4f5e-ad41-18510386c0c5 ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7
[2020-10-06 06:55:11,062][ceph_volume.process][INFO  ] stderr Volume group "ceph-755d5f48-33f7-4bc2-b909-d8e2bea211d7" has insufficient free space (3838 extents): 3839 required.
[2020-10-06 06:55:11,066][ceph_volume.devices.lvm.prepare][ERROR ] lvm prepare was unable to complete
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 250, in safe_prepare
    self.prepare()
  File "/usr/lib/python3.6/site-packages/ceph_volume/decorators.py", line 16, in is_root
    return func(*a, **kw)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 361, in prepare
    block_lv = self.prepare_data_device('block', osd_fsid)
  File "/usr/lib/python3.6/site-packages/ceph_volume/devices/lvm/prepare.py", line 219, in prepare_data_device
    **kwargs)
  File "/usr/lib/python3.6/site-packages/ceph_volume/api/lvm.py", line 949, in create_lv
    process.run(command)
  File "/usr/lib/python3.6/site-packages/ceph_volume/process.py", line 153, in run
    raise RuntimeError(msg)
RuntimeError: command returned non-zero exit status: 5

The lvcreate command asks for one more extent (3838 vs 3839 in this example).


Related issues

Related to mgr - Bug #47742: cephadm/test_dashboard_e2e.sh: OSDs are not created Resolved

History

#1 Updated by Kiefer Chang 18 days ago

  • Related to Bug #47742: cephadm/test_dashboard_e2e.sh: OSDs are not created added

#2 Updated by Kiefer Chang 18 days ago

The cause to this problem is the underlying block device reports a larger optimal I/O size (8M) than default extent size (4M), which makes pvcreate align the VG fisrt offset to 8M.

ceph-volume calculates VG's extent count by assuming space after LVM metadata is all usable. Thus it requests one more extent than what the PV can allocate.

 
[root@mgr0 ~]# lsblk -b /dev/sde /dev/sdf -o NAME,VENDOR,SIZE,MIN-IO,OPT-IO
NAME VENDOR          SIZE MIN-IO  OPT-IO
sde  ATA      16106127360    512       0         <--- qemu 15 GB disk
sdf  LIO-ORG  16106127360    512 8388608         <--- iSCSI 15 GB disk

        Optimal I/O Size
+----          8MB        ----+
                                                              16106127360 bytes = 15360 MB
+-------------+---------------+---------------------------------------------+
|LVM metadata |   reserved    |              VG                             |
|             |               |                                             |
+-------------+---------------+---------------------------------------------+

     4 MB            4 MB           (15360 MB - 8MB)/4MB = 3838 extents

ceph-volume calculates extents: (15360-4)/4 = 3839

Not sure if there are ordinary disks report larger optimal I/O size, I can change the reported size to workaround #47742.

#3 Updated by Jan Fajerski 17 days ago

hmm ok interesting. I'd propose we work around that for now in the test. In the real world we're unlikely to encounter this scenario.

Lets keep this issue open, since the free size calculation could certainly be a bit smarter.

Sound good?

#4 Updated by Kiefer Chang 13 days ago

Jan Fajerski wrote:

hmm ok interesting. I'd propose we work around that for now in the test. In the real world we're unlikely to encounter this scenario.

Lets keep this issue open, since the free size calculation could certainly be a bit smarter.

Sound good?

Yeah, a PR was created to workaround the test: https://github.com/ceph/ceph/pull/37575

Also available in: Atom PDF