Ceph

Hello,

we are building a new cluster and want to use ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432) on Red Hat Enterprise Linux Server release 7.2 (Maipo).
We are not able to prepare disks using dmcrypt:

Using empty disk /dev/sdh:

# lsblk -i /dev/sdh
NAME MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sdh    8:192  0  3.7T  0 disk

Preparing disk:

# ceph-disk -v prepare --zap-disk --dmcrypt /dev/sdh 
DEBUG:ceph-disk:Zapping partition table on /dev/sdh
INFO:ceph-disk:Running command: /sbin/sgdisk --zap-all -- /dev/sdh
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.

Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
INFO:ceph-disk:Running command: /sbin/sgdisk --clear --mbrtogpt -- /dev/sdh
Creating new GPT entries.
The operation has completed successfully.
INFO:ceph-disk:calling partx on zapped device /dev/sdh
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /sbin/partx -d /dev/sdh
partx: specified range <1:0> does not make sense
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_type
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
INFO:ceph-disk:Will colocate journal with data on /dev/sdh
DEBUG:ceph-disk:Creating journal partition num 2 size 4096 on /dev/sdh
INFO:ceph-disk:Running command: /sbin/sgdisk --new=2:0:4096M --change-name=2:ceph journal --partition-guid=2:434aaf0e-d20c-4224-b69b-35315da09ffa --typecode=2:89c57f98-2fe5-4dc0-89c1-35865ceff2be --mbrtogpt -- /dev/sdh
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/sdh
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdh
partx: /dev/sdh: error adding partition 2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/434aaf0e-d20c-4224-b69b-35315da09ffa
INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/434aaf0e-d20c-4224-b69b-35315da09ffa.luks.key luksFormat /dev/disk/by-partuuid/434aaf0e-d20c-4224-b69b-35315da09ffa
INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=2:45b0969e-9b03-4f30-b4c6-35865ceff106 -- /dev/sdh
The operation has completed successfully.
DEBUG:ceph-disk:Journal is GPT partition /dev/mapper/434aaf0e-d20c-4224-b69b-35315da09ffa
DEBUG:ceph-disk:Creating osd partition on /dev/sdh
INFO:ceph-disk:Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:2fd3d833-efde-47f9-a9f8-163be3664a67 --typecode=1:89c57f98-2fe5-4dc0-89c1-5ec00ceff2be -- /dev/sdh
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
INFO:ceph-disk:calling partx on created device /dev/sdh
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdh
partx: /dev/sdh: error adding partition 2
INFO:ceph-disk:Running command: /usr/bin/udevadm settle
INFO:ceph-disk:Running command: /sbin/cryptsetup --batch-mode --key-file /etc/ceph/dmcrypt-keys/2fd3d833-efde-47f9-a9f8-163be3664a67.luks.key luksFormat /dev/sdh1
INFO:ceph-disk:Running command: /sbin/cryptsetup --key-file /etc/ceph/dmcrypt-keys/2fd3d833-efde-47f9-a9f8-163be3664a67.luks.key luksOpen /dev/sdh1 2fd3d833-efde-47f9-a9f8-163be3664a67
DEBUG:ceph-disk:Creating xfs fs on /dev/mapper/2fd3d833-efde-47f9-a9f8-163be3664a67
INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/mapper/2fd3d833-efde-47f9-a9f8-163be3664a67
meta-data=/dev/mapper/2fd3d833-efde-47f9-a9f8-163be3664a67 isize=2048   agcount=4, agsize=243926325 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=0        finobt=0
data     =                       bsize=4096   blocks=975705297, imaxpct=5
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0
log      =internal log           bsize=4096   blocks=476418, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
DEBUG:ceph-disk:Mounting /dev/mapper/2fd3d833-efde-47f9-a9f8-163be3664a67 on /var/lib/ceph/tmp/mnt.Gbkvoo with options noatime,inode64
INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/mapper/2fd3d833-efde-47f9-a9f8-163be3664a67 /var/lib/ceph/tmp/mnt.Gbkvoo
DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.Gbkvoo
DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.Gbkvoo/journal -> /dev/mapper/434aaf0e-d20c-4224-b69b-35315da09ffa
DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.Gbkvoo/journal_dmcrypt -> /dev/disk/by-partuuid/434aaf0e-d20c-4224-b69b-35315da09ffa
DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.Gbkvoo
INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.Gbkvoo
INFO:ceph-disk:Running command: /sbin/cryptsetup remove 2fd3d833-efde-47f9-a9f8-163be3664a67
INFO:ceph-disk:Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-35865ceff05d -- /dev/sdh
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
INFO:ceph-disk:calling partx on prepared device /dev/sdh
INFO:ceph-disk:re-reading known partitions will display errors
INFO:ceph-disk:Running command: /sbin/partx -a /dev/sdh
partx: /dev/sdh: error adding partitions 1-2

Status of disk after preparation:

# lsblk -i /dev/sdh
NAME                                     MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sdh                                        8:112  0  3.7T  0 disk  
|-sdh1                                     8:113  0  3.7T  0 part  
`-sdh2                                     8:114  0    4G  0 part  
  `-434aaf0e-d20c-4224-b69b-35315da09ffa 253:3    0    4G  0 crypt

dmcrypt setup of /dev/sdh1 is missing.
Further information:
- Preparing a disk without dmcrypt happens to work, but the osd is not automatically started (as I would expect, and is the behavior on Debian).
- We don't have any problems on another cluster using Debian Wheezy 7.9 with ceph version 0.94.7 (d56bdf93ced6b80b07397d57e3fa68fe68304432). Preparing disks with dmcrypt works.

Kind regards,
Martin Traxl