Actions
Bug #20555
closedceph-disk fails to prepare an osd device if both --zap-disk and --dmcrypt are used:
Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Tags:
ceph-disk encryption dmcrypt zap sgdisk
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-disk
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
ceph-disk fails to prepare an osd device if both --zap-disk and --dmcrypt are used.
I have tested on J but not on K or L yet.
This looks the same as http://tracker.ceph.com/issues/16451, however, the suggestion there was to create a ceph-disk bug instead of a ceph-deploy bug. ceph-deploy uses (https://github.com/ceph/ceph-deploy/blob/v1.5.32/ceph_deploy/osd.py#L196-L226) ceph-disk underneath, therefore, it is natural that the same issue occurs with it as well.
Also related:
http://tracker.ceph.com/issues/16347
Steps to reproduce:
- clear a block device from any previous partition tables and file system signatures via wipefs
- run sudo ceph-disk prepare --dmcrypt --fs-type xfs --zap-disk <osd-device> <journal-device>
I have used a qemu/kvm VM with multiple virtio-scsi devices to reproduce that.
uname -r
4.4.0-83-generic
cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.2 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.2 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial
$ dpkg -l 'ceph*' | grep ii
ii ceph 10.2.7-0ubuntu0.16.04.1 amd64 distributed storage and file system
ii ceph-common 10.2.7-0ubuntu0.16.04.1 amd64 common utilities to mount and interact with a ceph storage cluster
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 64G 0 disk
└─sda1 8:1 0 64G 0 part /
sdb 8:16 0 64G 0 disk
└─sdb3 8:19 0 10M 0 part /var/lib/ceph/osd-lockbox/821cda07-6b4a-4aa3-b889-428be041ab5f
sdc 8:32 0 64G 0 disk
sdd 8:48 0 64G 0 disk
$ sudo umount /dev/sdb3
# wipe whatever was left from the previous attempt
$ sudo wipefs -a /dev/sdb
/dev/sdb: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 8 bytes were erased at offset 0xffffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdb: calling ioctl to re-read partition table: Success
# try to prepare with --zap-disk
$ sudo ceph-disk prepare --dmcrypt --fs-type xfs --zap-disk /dev/sdb /dev/sdd
Creating new GPT entries.
Setting name!
partNum is 2
REALLY setting name!
The operation has completed successfully.
mke2fs 1.42.13 (17-May-2015)
/dev/sdb3 contains a ext4 file system
last mounted on /var/lib/ceph/osd-lockbox/821cda07-6b4a-4aa3-b889-428be041ab5f on Sun Jul 9 20:31:18 2017
Proceed anyway? (y,n) y
Discarding device blocks: done
Creating filesystem with 10240 1k blocks and 2560 inodes
Filesystem UUID: 0e24df47-fdae-4749-843d-b0078e1f5a38
Superblock backups stored on blocks:
8193
Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done
value stored
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Caution: invalid backup GPT header, but valid main header; regenerating
backup header from main header.
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.
Warning! One or more CRCs don't match. You should repair the disk!
****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
Creating new GPT entries.
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
ceph-disk: Error: partprobe /dev/sdb failed : Error: Partition(s) 3 on /dev/sdb have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
$ sudo umount /dev/sdb3
$ sudo wipefs -a /dev/sdb
/dev/sdb: 8 bytes were erased at offset 0x00000200 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 8 bytes were erased at offset 0xffffffe00 (gpt): 45 46 49 20 50 41 52 54
/dev/sdb: 2 bytes were erased at offset 0x000001fe (PMBR): 55 aa
/dev/sdb: calling ioctl to re-read partition table: Success
# ========================================================
# ========================================================
# now without zapping
# ========================================================
# ========================================================
$ sudo ceph-disk prepare --dmcrypt --fs-type xfs /dev/sdb /dev/sdd
Creating new GPT entries.
Setting name!
partNum is 2
REALLY setting name!
The operation has completed successfully.
mke2fs 1.42.13 (17-May-2015)
/dev/sdb3 contains a ext4 file system
last mounted on /var/lib/ceph/osd-lockbox/62b6f170-894e-4069-8a04-742661b23eac on Sun Jul 9 21:14:08 2017
Proceed anyway? (y,n) y
Discarding device blocks: done
Creating filesystem with 10240 1k blocks and 2560 inodes
Filesystem UUID: a8e494d7-219a-4dc4-b7eb-9dff9a7a376d
Superblock backups stored on blocks:
8193
Allocating group tables: done
Writing inode tables: done
Creating journal (1024 blocks): done
Writing superblocks and filesystem accounting information: done
value stored
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
Creating new GPT entries.
Setting name!
partNum is 0
REALLY setting name!
The operation has completed successfully.
run:
run:
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
Setting name!
partNum is 0
REALLY setting name!
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
run:
run:
meta-data=/dev/mapper/09292b70-c808-4823-b8e9-4398f6cd37b2 isize=2048 agcount=4, agsize=4193471 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0
data = bsize=4096 blocks=16773883, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=8190, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot or after you
run partprobe(8) or kpartx(8)
The operation has completed successfully.
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 64G 0 disk
└─sda1 8:1 0 64G 0 part /
sdb 8:16 0 64G 0 disk
├─sdb1 8:17 0 64G 0 part
│ └─09292b70-c808-4823-b8e9-4398f6cd37b2 252:1 0 64G 0 crypt /var/lib/ceph/osd/ceph-0
└─sdb3 8:19 0 10M 0 part /var/lib/ceph/osd-lockbox/09292b70-c808-4823-b8e9-4398f6cd37b2
sdc 8:32 0 64G 0 disk
sdd 8:48 0 64G 0 disk
└─sdd1 8:49 0 1G 0 part
└─690661f2-dec5-4666-a392-714d2dbdebbd 252:0 0 1022M 0 crypt
Actions