Bug #6955
Updated by Loïc Dachary about 9 years ago
If an existing ceph data partition is re-used, the partition GUID is not reset when ceph-disk prepares it. As a consequence it will not be identified as a ceph data partition by the ceph udev rules and the OSD won't start when the machine boots.
Steps to reproduce:
<pre>
loic@fold:~/software/ceph/ceph/src$ sudo sgdisk --typecode=1:99999999-9d25-41b8-afd0-062c0ceff05d /dev/loop2
Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.
loic@fold:~/software/ceph/ceph/src$ sudo partprobe /dev/loop2
loic@fold:~/software/ceph/ceph/src$ sudo sgdisk --info 1 /dev/loop2
Partition GUID code: 99999999-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 66D87896-493A-48B4-A803-6ED6DE5D49AB
First sector: 2048 (at 1024.0 KiB)
Last sector: 819166 (at 400.0 MiB)
Partition size: 817119 sectors (399.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
loic@fold:~/software/ceph/ceph/src$ #sudo env PATH=$PATH ceph-disk --verbose prepare /dev/loop2 /tmp/journal
loic@fold:~/software/ceph/ceph/src$ touch /tmp/journal
loic@fold:~/software/ceph/ceph/src$ sudo env PATH=$PATH ceph-disk --verbose prepare /dev/loop2p1 /tmp/journal
INFO:ceph-disk:Running command: ceph-osd --cluster=ceph --show-config-value=fsid
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_type
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_type
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
INFO:ceph-disk:Running command: ceph-osd --cluster=ceph --show-config-value=osd_journal_size
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_cryptsetup_parameters
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_key_size
INFO:ceph-disk:Running command: ceph-conf --cluster=ceph --name=osd. --lookup osd_dmcrypt_type
DEBUG:ceph-disk:Journal is file /tmp/journal
WARNING:ceph-disk:OSD will not be hot-swappable if journal is not the same device as the osd data
DEBUG:ceph-disk:OSD data device /dev/loop2p1 is a partition
DEBUG:ceph-disk:Creating xfs fs on /dev/loop2p1
INFO:ceph-disk:Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/loop2p1
meta-data=/dev/loop2p1 isize=2048 agcount=4, agsize=25535 blks
= sectsz=512 attr=2, projid32bit=0
data = bsize=4096 blocks=102139, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0
log =internal log bsize=4096 blocks=1232, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
DEBUG:ceph-disk:Mounting /dev/loop2p1 on /var/lib/ceph/tmp/mnt.IUm2Xm with options noatime,inode64
INFO:ceph-disk:Running command: mount -t xfs -o noatime,inode64 -- /dev/loop2p1 /var/lib/ceph/tmp/mnt.IUm2Xm
DEBUG:ceph-disk:Preparing osd data dir /var/lib/ceph/tmp/mnt.IUm2Xm
DEBUG:ceph-disk:Creating symlink /var/lib/ceph/tmp/mnt.IUm2Xm/journal -> /tmp/journal
DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.IUm2Xm
INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.IUm2Xm
umount: /var/lib/ceph/tmp/mnt.IUm2Xm: device is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.IUm2Xm
INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.IUm2Xm
DEBUG:ceph-disk:Calling partprobe on prepared device /dev/loop2p1
INFO:ceph-disk:Running command: /sbin/partprobe /dev/loop2p1
Error: Partition(s) 1 on /dev/loop2p1 have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes.
loic@fold:~/software/ceph/ceph/src$ sudo sgdisk --info 1 /dev/loop2
Partition GUID code: 99999999-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 66D87896-493A-48B4-A803-6ED6DE5D49AB
First sector: 2048 (at 1024.0 KiB)
Last sector: 819166 (at 400.0 MiB)
Partition size: 817119 sectors (399.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
loic@fold:~/software/ceph/ceph/src$ sudo partprobe /dev/loop2
loic@fold:~/software/ceph/ceph/src$ sudo sgdisk --info 1 /dev/loop2
Partition GUID code: 99999999-9D25-41B8-AFD0-062C0CEFF05D (Unknown)
Partition unique GUID: 66D87896-493A-48B4-A803-6ED6DE5D49AB
First sector: 2048 (at 1024.0 KiB)
Last sector: 819166 (at 400.0 MiB)
Partition size: 817119 sectors (399.0 MiB)
Attribute flags: 0000000000000000
Partition name: 'ceph data'
</pre>
See mailing list thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-December/006650.html