Project

General

Profile

Bug #14094

ceph-disk list: OSError: [Errno 2] No such file or directory: '/dev/vdb1'

Added by Loic Dachary over 3 years ago. Updated over 3 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
12/16/2015
Due date:
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

After a failed run of the ceph-disk suite on CentOS 7.1, ceph-disk list tries to stat /dev/vdb1 although it does not exist. If run again immediately afterwards, the problem goes away. It is not a timing problem, nothing has been going one for over 15 minutes after the test stopped.

It showed twice but I'm not sure how to reproduce it reliably.

[root@target167114225077 ceph-disk]# ceph-disk list
Traceback (most recent call last):
  File "/sbin/ceph-disk", line 4054, in <module>
    main(sys.argv[1:])
  File "/sbin/ceph-disk", line 4010, in main
    main_catch(args.func, args)
  File "/sbin/ceph-disk", line 4032, in main_catch
    func(args)
  File "/sbin/ceph-disk", line 3324, in main_list
    devices = list_devices(args.path)
  File "/sbin/ceph-disk", line 3274, in list_devices
    ptype = get_partition_type(dev)
  File "/sbin/ceph-disk", line 3077, in get_partition_type
    return get_sgdisk_partition_info(part, 'Partition GUID code: (\S+)')
  File "/sbin/ceph-disk", line 3083, in get_sgdisk_partition_info
    (base, partnum) = split_dev_base_partnum(dev)
  File "/sbin/ceph-disk", line 3067, in split_dev_base_partnum
    if is_mpath(dev):
  File "/sbin/ceph-disk", line 460, in is_mpath
    uuid = get_dm_uuid(dev)
  File "/sbin/ceph-disk", line 448, in get_dm_uuid
    uuid_path = os.path.join(block_path(dev), 'dm', 'uuid')
  File "/sbin/ceph-disk", line 443, in block_path
    rdev = os.stat(path).st_rdev
OSError: [Errno 2] No such file or directory: '/dev/vdb1'
[root@target167114225077 ceph-disk]# ceph-disk list
/dev/vda :
 /dev/vda1 other, xfs, mounted on /
/dev/vdb other, unknown
/dev/vdc :
 /dev/vdc1 ceph journal
/dev/vdd other, unknown

Related issues

Related to Ceph - Bug #14080: ceph-disk: use blkid instead of sgdisk -i Resolved 12/14/2015

Associated revisions

Revision 9dce05a8 (diff)
Added by Loic Dachary over 3 years ago

ceph-disk: use blkid instead of sgdisk -i

sgdisk -i 1 /dev/vdb opens /dev/vdb in write mode which indirectly
triggers a BLKRRPART ioctl from udev (starting version 214 and up) when
the device is closed (see below for the udev release note). The
implementation of this ioctl by the kernel (even old kernels) removes
all partitions and adds them again (similar to what partprobe does
explicitly).

The side effects of partitions disappearing while ceph-disk is running
are devastating.

sgdisk is replaced by blkid which only opens the device in read mode and
will not trigger this unexpected behavior.

The problem does not show on Ubuntu 14.04 because it is running udev <
214 but shows on CentOS 7 which is running udev > 214.

git clone git://anonscm.debian.org/pkg-systemd/systemd.git
systemd/NEWS:
CHANGES WITH 214:

  • As an experimental feature, udev now tries to lock the
    disk device node (flock(LOCK_SH|LOCK_NB)) while it
    executes events for the disk or any of its partitions.
    Applications like partitioning programs can lock the
    disk device node (flock(LOCK_EX)) and claim temporary
    device ownership that way; udev will entirely skip all event
    handling for this disk and its partitions. If the disk
    was opened for writing, the close will trigger a partition
    table rescan in udev's "watch" facility, and if needed
    synthesize "change" events for the disk and all its partitions.
    This is now unconditionally enabled, and if it turns out to
    cause major problems, we might turn it on only for specific
    devices, or might need to disable it entirely. Device Mapper
    devices are excluded from this logic.

http://tracker.ceph.com/issues/14094 Fixes: #14094

Signed-off-by: Ilya Dryomov <>
Signed-off-by: Loic Dachary <>

Revision 88ffcc2c (diff)
Added by Loic Dachary about 3 years ago

ceph-disk: use blkid instead of sgdisk -i

sgdisk -i 1 /dev/vdb opens /dev/vdb in write mode which indirectly
triggers a BLKRRPART ioctl from udev (starting version 214 and up) when
the device is closed (see below for the udev release note). The
implementation of this ioctl by the kernel (even old kernels) removes
all partitions and adds them again (similar to what partprobe does
explicitly).

The side effects of partitions disappearing while ceph-disk is running
are devastating.

sgdisk is replaced by blkid which only opens the device in read mode and
will not trigger this unexpected behavior.

The problem does not show on Ubuntu 14.04 because it is running udev <
214 but shows on CentOS 7 which is running udev > 214.

git clone git://anonscm.debian.org/pkg-systemd/systemd.git
systemd/NEWS:
CHANGES WITH 214:

  • As an experimental feature, udev now tries to lock the
    disk device node (flock(LOCK_SH|LOCK_NB)) while it
    executes events for the disk or any of its partitions.
    Applications like partitioning programs can lock the
    disk device node (flock(LOCK_EX)) and claim temporary
    device ownership that way; udev will entirely skip all event
    handling for this disk and its partitions. If the disk
    was opened for writing, the close will trigger a partition
    table rescan in udev's "watch" facility, and if needed
    synthesize "change" events for the disk and all its partitions.
    This is now unconditionally enabled, and if it turns out to
    cause major problems, we might turn it on only for specific
    devices, or might need to disable it entirely. Device Mapper
    devices are excluded from this logic.

http://tracker.ceph.com/issues/14094 Fixes: #14094

Signed-off-by: Ilya Dryomov <>
Signed-off-by: Loic Dachary <>
(cherry picked from commit 9dce05a8cdfc564c5162885bbb67a04ad7b95c5a)

Conflicts:
src/ceph-disk: keep get_partition_type as it is because
some hammer users may rely on the fact that it is able
to fallback to sgdisk if blkid is old. Chances are an
old blkid also means an old udev that does not have the
problem this fix is adressing. The get_partition_uuid
is modified to try blkid first, with the same rationale.

History

#1 Updated by Loic Dachary over 3 years ago

  • Description updated (diff)

#2 Updated by Loic Dachary over 3 years ago

            part_uuid = get_partition_uuid(dev)
            if part_uuid:
                uuid_map[part_uuid] = dev
            ptype = get_partition_type(dev)

get_partition_uuid(dev) succeeds, meaning /dev/vdb1 exists and right after sgdisk -i 1 /dev/vdb it not longer exists. That suggests sgdisk -i 1 /dev/vdb has a side effect that immediately impacts the next call.

#3 Updated by Loic Dachary over 3 years ago

Or just that a partprobe is ongoing and /dev/sdb1 is re-created while ceph-disk list. Maybe protecting ceph-disk list with a udevadm settle is all there is to do.

#4 Updated by Loic Dachary over 3 years ago

2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        set -xe
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        ceph-disk --verbose deactivate --deactivate-by-id 3
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:        ceph-disk --verbose destroy --destroy-by-id 3 --zap
2015-12-17 11:54:49,077.077 INFO:tasks.workunit.client.0.target167114233028.stderr:
2015-12-17 11:54:49,081.081 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:+ ceph-disk --verbose deactivate --deactivate-by-id 3
2015-12-17 11:54:49,174.174 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/udevadm settle --timeout=600
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vda
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda uuid path is /sys/dev/block/253:0/dm/uuid
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdb
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:54:49,183.183 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdc
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc uuid path is /sys/dev/block/253:32/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdd
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd uuid path is /sys/dev/block/253:48/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,184.184 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,189.189 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,190.190 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,201.201 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vda1 ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4 uuid = cfebdfb5-563f-4823-b419-a697480fc98d
2015-12-17 11:54:49,202.202 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,202.202 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,211.211 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,211.211 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,224.224 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdc1 ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid = 9ad0073e-a4b4-11e5-b02e-fa163e8241b5
2015-12-17 11:54:49,224.224 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:54:49,227.227 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt.aY5whQ with options
2015-12-17 11:54:49,228.228 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,236.236 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,241.241 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,241.241 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.aY5whQ
2015-12-17 11:54:49,248.248 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,249.249 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,262.262 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,263.263 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,273.273 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd1 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 8c61499d-457f-4a29-ae1c-be361940bfcf
2015-12-17 11:54:49,274.274 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,274.274 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,286.286 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,286.286 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,299.299 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd2 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 08d91923-0d1a-4be2-a715-cea4c9068d3f
2015-12-17 11:54:49,299.299 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: {'vda': ['vda1'], 'vdc': ['vdc1'], 'vdb': [], 'vdd': ['vdd1', 'vdd2']}, uuid_map = {'cfebdfb5-563f-4823-b419-a697480fc98d': '/dev/vda1', '08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdd2', '8c61499d-457f-4a29-ae1c-be361940bfcf': '/dev/vdd1', '9ad0073e-a4b4-11e5-b02e-fa163e8241b5': '/dev/vdc1'}, journal_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdc1'}
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,300.300 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:54:49,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:54:49,316.316 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vda1, ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4)
2015-12-17 11:54:49,316.316 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vda1
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdb, ptype = unknown)
2015-12-17 11:54:49,321.321 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdb
2015-12-17 11:54:49,679.679 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,679.679 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,680.680 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:54:49,692.692 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:54:49,698.698 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdc1, ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d)
2015-12-17 11:54:49,700.700 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:54:49,706.706 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,727.727 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,738.738 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:54:49,739.739 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:54:49,746.746 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd1, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:54:49,746.746 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,747.747 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,747.747 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,754.754 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:54:49,755.755 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:54:49,767.767 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd2, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_devices: [{'path': '/dev/vda', 'partitions': [{'dmcrypt': {}, 'uuid': '09113c14-0fd3-45d5-bd25-99b5305b1063', 'mount': '/', 'ptype': '0fc63daf-8483-4772-8e79-3d69d8477de4', 'is_partition': True, 'fs_type': 'xfs', 'path': '/dev/vda1', 'type': 'other'}]}, {'path': '/dev/vdb', 'type': 'other', 'dmcrypt': {}, 'ptype': 'unknown', 'is_partition': False}, {'path': '/dev/vdc', 'partitions': [{'dmcrypt': {}, 'uuid': '9ad0073e-a4b4-11e5-b02e-fa163e8241b5', 'mount': '/var/lib/ceph/osd/ceph-3', 'ptype': '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', 'is_partition': True, 'cluster': 'ceph', 'state': 'active', 'fs_type': 'xfs', 'ceph_fsid': '7ca6c79d-e401-4b68-ad0f-f3b44f181b1e', 'path': '/dev/vdc1', 'type': 'data', 'whoami': '3', 'journal_dev': '/dev/vdd2', 'journal_uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f'}]}, {'path': '/dev/vdd', 'partitions': [{'dmcrypt': {}, 'uuid': '8c61499d-457f-4a29-ae1c-be361940bfcf', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'path': '/dev/vdd1', 'type': 'journal'}, {'dmcrypt': {}, 'uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'journal_for': '/dev/vdc1', 'path': '/dev/vdd2', 'type': 'journal'}]}]
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Checking osd id: 3 ...
2015-12-17 11:54:49,768.768 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph osd dump --cluster=ceph --format json
2015-12-17 11:54:50,070.070 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Stoping ceph osd.3...
2015-12-17 11:54:50,071.071 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/systemctl disable ceph-osd@3
2015-12-17 11:54:50,083.083 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:Removed symlink /etc/systemd/system/ceph-osd.target.wants/ceph-osd@3.service.
2015-12-17 11:54:50,169.169 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/systemctl stop ceph-osd@3
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `ready` file.
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `active` file.
2015-12-17 11:54:53,238.238 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup init
2015-12-17 11:54:53,250.250 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-detect-init --default sysvinit
2015-12-17 11:54:53,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Remove `systemd` file.
2015-12-17 11:54:53,312.312 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon -R /var/lib/ceph/osd/ceph-3/deactive
2015-12-17 11:54:53,324.324 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3/deactive
2015-12-17 11:54:53,327.327 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/osd/ceph-3
2015-12-17 11:54:53,328.328 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/osd/ceph-3
2015-12-17 11:54:53,353.353 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Umount `/var/lib/ceph/osd/ceph-3` successfully.
2015-12-17 11:54:53,361.361 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:+ ceph-disk --verbose destroy --destroy-by-id 3 --zap
2015-12-17 11:54:53,443.443 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/udevadm settle --timeout=600
2015-12-17 11:55:00,547.547 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vda
2015-12-17 11:55:00,547.547 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda uuid path is /sys/dev/block/253:0/dm/uuid
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdb
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdc
2015-12-17 11:55:00,550.550 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc uuid path is /sys/dev/block/253:32/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_all_partitions: vdd
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd uuid path is /sys/dev/block/253:48/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:00,551.551 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:00,564.564 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:00,564.564 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:00,572.572 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vda1 ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4 uuid = c61fa287-0238-4f44-a28c-3babec76072e
2015-12-17 11:55:00,573.573 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:00,573.573 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:00,588.588 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:00,588.588 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:00,958.958 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdc1 ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid = 9ad0073e-a4b4-11e5-b02e-fa163e8241b5
2015-12-17 11:55:00,958.958 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:55:04,832.832 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt.4aAduf with options
2015-12-17 11:55:04,832.832 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,265.265 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,333.333 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,334.334 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.4aAduf
2015-12-17 11:55:08,351.351 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:08,351.351 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:08,376.376 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:08,376.376 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:08,665.665 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd1 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 8c61499d-457f-4a29-ae1c-be361940bfcf
2015-12-17 11:55:08,665.665 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:08,666.666 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:08,674.674 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:08,676.676 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: /dev/vdd2 ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106 uuid = 08d91923-0d1a-4be2-a715-cea4c9068d3f
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:main_list: {'vda': ['vda1'], 'vdc': ['vdc1'], 'vdb': [], 'vdd': ['vdd1', 'vdd2']}, uuid_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdd2', 'c61fa287-0238-4f44-a28c-3babec76072e': '/dev/vda1', '8c61499d-457f-4a29-ae1c-be361940bfcf': '/dev/vdd1', '9ad0073e-a4b4-11e5-b02e-fa163e8241b5': '/dev/vdc1'}, journal_map = {'08d91923-0d1a-4be2-a715-cea4c9068d3f': '/dev/vdc1'}
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,689.689 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:08,698.698 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vda1 uuid path is /sys/dev/block/253:1/dm/uuid
2015-12-17 11:55:08,699.699 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vda
2015-12-17 11:55:08,710.710 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vda1, ptype = 0fc63daf-8483-4772-8e79-3d69d8477de4)
2015-12-17 11:55:08,710.710 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vda1
2015-12-17 11:55:08,714.714 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdb uuid path is /sys/dev/block/253:16/dm/uuid
2015-12-17 11:55:08,714.714 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdb, ptype = unknown)
2015-12-17 11:55:08,715.715 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdb
2015-12-17 11:55:09,936.936 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,937.937 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,937.937 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:09,949.949 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdc1 uuid path is /sys/dev/block/253:33/dm/uuid
2015-12-17 11:55:09,950.950 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdc
2015-12-17 11:55:09,963.963 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdc1, ptype = 4fbd7e29-9d25-41b8-afd0-062c0ceff05d)
2015-12-17 11:55:09,964.964 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/blkid -s TYPE /dev/vdc1
2015-12-17 11:55:12,833.833 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Mounting /dev/vdc1 on /var/lib/ceph/tmp/mnt._3QP_m with options
2015-12-17 11:55:12,833.833 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/mount -t xfs -o  -- /dev/vdc1 /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,846.846 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,864.864 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
2015-12-17 11:55:16,894.894 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:Unmounting /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,895.895 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /bin/umount -- /var/lib/ceph/tmp/mnt._3QP_m
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,915.915 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:16,941.941 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd1 uuid path is /sys/dev/block/253:49/dm/uuid
2015-12-17 11:55:16,941.941 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 1 /dev/vdd
2015-12-17 11:55:17,030.030 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd1, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:55:17,030.030 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,031.031 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,031.031 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:17,039.039 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:get_dm_uuid /dev/vdd2 uuid path is /sys/dev/block/253:50/dm/uuid
2015-12-17 11:55:17,040.040 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:Running command: /usr/sbin/sgdisk -i 2 /dev/vdd
2015-12-17 11:55:17,047.047 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:INFO:ceph-disk:list_dev(dev = /dev/vdd2, ptype = 45b0969e-9b03-4f30-b4c6-b4b80ceff106)
2015-12-17 11:55:17,047.047 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:DEBUG:ceph-disk:list_devices: [{'path': '/dev/vda', 'partitions': [{'dmcrypt': {}, 'uuid': '62088de7-53a9-4dc7-93b1-32baf094c096', 'mount': '/', 'ptype': '0fc63daf-8483-4772-8e79-3d69d8477de4', 'is_partition': True, 'fs_type': 'xfs', 'path': '/dev/vda1', 'type': 'other'}]}, {'path': '/dev/vdb', 'type': 'other', 'dmcrypt': {}, 'ptype': 'unknown', 'is_partition': False}, {'path': '/dev/vdc', 'partitions': [{'magic': 'ceph osd volume v026', 'dmcrypt': {}, 'uuid': '9ad0073e-a4b4-11e5-b02e-fa163e8241b5', 'mount': None, 'ptype': '4fbd7e29-9d25-41b8-afd0-062c0ceff05d', 'is_partition': True, 'cluster': 'ceph', 'state': 'prepared', 'fs_type': 'xfs', 'ceph_fsid': '7ca6c79d-e401-4b68-ad0f-f3b44f181b1e', 'path': '/dev/vdc1', 'type': 'data', 'whoami': '3', 'journal_dev': '/dev/vdd2', 'journal_uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f'}]}, {'path': '/dev/vdd', 'partitions': [{'dmcrypt': {}, 'uuid': '8c61499d-457f-4a29-ae1c-be361940bfcf', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'path': '/dev/vdd1', 'type': 'journal'}, {'dmcrypt': {}, 'uuid': '08d91923-0d1a-4be2-a715-cea4c9068d3f', 'ptype': '45b0969e-9b03-4f30-b4c6-b4b80ceff106', 'is_partition': True, 'journal_for': '/dev/vdc1', 'path': '/dev/vdd2', 'type': 'journal'}]}]
2015-12-17 11:55:17,129.129 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:Traceback (most recent call last):
2015-12-17 11:55:17,130.130 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 4057, in <module>
2015-12-17 11:55:17,131.131 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:main(sys.argv[1:])
2015-12-17 11:55:17,131.131 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 4011, in main
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:args.func(args)
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 2841, in main_destroy
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:base_dev = get_partition_base(dev_path)
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 611, in get_partition_base
2015-12-17 11:55:17,132.132 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:if not stat.S_ISBLK(os.lstat(dev).st_mode):
2015-12-17 11:55:17,133.133 INFO:tasks.workunit.client.0.target167114233028.stderr:DEBUG:CephDisk:OSError: [Errno 2] No such file or directory: '/dev/vdc1'
2015-12-17 11:55:17,158.158 INFO:tasks.workunit.client.0.target167114233028.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py::TestCephDisk::test_activate_two_separated_journal FAILED

#5 Updated by Loic Dachary over 3 years ago

  • Related to Bug #14080: ceph-disk: use blkid instead of sgdisk -i added

#6 Updated by Loic Dachary over 3 years ago

It turns out that sgdisk 0.8.6 -i 2 /dev/vdb removes partitions and re-adds them on CentOS 7 with a 3.10.0-229.11.1.el7 kernel, in the same way partprobe does. It is used intensively by ceph-disk and inevitably leads to races where a device temporarily disapears. The same command (sgdisk 0.8.8) on Ubuntu 14.04 with a 3.13.0-62-generic kernel only generates two udev change events and does not remove / add partitions. The source code between sgdisk 0.8.6 and sgdisk 0.8.8 did not change in a significant way and the output of strace -e ioctl sgdisk -i 2 /dev/vdb is identical in both environments.

ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, HDIO_GETGEO, {heads=16, sectors=63, cylinders=16383, start=0}) = 0
ioctl(3, HDIO_GETGEO, {heads=16, sectors=63, cylinders=16383, start=0}) = 0
ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKGETSIZE, 20971520)          = 0
ioctl(3, BLKGETSIZE64, 10737418240)     = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0
ioctl(3, BLKSSZGET, 512)                = 0

This leads me to the conclusion that the difference is in how the kernel reacts to these ioctl.

#7 Updated by Loic Dachary over 3 years ago

  • Backport set to infernalis

#8 Updated by Loic Dachary over 3 years ago

  • Status changed from Verified to Duplicate
  • Backport deleted (infernalis)

#9 Updated by Loic Dachary over 3 years ago

  • Related to deleted (Bug #14080: ceph-disk: use blkid instead of sgdisk -i)

#10 Updated by Loic Dachary over 3 years ago

  • Related to Bug #14080: ceph-disk: use blkid instead of sgdisk -i added

#11 Updated by Loic Dachary over 3 years ago

The disappearance of /dev/vdb1 is a side effect of sgdisk -i indirectly triggering the equivalent of partprobe (which removes devices).

Also available in: Atom PDF