[ajdev0@osd1 by-partuuid]$ ls -l total 0 lrwxrwxrwx 1 root root 10 Oct 28 12:14 a4125f1d-e119-42c9-8c90-2c4ff6fcc59b -> ../../sde1 lrwxrwxrwx 1 root root 10 Oct 28 12:14 ba74e920-9575-4014-8fb8-c3a2b3d123dd -> ../../sdc1 lrwxrwxrwx 1 root root 10 Oct 28 12:14 eea2acd9-811d-4548-9151-e8ffdda503da -> ../../sdd1 [ajdev0@osd1 ~]$ sudo systemctl list-units |grep ceph sys-devices-platform-host6-session1-target6:0:0-6:0:0:0-block-sdc-sdc1.device loaded active plugged iSCSI_Disk ceph\x20data sys-devices-platform-host7-session2-target7:0:0-7:0:0:0-block-sdd-sdd1.device loaded active plugged iSCSI_Disk ceph\x20data sys-devices-platform-host8-session3-target8:0:0-8:0:0:0-block-sde-sde1.device loaded active plugged iSCSI_Disk ceph\x20data ceph-disk@dev-sdc1.service loaded failed failed Ceph disk activation: /dev/sdc1 ceph-disk@dev-sdd1.service loaded failed failed Ceph disk activation: /dev/sdd1 ceph-disk@dev-sde1.service loaded failed failed Ceph disk activation: /dev/sde1 system-ceph\x2ddisk.slice loaded active active system-ceph\x2ddisk.slice ceph-mds.target loaded active active ceph target allowing to start/stop all ceph-mds@.service instances at once ceph-mgr.target loaded active active ceph target allowing to start/stop all ceph-mgr@.service instances at once ceph-mon.target loaded active active ceph target allowing to start/stop all ceph-mon@.service instances at once ceph-osd.target loaded active active ceph target allowing to start/stop all ceph-osd@.service instances at once ceph.target loaded active active ceph target allowing to start/stop all ceph*@.service instances at once [ajdev0@osd1 ~]$ sudo ceph-disk list /dev/sda : /dev/sda1 other, ext3, mounted on / /dev/sda2 other, 0x0 /dev/sda3 other, swap /dev/sda4 other, 0x0 /dev/sdb other, unknown /dev/sdc : /dev/sdc1 ceph data, prepared, cluster ceph, osd.0 /dev/sdd : /dev/sdd1 ceph data, prepared, cluster ceph, osd.2 /dev/sde : /dev/sde1 ceph data, prepared, cluster ceph, osd.1 [ajdev0@osd1 ~]$ sudo systemctl status ceph* ceph-disk@dev-sdd1.service - Ceph disk activation: /dev/sdd1 Loaded: loaded (/usr/lib/systemd/system/ceph-disk@.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2016-10-28 12:14:32 MDT; 2min 52s ago Process: 67679 ExecStart=/bin/sh -c timeout 120 flock /var/lock/ceph-disk /usr/sbin/ceph-disk --verbose --log-stdout trigger --sync %f (code=exited, status=1/FAILURE) Main PID: 67679 (code=exited, status=1/FAILURE) Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: main(sys.argv[1:]) Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5093... main Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: args.func(args) Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4530...igger Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: raise Error('return code ' + str(ret)) Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net sh[67679]: ceph_disk.main.Error: Error: return code 1 Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sdd1.service: main process exited, code=exited, statu...ILURE Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Failed to start Ceph disk activation: /dev/sdd1. Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Unit ceph-disk@dev-sdd1.service entered failed state. Oct 28 12:14:32 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sdd1.service failed. ceph-disk@dev-sdc1.service - Ceph disk activation: /dev/sdc1 Loaded: loaded (/usr/lib/systemd/system/ceph-disk@.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2016-10-28 12:14:22 MDT; 3min 1s ago Process: 66844 ExecStart=/bin/sh -c timeout 120 flock /var/lock/ceph-disk /usr/sbin/ceph-disk --verbose --log-stdout trigger --sync %f (code=exited, status=1/FAILURE) Main PID: 66844 (code=exited, status=1/FAILURE) Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: main(sys.argv[1:]) Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5093... main Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: args.func(args) Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4530...igger Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: raise Error('return code ' + str(ret)) Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net sh[66844]: ceph_disk.main.Error: Error: return code 1 Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sdc1.service: main process exited, code=exited, statu...ILURE Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Failed to start Ceph disk activation: /dev/sdc1. Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Unit ceph-disk@dev-sdc1.service entered failed state. Oct 28 12:14:22 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sdc1.service failed. ceph-disk@dev-sde1.service - Ceph disk activation: /dev/sde1 Loaded: loaded (/usr/lib/systemd/system/ceph-disk@.service; static; vendor preset: disabled) Active: failed (Result: exit-code) since Fri 2016-10-28 12:14:27 MDT; 2min 57s ago Process: 67254 ExecStart=/bin/sh -c timeout 120 flock /var/lock/ceph-disk /usr/sbin/ceph-disk --verbose --log-stdout trigger --sync %f (code=exited, status=1/FAILURE) Main PID: 67254 (code=exited, status=1/FAILURE) Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: main(sys.argv[1:]) Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5093... main Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: args.func(args) Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4530...igger Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: raise Error('return code ' + str(ret)) Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net sh[67254]: ceph_disk.main.Error: Error: return code 1 Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sde1.service: main process exited, code=exited, statu...ILURE Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Failed to start Ceph disk activation: /dev/sde1. Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: Unit ceph-disk@dev-sde1.service entered failed state. Oct 28 12:14:27 osd1.finalexp.idms-pg0.apt.emulab.net systemd[1]: ceph-disk@dev-sde1.service failed. Hint: Some lines were ellipsized, use -l to show in full.