Actions
Bug #19822
closedceph-disk fails to activate disk - Debian Jessie + Kraken
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-disk
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
I successfully used ceph-deploy to the ceph packages, and setup the monitor nodes.
However, adding an osd to the cluster doesn't seem to work:
~ ceph-deploy --username root osd create n01.da.ceph.frickel.tech:sda:nvme0n1p1
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/flokli/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (1.5.37): /home/flokli/dev/flokli/ceph/.venv/bin/ceph-deploy --username root osd create n01.da.ceph.frickel.tech:sda:nvme0n1p1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] username : root
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] subcommand : create
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf object at 0x7f1b1a8690b8>
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] func : <function osd at 0x7f1b1aab2268>
[ceph_deploy.cli][INFO ] zap_disk : False
[ceph_deploy.cli][INFO ] fs_type : xfs
[ceph_deploy.cli][INFO ] dmcrypt : False
[ceph_deploy.cli][INFO ] dmcrypt_key_dir : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO ] bluestore : None
[ceph_deploy.cli][INFO ] disk : [('n01.da.ceph.frickel.tech', '/dev/sda', '/dev/nvme0n1p1')]
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks n01.da.ceph.frickel.tech:/dev/sda:/dev/nvme0n1p1
[n01.da.ceph.frickel.tech][DEBUG ] connected to host: root@n01.da.ceph.frickel.tech
[ceph_deploy.osd][INFO ] Distro info: debian 8.7 jessie
[ceph_deploy.osd][DEBUG ] Deploying osd to n01.da.ceph.frickel.tech
[ceph_deploy.osd][DEBUG ] Preparing host n01.da.ceph.frickel.tech disk /dev/sda journal /dev/nvme0n1p1 activate True
[n01.da.ceph.frickel.tech][INFO ] Running command: /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/sda /dev/nvme0n1p1
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster ceph
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/nvme0n1p1 uuid path is /sys/dev/block/259:4/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] prepare_device: Journal /dev/nvme0n1p1 is a partition
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/nvme0n1p1 uuid path is /sys/dev/block/259:4/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /sbin/blkid -o udev -p /dev/nvme0n1p1
[n01.da.ceph.frickel.tech][WARNIN] prepare_device: Journal /dev/nvme0n1p1 was not prepared with ceph-disk. Symlinking directly.
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] set_data_partition: Creating osd partition on /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] ptype_tobe_for_name: name = data
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] create_partition: Creating data partition num 1 size 0 on /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:33639697-4b71-4c2f-beff-a7b535f73aa9 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be --mbrtogpt -- /dev/sda
[n01.da.ceph.frickel.tech][DEBUG ] Setting name!
[n01.da.ceph.frickel.tech][WARNIN] update_partition: Calling partprobe on created device /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/udevadm settle --timeout=600
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/flock -s /dev/sda /sbin/partprobe /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/udevadm settle --timeout=600
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda1 uuid path is /sys/dev/block/8:1/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] populate_data_path_device: Creating xfs fs on /dev/sda1
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/mkfs -t xfs -f -i size=2048 -- /dev/sda1
[n01.da.ceph.frickel.tech][DEBUG ] partNum is 0
[n01.da.ceph.frickel.tech][DEBUG ] REALLY setting name!
[n01.da.ceph.frickel.tech][DEBUG ] The operation has completed successfully.
[n01.da.ceph.frickel.tech][DEBUG ] meta-data=/dev/sda1 isize=2048 agcount=6, agsize=268435455 blks
[n01.da.ceph.frickel.tech][WARNIN] mount: Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.X0KM7k with options noatime,inode64
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sda1 /var/lib/ceph/tmp/mnt.X0KM7k
[n01.da.ceph.frickel.tech][WARNIN] populate_data_path: Preparing osd data dir /var/lib/ceph/tmp/mnt.X0KM7k
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X0KM7k/ceph_fsid.14528.tmp
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X0KM7k/fsid.14528.tmp
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X0KM7k/magic.14528.tmp
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X0KM7k/journal_uuid.14528.tmp
[n01.da.ceph.frickel.tech][WARNIN] adjust_symlink: Creating symlink /var/lib/ceph/tmp/mnt.X0KM7k/journal -> /dev/nvme0n1p1
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /bin/chown -R ceph:ceph /var/lib/ceph/tmp/mnt.X0KM7k
[n01.da.ceph.frickel.tech][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.X0KM7k
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.X0KM7k
[n01.da.ceph.frickel.tech][WARNIN] get_dm_uuid: get_dm_uuid /dev/sda uuid path is /sys/dev/block/8:0/dm/uuid
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/sgdisk --typecode=1:4fbd7e29-9d25-41b8-afd0-062c0ceff05d -- /dev/sda
[n01.da.ceph.frickel.tech][DEBUG ] = sectsz=4096 attr=2, projid32bit=1
[n01.da.ceph.frickel.tech][DEBUG ] = crc=0 finobt=0
[n01.da.ceph.frickel.tech][DEBUG ] data = bsize=4096 blocks=1465130385, imaxpct=5
[n01.da.ceph.frickel.tech][DEBUG ] = sunit=0 swidth=0 blks
[n01.da.ceph.frickel.tech][DEBUG ] naming =version 2 bsize=4096 ascii-ci=0 ftype=0
[n01.da.ceph.frickel.tech][DEBUG ] log =internal log bsize=4096 blocks=521728, version=2
[n01.da.ceph.frickel.tech][DEBUG ] = sectsz=4096 sunit=1 blks, lazy-count=1
[n01.da.ceph.frickel.tech][DEBUG ] realtime =none extsz=4096 blocks=0, rtextents=0
[n01.da.ceph.frickel.tech][DEBUG ] The operation has completed successfully.
[n01.da.ceph.frickel.tech][WARNIN] update_partition: Calling partprobe on prepared device /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/udevadm settle --timeout=600
[n01.da.ceph.frickel.tech][WARNIN] command: Running command: /usr/bin/flock -s /dev/sda /sbin/partprobe /dev/sda
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/udevadm settle --timeout=600
[n01.da.ceph.frickel.tech][WARNIN] command_check_call: Running command: /sbin/udevadm trigger --action=add --sysname-match sda1
[n01.da.ceph.frickel.tech][INFO ] Running command: systemctl enable ceph.target
[n01.da.ceph.frickel.tech][INFO ] checking OSD status...
[n01.da.ceph.frickel.tech][INFO ] Running command: /usr/bin/ceph --cluster=ceph osd stat --format=json
[ceph_deploy.osd][DEBUG ] Host n01.da.ceph.frickel.tech is now ready for osd use.
This looks as if the command succeeded, however, after logging into the machine, I don't see any osd running. Peeking at the logs shows the following during ceph-deploy run:
May 02 11:02:47 n01.da.ceph.frickel.tech sshd[14516]: Accepted publickey for root from 2a00:xxxx port 37970 ssh2: RSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
May 02 11:02:47 n01.da.ceph.frickel.tech sshd[14516]: pam_unix(sshd:session): session opened for user root by (uid=0)
May 02 11:02:47 n01.da.ceph.frickel.tech sshd[14516]: Received disconnect from 2a00:xxxx: 11: disconnected by user
May 02 11:02:47 n01.da.ceph.frickel.tech sshd[14516]: pam_unix(sshd:session): session closed for user root
May 02 11:02:48 n01.da.ceph.frickel.tech sshd[14520]: Accepted publickey for root from 2a00:xxxx port 37972 ssh2: RSA xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
May 02 11:02:48 n01.da.ceph.frickel.tech sshd[14520]: pam_unix(sshd:session): session opened for user root by (uid=0)
May 02 11:02:48 n01.da.ceph.frickel.tech kernel: sda:
May 02 11:02:49 n01.da.ceph.frickel.tech kernel: sda: sda1
May 02 11:03:01 n01.da.ceph.frickel.tech kernel: XFS (sda1): Mounting V4 Filesystem
May 02 11:03:01 n01.da.ceph.frickel.tech kernel: XFS (sda1): Ending clean mount
May 02 11:03:01 n01.da.ceph.frickel.tech kernel: XFS (sda1): Unmounting Filesystem
May 02 11:03:01 n01.da.ceph.frickel.tech kernel: sda: sda1
May 02 11:03:02 n01.da.ceph.frickel.tech systemd[1]: Starting Ceph disk activation: /dev/sda1...
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sda1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7fa87f7ffcf8>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /sbin/init --version
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main_trigger: trigger /dev/sda1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech kernel: XFS (sda1): Mounting V4 Filesystem
May 02 11:03:02 n01.da.ceph.frickel.tech kernel: XFS (sda1): Ending clean mount
May 02 11:03:02 n01.da.ceph.frickel.tech kernel: XFS (sda1): Unmounting Filesystem
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main_trigger:
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main_trigger: main_activate: path = /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: get_dm_uuid: get_dm_uuid /dev/sda1 uuid path is /sys/dev/block/8:1/dm/uuid
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sda1
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: mount: Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.vlQL9k with options noatime,inode64
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sda1 /var/lib/ceph/tmp/mnt.vlQL9k
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: activate: Cluster uuid is bd992a8d-0109-4cbe-96a1-c2a3cfe9f670
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: activate: Cluster name is ceph
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: activate: OSD uuid is 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: allocate_osd_id: Allocating OSD id...
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: mount_activate: Failed to activate
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: unmount: Unmounting /var/lib/ceph/tmp/mnt.vlQL9k
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.vlQL9k
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: Traceback (most recent call last):
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main(sys.argv[1:])
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: args.func(args)
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3532, in main_activate
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: reactivate=args.reactivate,
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3289, in mount_activate
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: (osd_id, cluster) = activate(path, activate_key_template, init)
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3452, in activate
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: keyring=keyring,
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 989, in allocate_osd_id
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: raise Error('ceph osd create failed', e, e.output)
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: ceph_disk.main.Error
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: Traceback (most recent call last):
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: main(sys.argv[1:])
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: args.func(args)
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4618, in main_trigger
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: raise Error('return code ' + str(ret))
May 02 11:03:02 n01.da.ceph.frickel.tech sh[14627]: ceph_disk.main.Error: Error: return code 1
May 02 11:03:02 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Main process exited, code=exited, status=1/FAILURE
May 02 11:03:02 n01.da.ceph.frickel.tech systemd[1]: Failed to start Ceph disk activation: /dev/sda1.
May 02 11:03:02 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Unit entered failed state.
May 02 11:03:02 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Failed with result 'exit-code'.
May 02 11:03:02 n01.da.ceph.frickel.tech kernel: sda: sda1
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: Starting Ceph disk activation: /dev/sda1...
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sda1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f19be2d6cf8>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /sbin/init --version
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main_trigger: trigger /dev/sda1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Mounting V4 Filesystem
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Ending clean mount
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: Reloading.
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: Stopped Ceph disk activation: /dev/sda1.
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: Starting Ceph disk activation: /dev/sda1...
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Unmounting Filesystem
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main_trigger:
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main_trigger: main_activate: path = /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: get_dm_uuid: get_dm_uuid /dev/sda1 uuid path is /sys/dev/block/8:1/dm/uuid
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: mount: Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.PoI73g with options noatime,inode64
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sda1 /var/lib/ceph/tmp/mnt.PoI73g
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: activate: Cluster uuid is bd992a8d-0109-4cbe-96a1-c2a3cfe9f670
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: activate: Cluster name is ceph
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: activate: OSD uuid is 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: allocate_osd_id: Allocating OSD id...
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: mount_activate: Failed to activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: unmount: Unmounting /var/lib/ceph/tmp/mnt.PoI73g
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.PoI73g
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: Traceback (most recent call last):
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main(sys.argv[1:])
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: args.func(args)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3532, in main_activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: reactivate=args.reactivate,
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3289, in mount_activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: (osd_id, cluster) = activate(path, activate_key_template, init)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3452, in activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: keyring=keyring,
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 989, in allocate_osd_id
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: raise Error('ceph osd create failed', e, e.output)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: ceph_disk.main.Error
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: Traceback (most recent call last):
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: main(sys.argv[1:])
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: args.func(args)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4618, in main_trigger
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: raise Error('return code ' + str(ret))
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: ceph_disk.main.Error: Error: return code 1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main_trigger: main_trigger: Namespace(cluster='ceph', dev='/dev/sda1', dmcrypt=None, dmcrypt_key_dir='/etc/ceph/dmcrypt-keys', func=<function main_trigger at 0x7f382562acf8>, log_stdout=True, prepend_to_path='/usr/bin', prog='ceph-disk', setgroup=None, setuser=None, statedir='/var/lib/ceph', sync=True, sysconfdir='/etc/ceph', verbose=True)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /sbin/init --version
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command_check_call: Running command: /bin/chown ceph:ceph /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main_trigger: trigger /dev/sda1 parttype 4fbd7e29-9d25-41b8-afd0-062c0ceff05d uuid 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /usr/sbin/ceph-disk --verbose activate /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Mounting V4 Filesystem
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Ending clean mount
May 02 11:03:03 n01.da.ceph.frickel.tech kernel: XFS (sda1): Unmounting Filesystem
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main_trigger:
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main_trigger: main_activate: path = /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: get_dm_uuid: get_dm_uuid /dev/sda1 uuid path is /sys/dev/block/8:1/dm/uuid
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /sbin/blkid -o udev -p /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/sda1
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: mount: Mounting /dev/sda1 on /var/lib/ceph/tmp/mnt.sgCl8W with options noatime,inode64
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command_check_call: Running command: /bin/mount -t xfs -o noatime,inode64 -- /dev/sda1 /var/lib/ceph/tmp/mnt.sgCl8W
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: activate: Cluster uuid is bd992a8d-0109-4cbe-96a1-c2a3cfe9f670
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: activate: Cluster name is ceph
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: activate: OSD uuid is 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: allocate_osd_id: Allocating OSD id...
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 33639697-4b71-4c2f-beff-a7b535f73aa9
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: mount_activate: Failed to activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: unmount: Unmounting /var/lib/ceph/tmp/mnt.sgCl8W
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.sgCl8W
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: Traceback (most recent call last):
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main(sys.argv[1:])
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: args.func(args)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3532, in main_activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: reactivate=args.reactivate,
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3289, in mount_activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: (osd_id, cluster) = activate(path, activate_key_template, init)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 3452, in activate
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: keyring=keyring,
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 989, in allocate_osd_id
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: raise Error('ceph osd create failed', e, e.output)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: ceph_disk.main.Error
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: Traceback (most recent call last):
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/sbin/ceph-disk", line 9, in <module>
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5230, in run
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: main(sys.argv[1:])
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 5181, in main
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: args.func(args)
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: File "/usr/lib/python2.7/dist-packages/ceph_disk/main.py", line 4618, in main_trigger
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: raise Error('return code ' + str(ret))
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14794]: ceph_disk.main.Error: Error: return code 1
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Main process exited, code=exited, status=1/FAILURE
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: Failed to start Ceph disk activation: /dev/sda1.
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Unit entered failed state.
May 02 11:03:03 n01.da.ceph.frickel.tech systemd[1]: ceph-disk@dev-sda1.service: Failed with result 'exit-code'.
May 02 11:03:08 n01.da.ceph.frickel.tech sshd[14520]: Received disconnect from 2a00:xxxx: 11: disconnected by user
May 02 11:03:08 n01.da.ceph.frickel.tech sshd[14520]: pam_unix(sshd:session): session closed for user root
It seems like ceph-disk is running some commands which error out, but this is not recognized by ceph-deploy (probably should). Also, it might help if ceph-disk would print the command output, not only the non-0 exit code, to see whats going on.
Before the run, all partitions from /dev/sda where removed and their first 1MB zero'ed. /dev/nvme0n1p1 is a 10GB partition on a (larger) NVMe disk, which should be used as journal for the to-be-created OSD on /dev/sda.
Debian Jessie, with deb https://download.ceph.com/debian-kraken jessie main packages (11.2.0-1~bpo80+1)
Updated by Vasu Kulkarni almost 7 years ago
- Project changed from devops to Ceph
- Subject changed from ceph-deploy osd create fails to ceph-disk fails to activate disk - Debian Jessie + Kraken
- Assignee set to Loïc Dachary
- ceph-qa-suite deleted (
ceph-deploy)
Assigning this to Loic as it is ceph-disk issue.
Updated by Loïc Dachary almost 7 years ago
May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: command: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 33639697-4b71-4c2f-beff-a7b535f73aa9 May 02 11:03:03 n01.da.ceph.frickel.tech sh[14691]: mount_activate: Failed to activate
When the disk activates, it needs access to the cluster. If you manually run /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring osd create --concise 33639697-4b71-4c2f-beff-a7b535f73aa9 on the machine, you can debug why it does not work. It could simply be a misconfiguration of the Ceph cluster (wrong /etc/ceph.conf or similar).
Updated by Loïc Dachary almost 7 years ago
- Status changed from New to Need More Info
Updated by Florian Klink almost 7 years ago
I gave up on ceph-deploy, and was successful with ceph-ansible instead. ;-)
Regarding this bug, it seems like the cluster monitors were not yet paired completely, so ceph-disk failed.
However, ceph-deploy should still fail if a ran command failed, and show the failing program's output IMHO.
Updated by Loïc Dachary almost 7 years ago
- Status changed from Need More Info to Rejected
Thanks for the update, very much appreciated !
Updated by Florian Klink almost 7 years ago
@Loïc Dachary: What about the failing command, and ceph-deploy not recognizing this? Should I open a new issue for that?
Updated by Loïc Dachary almost 7 years ago
You could open an issue against ceph-deploy, yes.
Actions