Project

General

Profile

Actions
Copy link

Bug #19489

closed

ceph-disk: failing to activate osd with multipath

Added by Matt Stroud about 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
jewel
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

We are trying to setup a new cluster using multipath disks. I'm able to prepare the osd just fine, but it fails to activate. Here is the output:

[root@mon01 ceph-config]# ceph-deploy osd activate osd01:mapper/mpatha
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /usr/bin/ceph-deploy osd activate osd01:mapper/mpatha
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : activate
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x170e710>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x1701b90>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('osd01', '/dev/mapper/mpatha', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks osd01:/dev/mapper/mpatha:
[osd01][DEBUG ] connected to host: osd01
[osd01][DEBUG ] detect platform information from remote host
[osd01][DEBUG ] detect machine type
[osd01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host osd01 disk /dev/mapper/mpatha
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[osd01][DEBUG ] find the location of an executable
[osd01][INFO  ] Running command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/mapper/mpatha
[osd01][WARNIN] main_activate: path = /dev/mapper/mpatha
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:3/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:3/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/mapper/mpatha
[osd01][WARNIN] Traceback (most recent call last):
[osd01][WARNIN]   File "/usr/sbin/ceph-disk", line 9, in <module>
[osd01][WARNIN]     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5047, in run
[osd01][WARNIN]     main(sys.argv[1:])
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4998, in main
[osd01][WARNIN]     args.func(args)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3357, in main_activate
[osd01][WARNIN]     reactivate=args.reactivate,
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3067, in mount_activate
[osd01][WARNIN]     e,
[osd01][WARNIN] ceph_disk.main.FilesystemTypeError: Cannot discover filesystem type: device /dev/mapper/mpatha: Line is truncated:
[osd01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/mapper/mpatha

I have tried switching between using root and the ceph user. With the ceph user I get permission denied while trying to activate. I'm trying to use Jewel due to its long term support, however I'm going to give kraken a try after reporting this.

Related issues 1 (0 open — 1 closed)

Copied to Ceph - Backport #20837: jewel: ceph-disk: failing to activate osd with multipathResolvedDavid DisseldorpActions
Actions
Copy link
 #1

Updated by Matt Stroud about 7 years ago

Sorry I pasted incorrect output:

[root@mon01 ceph-config]# ceph-deploy osd activate osd01:mapper/mpatha1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /usr/bin/ceph-deploy osd activate osd01:mapper/mpatha1
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : activate
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7fbd52801758>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x7fbd527f4b90>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  disk                          : [('osd01', '/dev/mapper/mpatha1', None)]
[ceph_deploy.osd][DEBUG ] Activating cluster ceph disks osd01:/dev/mapper/mpatha1:
[osd01][DEBUG ] connected to host: osd01
[osd01][DEBUG ] detect platform information from remote host
[osd01][DEBUG ] detect machine type
[osd01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] activating host osd01 disk /dev/mapper/mpatha1
[ceph_deploy.osd][DEBUG ] will use init type: systemd
[osd01][DEBUG ] find the location of an executable
[osd01][INFO  ] Running command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/mapper/mpatha1
[osd01][WARNIN] main_activate: path = /dev/mapper/mpatha1
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid is part1-mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid is part1-mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /usr/sbin/blkid -o udev -p /dev/mapper/mpatha1
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha1 uuid is part1-mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /sbin/blkid -p -s TYPE -o value -- /dev/mapper/mpatha1
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[osd01][WARNIN] mount: Mounting /dev/mapper/mpatha1 on /var/lib/ceph/tmp/mnt.EKR3Pp with options noatime,inode64
[osd01][WARNIN] command_check_call: Running command: /usr/bin/mount -t xfs -o noatime,inode64 -- /dev/mapper/mpatha1 /var/lib/ceph/tmp/mnt.EKR3Pp
[osd01][WARNIN] command: Running command: /usr/sbin/restorecon /var/lib/ceph/tmp/mnt.EKR3Pp
[osd01][WARNIN] activate: Cluster uuid is fb991e48-c425-4f82-a70e-5ce748ae186b
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[osd01][WARNIN] activate: Cluster name is ceph
[osd01][WARNIN] activate: OSD uuid is 924f6191-152c-4d1d-b920-9e7237467573
[osd01][WARNIN] activate: OSD id is 1
[osd01][WARNIN] activate: Initializing OSD...
[osd01][WARNIN] command_check_call: Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring mon getmap -o /var/lib/ceph/tmp/mnt.EKR3Pp/activate.monmap
[osd01][WARNIN] got monmap epoch 2
[osd01][WARNIN] command: Running command: /usr/bin/timeout 300 ceph-osd --cluster ceph --mkfs --mkkey -i 1 --monmap /var/lib/ceph/tmp/mnt.EKR3Pp/activate.monmap --osd-data /var/lib/ceph/tmp/mnt.EKR3Pp --osd-journal /var/lib/ceph/tmp/mnt.EKR3Pp/journal --osd-uuid 924f6191-152c-4d1d-b920-9e7237467573 --keyring /var/lib/ceph/tmp/mnt.EKR3Pp/keyring --setuser ceph --setgroup ceph
[osd01][WARNIN] mount_activate: Failed to activate
[osd01][WARNIN] unmount: Unmounting /var/lib/ceph/tmp/mnt.EKR3Pp
[osd01][WARNIN] command_check_call: Running command: /bin/umount -- /var/lib/ceph/tmp/mnt.EKR3Pp
[osd01][WARNIN] Traceback (most recent call last):
[osd01][WARNIN]   File "/usr/sbin/ceph-disk", line 9, in <module>
[osd01][WARNIN]     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5230, in run
[osd01][WARNIN]     main(sys.argv[1:])
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5181, in main
[osd01][WARNIN]     args.func(args)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3532, in main_activate
[osd01][WARNIN]     reactivate=args.reactivate,
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3289, in mount_activate
[osd01][WARNIN]     (osd_id, cluster) = activate(path, activate_key_template, init)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 3465, in activate
[osd01][WARNIN]     keyring=keyring,
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2927, in mkfs
[osd01][WARNIN]     '--setgroup', get_ceph_group(),
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2874, in ceph_osd_mkfs
[osd01][WARNIN]     raise Error('%s failed : %s' % (str(arguments), error))
[osd01][WARNIN] ceph_disk.main.Error: Error: ['ceph-osd', '--cluster', 'ceph', '--mkfs', '--mkkey', '-i', u'1', '--monmap', '/var/lib/ceph/tmp/mnt.EKR3Pp/activate.monmap', '--osd-data', '/var/lib/ceph/tmp/mnt.EKR3Pp', '--osd-journal', '/var/lib/ceph/tmp/mnt.EKR3Pp/journal', '--osd-uuid', u'924f6191-152c-4d1d-b920-9e7237467573', '--keyring', '/var/lib/ceph/tmp/mnt.EKR3Pp/keyring', '--setuser', 'ceph', '--setgroup', 'ceph'] failed : 2017-04-04 09:10:58.437244 7f045872c940 -1 filestore(/var/lib/ceph/tmp/mnt.EKR3Pp) mkjournal error creating journal on /var/lib/ceph/tmp/mnt.EKR3Pp/journal: (13) Permission denied
[osd01][WARNIN] 2017-04-04 09:10:58.437284 7f045872c940 -1 OSD::mkfs: ObjectStore::mkfs failed with error -13
[osd01][WARNIN] 2017-04-04 09:10:58.437395 7f045872c940 -1  ** ERROR: error creating empty object store in /var/lib/ceph/tmp/mnt.EKR3Pp: (13) Permission denied
[osd01][WARNIN]
[osd01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy][ERROR ] RuntimeError: Failed to execute command: /usr/sbin/ceph-disk -v activate --mark-init systemd --mount /dev/mapper/mpatha1

Actions
Copy link
 #2

Updated by Matt Stroud about 7 years ago

More issues:

[ceph@mon01 ceph-config]$ ceph-deploy osd prepare osd01:mapper/mpathc:mapper/mpatha
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (1.5.37): /bin/ceph-deploy osd prepare osd01:mapper/mpathc:mapper/mpatha
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  disk                          : [('osd01', '/dev/mapper/mpathc', '/dev/mapper/mpatha')]
[ceph_deploy.cli][INFO  ]  dmcrypt                       : False
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  bluestore                     : None
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : prepare
[ceph_deploy.cli][INFO  ]  dmcrypt_key_dir               : /etc/ceph/dmcrypt-keys
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x175e710>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  fs_type                       : xfs
[ceph_deploy.cli][INFO  ]  func                          : <function osd at 0x1751b90>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph_deploy.cli][INFO  ]  zap_disk                      : False
[ceph_deploy.osd][DEBUG ] Preparing cluster ceph disks osd01:/dev/mapper/mpathc:/dev/mapper/mpatha
[osd01][DEBUG ] connection detected need for sudo
[osd01][DEBUG ] connected to host: osd01
[osd01][DEBUG ] detect platform information from remote host
[osd01][DEBUG ] detect machine type
[osd01][DEBUG ] find the location of an executable
[ceph_deploy.osd][INFO  ] Distro info: CentOS Linux 7.3.1611 Core
[ceph_deploy.osd][DEBUG ] Deploying osd to osd01
[osd01][DEBUG ] write cluster configuration to /etc/ceph/{cluster}.conf
[osd01][WARNIN] osd keyring does not exist yet, creating one
[osd01][DEBUG ] create a keyring file
[ceph_deploy.osd][DEBUG ] Preparing host osd01 disk /dev/mapper/mpathc journal /dev/mapper/mpatha activate False
[osd01][DEBUG ] find the location of an executable
[osd01][INFO  ] Running command: sudo /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/mapper/mpathc /dev/mapper/mpatha
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=fsid
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --check-allows-journal -i 0 --cluster ceph --setuser ceph --setgroup ceph
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --check-wants-journal -i 0 --cluster ceph --setuser ceph --setgroup ceph
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --check-needs-journal -i 0 --cluster ceph --setuser ceph --setgroup ceph
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /usr/bin/ceph-osd --cluster=ceph --show-config-value=osd_journal_size
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpathc uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/dm-4 uuid path is /sys/dev/block/253:4/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/dm-4 uuid is mpath-360060e80074e840000304e8400004002
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mkfs_options_xfs
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mkfs_options_xfs
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_mount_options_xfs
[osd01][WARNIN] command: Running command: /usr/bin/ceph-conf --cluster=ceph --name=osd. --lookup osd_fs_mount_options_xfs
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:2/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:2/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] prepare_device: OSD will not be hot-swappable if journal is not the same device as the osd data
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:2/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:2/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] ptype_tobe_for_name: name = journal
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:2/dm/uuid
[osd01][WARNIN] get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-360060e80074e840000304e8400004000
[osd01][WARNIN]
[osd01][WARNIN] command: Running command: /sbin/parted --machine -- /dev/mapper/mpatha print
[osd01][WARNIN] get_free_partition_index: get_free_partition_index: analyzing BYT;
[osd01][WARNIN] /dev/mapper/mpatha:2199GB:dm:512:512:gpt:Linux device-mapper (multipath):;
[osd01][WARNIN]
[osd01][WARNIN] Traceback (most recent call last):
[osd01][WARNIN]   File "/usr/sbin/ceph-disk", line 9, in <module>
[osd01][WARNIN]     load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 5047, in run
[osd01][WARNIN]     main(sys.argv[1:])
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4998, in main
[osd01][WARNIN]     args.func(args)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1812, in main
[osd01][WARNIN]     Prepare.factory(args).prepare()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1801, in prepare
[osd01][WARNIN]     self.prepare_locked()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1832, in prepare_locked
[osd01][WARNIN]     self.data.prepare(self.journal)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2494, in prepare
[osd01][WARNIN]     self.prepare_device(*to_prepare_list)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2670, in prepare_device
[osd01][WARNIN]     to_prepare.prepare()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2003, in prepare
[osd01][WARNIN]     self.prepare_device()
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2095, in prepare_device
[osd01][WARNIN]     num=num)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1531, in create_partition
[osd01][WARNIN]     num = get_free_partition_index(dev=self.path)
[osd01][WARNIN]   File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1380, in get_free_partition_index
[osd01][WARNIN]     raise Error('parted output expected to contain ' + dev + ': ' + lines)
[osd01][WARNIN] ceph_disk.main.Error: Error: parted output expected to contain /dev/mapper/mpatha: BYT;
[osd01][WARNIN] /dev/mapper/mpatha:2199GB:dm:512:512:gpt:Linux device-mapper (multipath):;
[osd01][WARNIN]
[osd01][ERROR ] RuntimeError: command returned non-zero exit status: 1
[ceph_deploy.osd][ERROR ] Failed to execute command: /usr/sbin/ceph-disk -v prepare --cluster ceph --fs-type xfs -- /dev/mapper/mpathc /dev/mapper/mpatha
[ceph_deploy][ERROR ] GenericError: Failed to create 1 OSDs

Actions
Copy link
 #3

Updated by Nathan Cutler about 7 years ago

  • Project changed from Ceph to devops
  • Category deleted (OSD)
Actions
Copy link
 #4

Updated by Nathan Cutler about 7 years ago

Actions
Copy link
 #5

Updated by Loïc Dachary about 7 years ago

  • Project changed from devops to Ceph
  • Subject changed from Failing to activate osd to ceph-disk: failing to activate osd with multipath
Actions
Copy link
 #6

Updated by Loïc Dachary about 7 years ago

Are you able to workaround the problem with chown ceph:ceph on the relevant device ? I'm not suggesting this is the right thing to do, just trying to assert where the problem is ;-)

Actions
Copy link
 #7

Updated by Matt Stroud about 7 years ago

Loic Dachary wrote:

Are you able to workaround the problem with chown ceph:ceph on the relevant device ? I'm not suggesting this is the right thing to do, just trying to assert where the problem is ;-)

The fix I came up with was in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules line number 29 I changed the following:

ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-parttypeuuid/$env{ID_PART_ENTRY_TYPE}.$env{ID_PART_ENTRY_UUID}"

to
ENV{ID_PART_ENTRY_SCHEME}=="gpt", ENV{ID_PART_ENTRY_TYPE}=="?*", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-parttypeuuid/$env{ID_PART_ENTRY_TYPE}.$env{ID_PART_ENTRY_UUID}", SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"

Actions
Copy link
 #8

Updated by Matt Stroud almost 7 years ago

Any updates to this? I wouldn't expect this kind of behavior for a fresh install.

Actions
Copy link
 #9

Updated by David Disseldorp almost 7 years ago

  • Assignee set to David Disseldorp

Matt Stroud wrote:

Loic Dachary wrote:

Are you able to workaround the problem with chown ceph:ceph on the relevant device ? I'm not suggesting this is the right thing to do, just trying to assert where the problem is ;-)

The fix I came up with was in /lib/udev/rules.d/60-ceph-by-parttypeuuid.rules line number 29 I changed the following:
[...]
to
[...]

Thanks for the patch, Matt. I think the by-partuuid symlink should be created by 60-persistent-storage.rules, which is part of upstream udev. Please try the following change to 60-persistent-storage.rules instead:

--- 60-persistent-storage.rules 2017-07-04 21:07:57.136282571 +0200
+++ 60-persistent-storage.rules.new     2017-07-04 21:08:52.168990967 +0200
@@ -6,7 +6,7 @@
 ACTION=="remove", GOTO="persistent_storage_end" 

 SUBSYSTEM!="block", GOTO="persistent_storage_end" 
-KERNEL!="loop*|mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|nvme*|hd*|sd*|sr*|vd*|xvd*|bcache*|cciss*|dasd*|ubd*|scm*|pmem*", GOTO="persistent_storage_end" 
+KERNEL!="loop*|mmcblk*[0-9]|msblk*[0-9]|mspblk*[0-9]|nvme*|hd*|sd*|sr*|vd*|xvd*|bcache*|cciss*|dasd*|ubd*|scm*|pmem*|dm*", GOTO="persistent_storage_end" 

 # ignore partitions that span the entire disk
 TEST=="whole_disk", GOTO="persistent_storage_end"

Works for me... That said, there are still a couple of other minor issues that I'm still chasing up:
- the sd devices that correspond to individual paths for the dm device are triggering the ceph-disk activate udev rules
+ for the case of the journal, even ceph-disk suppress-activate still results in an error

Actions
Copy link
 #10

Updated by David Disseldorp almost 7 years ago

David Disseldorp wrote:
...

Thanks for the patch, Matt. I think the by-partuuid symlink should be created by 60-persistent-storage.rules, which is part of upstream udev. Please try the following change to 60-persistent-storage.rules instead:

[...]

Works for me... That said, there are still a couple of other minor issues that I'm still chasing up:
- the sd devices that correspond to individual paths for the dm device are triggering the ceph-disk activate udev rules
+ for the case of the journal, even ceph-disk suppress-activate still results in an error

Following further local testing, I submitted the udev 60-persistent-storage.rules fix upstream (https://lists.freedesktop.org/archives/systemd-devel/2017-July/039167.html), and proposed a fix for the ceph-disk suppress-activate journal issue via https://github.com/ceph/ceph/pull/16123 .

Actions
Copy link
 #11

Updated by David Disseldorp almost 7 years ago

David Disseldorp wrote:
...

Following further local testing, I submitted the udev 60-persistent-storage.rules fix upstream (https://lists.freedesktop.org/archives/systemd-devel/2017-July/039167.html), and proposed a fix for the ceph-disk suppress-activate journal issue via https://github.com/ceph/ceph/pull/16123 .

The udev dm by-partuuid symlink functionality has gone into upstream 13-dm-disk.rules (provided by lvm2/device-mapper), rather than 60-persistent-storage.rules:
https://sourceware.org/git/?p=lvm2.git;a=commit;h=c48149cf80c6582c2369bc7f8a33d794021d9dae

IMO this bug can be closed once https://github.com/ceph/ceph/pull/16123 has been merged.

Actions
Copy link
 #12

Updated by Nathan Cutler almost 7 years ago

  • Status changed from New to Fix Under Review
  • Target version deleted (v10.2.7)
  • Backport set to jewel

David, I guess it might take some time before distros (especially the stable versions like Trusty) roll out the updated 13-dm-disk.rules . . . is this safe to backport to jewel immediately?

Actions
Copy link
 #13

Updated by David Disseldorp almost 7 years ago

Nathan Cutler wrote:

David, I guess it might take some time before distros (especially the stable versions like Trusty) roll out the updated 13-dm-disk.rules . . . is this safe to backport to jewel immediately?

https://github.com/ceph/ceph/pull/16123 is an independent change. It's safe to go in without the udev changes.

Actions
Copy link
 #14

Updated by Kefu Chai over 6 years ago

  • Status changed from Fix Under Review to Resolved

instead of https://lists.freedesktop.org/archives/systemd-devel/2017-July/039167.html, the lvm2 upstream has a fix https://sourceware.org/git/?p=lvm2.git;a=commit;h=c48149cf80c6582c2369bc7f8a33d794021d9dae. so we should just wait until the lvm2 change is included by the downstream distros.

Actions
Copy link
 #15

Updated by Kefu Chai over 6 years ago

  • Status changed from Resolved to Pending Backport
  • Assignee deleted (David Disseldorp)
Actions
Copy link
 #16

Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #20837: jewel: ceph-disk: failing to activate osd with multipath added
Actions
Copy link
 #17

Updated by Nathan Cutler over 6 years ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link

Also available in: Atom PDF