Actions
Bug #15350
closedceph-disk failed on centos in ceph-disk-jewel-distro-basic-mira
Status:
Duplicate
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
ceph-disk
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2016-03-31_23:13:02-ceph-disk-jewel-distro-basic-mira/
Job: 101561
LOgs: http://qa-proxy.ceph.com/teuthology/teuthology-2016-03-31_23:13:02-ceph-disk-jewel-distro-basic-mira/101561/teuthology.log
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid path is /sys/dev/block/253:0/dm/uuid
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:get_dm_uuid: get_dm_uuid /dev/mapper/mpatha uuid is mpath-2001b4d2000000000
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:populate_data_path_device: Creating xfs fs on /dev/dm-1
2016-04-01T00:08:39.307 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:command_check_call: Running command: /usr/sbin/mkfs -t xfs -f -i size=2048 -- /dev/dm-1
2016-04-01T00:08:39.710 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:existing superblock read failed: Input/output error
2016-04-01T00:08:39.710 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:mkfs.xfs: pwrite64 failed: Input/output error
2016-04-01T00:08:39.710 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:meta-data=/dev/dm-1 isize=2048 agcount=4, agsize=61041197 blks
2016-04-01T00:08:39.710 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:= sectsz=512 attr=2, projid32bit=1
2016-04-01T00:08:39.711 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:= crc=0 finobt=0
2016-04-01T00:08:39.711 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:data = bsize=4096 blocks=244164785, imaxpct=25
2016-04-01T00:08:39.711 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:= sunit=0 swidth=0 blks
2016-04-01T00:08:39.712 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:naming =version 2 bsize=4096 ascii-ci=0 ftype=0
2016-04-01T00:08:39.712 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:log =internal log bsize=4096 blocks=119221, version=2
2016-04-01T00:08:39.712 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:= sectsz=512 sunit=0 blks, lazy-count=1
2016-04-01T00:08:39.712 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:realtime =none extsz=4096 blocks=0, rtextents=0
2016-04-01T00:08:39.879 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:Traceback (most recent call last):
2016-04-01T00:08:39.879 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/sbin/ceph-disk", line 9, in <module>
2016-04-01T00:08:39.879 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:load_entry_point('ceph-disk==1.0.0', 'console_scripts', 'ceph-disk')()
2016-04-01T00:08:39.879 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4964, in run
2016-04-01T00:08:39.880 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:main(sys.argv[1:])
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 4915, in main
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:args.func(args)
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1774, in main
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:Prepare.factory(args).prepare()
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1762, in prepare
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:self.prepare_locked()
2016-04-01T00:08:39.881 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 1794, in prepare_locked
2016-04-01T00:08:39.882 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:self.data.prepare(self.journal)
2016-04-01T00:08:39.882 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2446, in prepare
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:self.prepare_device(*to_prepare_list)
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2624, in prepare_device
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:self.populate_data_path_device(*to_prepare_list)
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:File "/usr/lib/python2.7/site-packages/ceph_disk/main.py", line 2579, in populate_data_path_device
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:raise Error(e)
2016-04-01T00:08:39.883 INFO:tasks.workunit.client.0.mira037.stderr:DEBUG:CephDisk:ceph_disk.main.Error: Error: Command '['/usr/sbin/mkfs', '-t', 'xfs', '-f', '-i', 'size=2048', '--', '/dev/dm-1']' returned non-zero exit status 1
2016-04-01T00:08:39.937 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py::TestCephDisk::test_activate_multipath FAILED
2016-04-01T00:08:39.937 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.937 INFO:tasks.workunit.client.0.mira037.stdout:=================================== FAILURES ===================================
2016-04-01T00:08:39.937 INFO:tasks.workunit.client.0.mira037.stdout:_____________________ TestCephDisk.test_destroy_osd_by_id ______________________
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout:self = <ceph-disk-test.TestCephDisk object at 0x1fe79d0>
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout: def test_destroy_osd_by_id(self):
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout: c = CephDisk()
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout: disk = c.unused_disks()[0]
2016-04-01T00:08:39.938 INFO:tasks.workunit.client.0.mira037.stdout: osd_uuid = str(uuid.uuid1())
2016-04-01T00:08:39.939 INFO:tasks.workunit.client.0.mira037.stdout: c.sh("ceph-disk --verbose prepare --osd-uuid " + osd_uuid + " " + disk)
2016-04-01T00:08:39.939 INFO:tasks.workunit.client.0.mira037.stdout: c.wait_for_osd_up(osd_uuid)
2016-04-01T00:08:39.939 INFO:tasks.workunit.client.0.mira037.stdout: c.check_osd_status(osd_uuid)
2016-04-01T00:08:39.939 INFO:tasks.workunit.client.0.mira037.stdout:> c.destroy_osd(osd_uuid)
2016-04-01T00:08:39.939 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py:277:
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py:168: in destroy_osd
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout: """.format(id=id))
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.940 INFO:tasks.workunit.client.0.mira037.stdout:command = '\n set -xe\n ceph-disk --verbose deactivate --deactivate-by-id 2\n ceph-disk --verbose destroy --destroy-by-id 2 --zap\n '
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: @staticmethod
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: def sh(command):
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug(":sh: " + command)
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: proc = subprocess.Popen(
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: args=command,
2016-04-01T00:08:39.941 INFO:tasks.workunit.client.0.mira037.stdout: stdout=subprocess.PIPE,
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: stderr=subprocess.STDOUT,
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: shell=True,
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: bufsize=1)
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: lines = []
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: with proc.stdout:
2016-04-01T00:08:39.942 INFO:tasks.workunit.client.0.mira037.stdout: for line in iter(proc.stdout.readline, b''):
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: line = line.decode('utf-8')
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: if 'dangerous and experimental' in line:
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug('SKIP dangerous and experimental')
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: continue
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: lines.append(line)
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug(line.strip().encode('ascii', 'ignore'))
2016-04-01T00:08:39.943 INFO:tasks.workunit.client.0.mira037.stdout: if proc.wait() != 0:
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout: raise subprocess.CalledProcessError(
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout: returncode=proc.returncode,
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout:> cmd=command
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout: )
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout:E CalledProcessError: Command '
2016-04-01T00:08:39.944 INFO:tasks.workunit.client.0.mira037.stdout:E set -xe
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:E ceph-disk --verbose deactivate --deactivate-by-id 2
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:E ceph-disk --verbose destroy --destroy-by-id 2 --zap
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:E ' returned non-zero exit status 1
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py:95: CalledProcessError
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:_____________________ TestCephDisk.test_activate_multipath _____________________
2016-04-01T00:08:39.945 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout:self = <ceph-disk-test.TestCephDisk object at 0x1f88850>
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout: def test_activate_multipath(self):
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout: c = CephDisk()
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout: if c.sh("lsb_release -si").strip() != 'CentOS':
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout: pytest.skip(
2016-04-01T00:08:39.946 INFO:tasks.workunit.client.0.mira037.stdout: "see issue https://bugs.launchpad.net/ubuntu/+source/multipath-tools/+bug/1488688")
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: c.ensure_sd()
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: #
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: # Figure out the name of the multipath device
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: #
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: disk = c.unused_disks('sd.')[0]
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: c.sh("mpathconf --enable || true")
2016-04-01T00:08:39.947 INFO:tasks.workunit.client.0.mira037.stdout: c.sh("multipath " + disk)
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: holders = os.listdir(
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: "/sys/block/" + os.path.basename(disk) + "/holders")
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: assert 1 == len(holders)
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: name = open("/sys/block/" + holders[0] + "/dm/name").read()
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: multipath = "/dev/mapper/" + name
2016-04-01T00:08:39.948 INFO:tasks.workunit.client.0.mira037.stdout: #
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout: # Prepare the multipath device
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout: #
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout: osd_uuid = str(uuid.uuid1())
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout: c.sh("ceph-disk --verbose zap " + multipath)
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout: c.sh("ceph-disk --verbose prepare --osd-uuid " + osd_uuid +
2016-04-01T00:08:39.949 INFO:tasks.workunit.client.0.mira037.stdout:> " " + multipath)
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py:613:
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:command = 'ceph-disk --verbose prepare --osd-uuid 8bf997e4-f7d8-11e5-ac46-002590085a60 /dev/mapper/mpatha\n'
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.950 INFO:tasks.workunit.client.0.mira037.stdout: @staticmethod
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: def sh(command):
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug(":sh: " + command)
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: proc = subprocess.Popen(
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: args=command,
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: stdout=subprocess.PIPE,
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: stderr=subprocess.STDOUT,
2016-04-01T00:08:39.951 INFO:tasks.workunit.client.0.mira037.stdout: shell=True,
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: bufsize=1)
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: lines = []
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: with proc.stdout:
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: for line in iter(proc.stdout.readline, b''):
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: line = line.decode('utf-8')
2016-04-01T00:08:39.952 INFO:tasks.workunit.client.0.mira037.stdout: if 'dangerous and experimental' in line:
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug('SKIP dangerous and experimental')
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: continue
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: lines.append(line)
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: LOG.debug(line.strip().encode('ascii', 'ignore'))
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: if proc.wait() != 0:
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: raise subprocess.CalledProcessError(
2016-04-01T00:08:39.953 INFO:tasks.workunit.client.0.mira037.stdout: returncode=proc.returncode,
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:> cmd=command
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout: )
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:E CalledProcessError: Command 'ceph-disk --verbose prepare --osd-uuid 8bf997e4-f7d8-11e5-ac46-002590085a60 /dev/mapper/mpatha
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:E ' returned non-zero exit status 1
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:../../../workunit.client.0/ceph-disk/ceph-disk-test.py:95: CalledProcessError
2016-04-01T00:08:39.954 INFO:tasks.workunit.client.0.mira037.stdout:==================== 2 failed, 19 passed in 2623.10 seconds ====================
2016-04-01T00:08:39.955 INFO:tasks.workunit.client.0.mira037.stderr:+ result=1
2016-04-01T00:08:39.955 INFO:tasks.workunit.client.0.mira037.stderr:++ id -u
2016-04-01T00:08:39.956 INFO:tasks.workunit.client.0.mira037.stderr:++ dirname /home/ubuntu/cephtest/workunit.client.0/ceph-disk/ceph-disk.sh
2016-04-01T00:08:39.956 INFO:tasks.workunit.client.0.mira037.stderr:+ sudo chown -R 1000 /home/ubuntu/cephtest/workunit.client.0/ceph-disk
2016-04-01T00:08:39.964 INFO:tasks.workunit.client.0.mira037.stderr:+ exit 1
2016-04-01T00:08:39.965 INFO:tasks.workunit:Stopping ['ceph-disk/ceph-disk.sh'] on client.0...
Actions