Actions
Bug #8988
closedAssertionError(s) in upgrade:firefly-x:stress-split-next---basic-plana
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755/
and
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387763/
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:====================================================================== 2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount) 2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:---------------------------------------------------------------------- 2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last): 2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp 2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: cls.env.setUp() 2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp 2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: raise ResponseError(cls.conn.response) 2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:ResponseError: 500: Internal Server Error 2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:====================================================================== 2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testContainerListing (test.functional.tests.TestAccount) 2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:---------------------------------------------------------------------- 2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last): 2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 262, in testContainerListing 2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr: self.assertEquals(a, b) 2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: Lists differ: ['167e6c2f7790448cb665118c0ca1... != ['167e6c2f7790448cb665118c0ca1... 2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:First differing element 4: 2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:72c48b7e4f774b6ebae34c9e9008b690 2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:525bc170f49f4e408b74dd2ca8f10b28 2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:Second list contains 1 additional elements. 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:First extra element 10: 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:e53eab2df63e444a8c2248f08eefbac3 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: ['167e6c2f7790448cb665118c0ca14064', 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '473c9ce271f24bb69255ea3a60f41e26', 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '4ad0858c06204b71968a45d33782e90d', 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '4de3250a2b3847bfb65743044ece7718', 2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:+ '525bc170f49f4e408b74dd2ca8f10b28', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: '72c48b7e4f774b6ebae34c9e9008b690', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: '8f49eb55dbae4f90a360e66260a29e05', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'b63f263822cb4b4d900af8a122573259', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'd1e9f200a3b8493f8f5c292dcbfb241e', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'dcf96a03c43049ffa799ea9fceffc27d', 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'e53eab2df63e444a8c2248f08eefbac3'] 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:====================================================================== 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testLastContainerMarker (test.functional.tests.TestAccount) 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:---------------------------------------------------------------------- 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last): 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 272, in testLastContainerMarker 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr: self.assertEquals(len(containers), len(self.env.containers)) 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: 11 != 10 2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:---------------------------------------------------------------------- 2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:Ran 137 tests in 874.291s 2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr: 2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:FAILED (errors=1, failures=2) 2014-07-30T23:59:17.951 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests args=args, File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run r.wait() File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 2014-07-30T23:59:17.952 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data' 2014-07-30T23:59:20.927 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data' 2014-07-30T23:59:21.377 INFO:teuthology.task.swift:Removing swift... 2014-07-30T23:59:21.377 INFO:teuthology.orchestra.run.plana47:Running: 'rm -rf /home/ubuntu/cephtest/swift' 2014-07-30T23:59:21.456 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task lambda: run_tests(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests args=args, File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run r.wait() File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 2014-07-30T23:59:21.457 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2014-07-30T23:59:21.457 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 29, in nested yield vars File "/home/teuthworker/src/teuthology_next/teuthology/task/rgw.py", line 808, in task yield File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks manager.__enter__() File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task lambda: run_tests(ctx=ctx, config=config), File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests args=args, File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run r.wait() File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait exitstatus=status, node=self.hostname) CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 2014-07-30T23:59:21.458 INFO:teuthology.task.rgw:Stopping apache...
archive_path: /var/lib/teuthworker/archive/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755 branch: next description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml 6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} email: ceph-qa@ceph.com job_id: '387755' last_in_suite: false machine_type: plana name: teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana nuke-on-error: true os_type: ubuntu os_version: '12.04' overrides: admin_socket: branch: next ceph: conf: mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 20 log-whitelist: - slow request - wrongly marked me down - objects unfound and apparently lost - log bound mismatch sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91 ceph-deploy: branch: dev: next conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: debug mon: 1 debug ms: 20 debug paxos: 20 osd default pool size: 2 install: ceph: sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91 s3tests: branch: next workunit: sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91 owner: scheduled_teuthology@teuthology priority: 1000 roles: - - mon.a - mon.b - mds.a - osd.0 - osd.1 - osd.2 - - osd.3 - osd.4 - osd.5 - mon.c - - client.0 suite: upgrade:firefly-x:stress-split suite_branch: master targets: ubuntu@plana47.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIDs+Sl9mfdFX0VZy41zfHkMJlD1OtErcQsSPLq3wrQ5HPPflg8IRtS45LHsYBAyBcAl4fmeG1q45jOrHj6yibJhT/HzxO6TOikYKUgr3uWy804NsTAP/A2SJ4wA8Y+1WyiAjBs0EAnISCnIl5Arj17NYPBroDeclKqZEJpL5UbPCmMibqEmjIPNVNulukox/Dh9GJZWlxqD0aKfTRnIOM+6RWHv2h9mCYVtU2ZbMx+QR0sxji5IJzg4ITlGaR9g+U3DnOU95pkxb3x6mE+H8udB58FONty/DHaAC/rXGfjkg056R3iq8FJUpF3HKKjILLRHjLb3ueK3tN1zrEueCt ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFHCeGWMPGOLyScKFkduv7aJL9bpMUPZQATO9lxpWu1NtzYndPJtWcyUxgWlItu75SJwpXx/l2GhPYcDKrR1Nl37+dbgs5TeDTbr9YdQBuLPbkbIZMQqO4GqUjurEwLU3vFUZ0X7PTlUqn6qwpT+I2YJua19eF2cRQFIGYVZMzaezm47uh67cdKFh0RTA1pSJ2qM/WMn91boRWcsRQrmn4BeOzfpGfSPDRjrHXHiPx3Br4zcOi/3lOxNFcEeoBrA47PMxvxVIlbmxKDfNjHpQQT18VFWb+qcTAzf+zdBy3iDRFFS45fPrqlWjGn9sK74EbRQanDrZlrFkg2a/HIe5T ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Dgv5XfnXOHEuwwOoK+xQtHEZusRHTm9I4oBJ41RYkPq2y5GA5OrXiWVlOrwBoXXdtCeW4ynErDhqiFuL3tYmwNEYzRWqnyqZte4qfsTz93Lhv7UEkageJ2iHNaUNt+H071A8JULR2CRtIxXu6zSSKC8vwmEirxqYj3pPRVm9TCa1iPaj8R3wPmeBjwVD9IU+zAuvIi6oWcqKrxZEdEOciMa72nGO58V7Wo0yICMST6day1jxIBnNaOqGnKafMQSiLAIUSChY+Q544o0LRZO3HW6k9eZlO5yqRUJN1p2H+QxOSG/PicKR2Trode3A/tZmtYqelF2FgOfLjEgBkkKF tasks: - internal.lock_machines: - 3 - plana - internal.save_config: null - internal.check_lock: null - internal.connect: null - internal.serialize_remote_roles: null - internal.check_conflict: null - internal.check_ceph_data: null - internal.vm_setup: null - internal.base: null - internal.archive: null - internal.coredump: null - internal.sudo: null - internal.syslog: null - internal.timer: null - chef: null - clock.check: null - install: branch: firefly - ceph: fs: xfs - install.upgrade: osd.0: null - ceph.restart: daemons: - osd.0 - osd.1 - osd.2 - thrashosds: chance_pgnum_grow: 1 chance_pgpnum_fix: 1 timeout: 1200 - ceph.restart: daemons: - mon.a wait-for-healthy: false wait-for-osds-up: true - workunit: branch: firefly clients: client.0: - cls/test_cls_rbd.sh - ceph.restart: daemons: - mon.b wait-for-healthy: false wait-for-osds-up: true - workunit: branch: firefly clients: client.0: - rbd/test_librbd.sh - install.upgrade: mon.c: null - ceph.restart: daemons: - mon.c wait-for-healthy: false wait-for-osds-up: true - ceph.wait_for_mon_quorum: - a - b - c - workunit: branch: firefly clients: client.0: - rbd/test_librbd_python.sh - rgw: client.0: idle_timeout: 300 - swift: client.0: rgw_server: client.0 - rados: clients: - client.0 objects: 500 op_weights: delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: next tube: plana verbose: true worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11466
description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml 2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml 6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml} duration: 3735.700578927994 failure_reason: 'Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a ''!fails_on_rgw''"' flavor: basic owner: scheduled_teuthology@teuthology success: false
Files
Actions