Bug #8988
AssertionError(s) in upgrade:firefly-x:stress-split-next---basic-plana
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Logs are in http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755/
and
http://qa-proxy.ceph.com/teuthology/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387763/
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount)
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.933 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: cls.env.setUp()
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr: raise ResponseError(cls.conn.response)
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:ResponseError: 500: Internal Server Error
2014-07-30T23:59:17.934 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testContainerListing (test.functional.tests.TestAccount)
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.935 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 262, in testContainerListing
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr: self.assertEquals(a, b)
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: Lists differ: ['167e6c2f7790448cb665118c0ca1... != ['167e6c2f7790448cb665118c0ca1...
2014-07-30T23:59:17.936 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:First differing element 4:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:72c48b7e4f774b6ebae34c9e9008b690
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:525bc170f49f4e408b74dd2ca8f10b28
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.937 INFO:teuthology.orchestra.run.plana47.stderr:Second list contains 1 additional elements.
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:First extra element 10:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:e53eab2df63e444a8c2248f08eefbac3
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: ['167e6c2f7790448cb665118c0ca14064',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '473c9ce271f24bb69255ea3a60f41e26',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '4ad0858c06204b71968a45d33782e90d',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr: '4de3250a2b3847bfb65743044ece7718',
2014-07-30T23:59:17.938 INFO:teuthology.orchestra.run.plana47.stderr:+ '525bc170f49f4e408b74dd2ca8f10b28',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: '72c48b7e4f774b6ebae34c9e9008b690',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: '8f49eb55dbae4f90a360e66260a29e05',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'b63f263822cb4b4d900af8a122573259',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'd1e9f200a3b8493f8f5c292dcbfb241e',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'dcf96a03c43049ffa799ea9fceffc27d',
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr: 'e53eab2df63e444a8c2248f08eefbac3']
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.939 INFO:teuthology.orchestra.run.plana47.stderr:======================================================================
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:FAIL: testLastContainerMarker (test.functional.tests.TestAccount)
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:Traceback (most recent call last):
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 272, in testLastContainerMarker
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr: self.assertEquals(len(containers), len(self.env.containers))
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:AssertionError: 11 != 10
2014-07-30T23:59:17.940 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:----------------------------------------------------------------------
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:Ran 137 tests in 874.291s
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:
2014-07-30T23:59:17.941 INFO:teuthology.orchestra.run.plana47.stderr:FAILED (errors=1, failures=2)
2014-07-30T23:59:17.951 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
args=args,
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
r.wait()
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"
2014-07-30T23:59:17.952 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid foo.client.0 --purge-data'
2014-07-30T23:59:20.927 INFO:teuthology.orchestra.run.plana47:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin -n client.0 user rm --uid bar.client.0 --purge-data'
2014-07-30T23:59:21.377 INFO:teuthology.task.swift:Removing swift...
2014-07-30T23:59:21.377 INFO:teuthology.orchestra.run.plana47:Running: 'rm -rf /home/ubuntu/cephtest/swift'
2014-07-30T23:59:21.456 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
lambda: run_tests(ctx=ctx, config=config),
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
args=args,
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
r.wait()
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"
2014-07-30T23:59:21.457 DEBUG:teuthology.run_tasks:Unwinding manager rgw
2014-07-30T23:59:21.457 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 29, in nested
yield vars
File "/home/teuthworker/src/teuthology_next/teuthology/task/rgw.py", line 808, in task
yield
File "/home/teuthworker/src/teuthology_next/teuthology/run_tasks.py", line 45, in run_tasks
manager.__enter__()
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 255, in task
lambda: run_tests(ctx=ctx, config=config),
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/contextutil.py", line 27, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/home/teuthworker/src/teuthology_next/teuthology/task/swift.py", line 175, in run_tests
args=args,
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/cluster.py", line 64, in run
return [remote.run(**kwargs) for remote in remotes]
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/remote.py", line 114, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 401, in run
r.wait()
File "/home/teuthworker/src/teuthology_next/teuthology/orchestra/run.py", line 102, in wait
exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"
2014-07-30T23:59:21.458 INFO:teuthology.task.rgw:Stopping apache...
archive_path: /var/lib/teuthworker/archive/teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana/387755
branch: next
description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
email: ceph-qa@ceph.com
job_id: '387755'
last_in_suite: false
machine_type: plana
name: teuthology-2014-07-30_13:09:01-upgrade:firefly-x:stress-split-next---basic-plana
nuke-on-error: true
os_type: ubuntu
os_version: '12.04'
overrides:
admin_socket:
branch: next
ceph:
conf:
mon:
debug mon: 20
debug ms: 1
debug paxos: 20
osd:
debug filestore: 20
debug journal: 20
debug ms: 1
debug osd: 20
log-whitelist:
- slow request
- wrongly marked me down
- objects unfound and apparently lost
- log bound mismatch
sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
ceph-deploy:
branch:
dev: next
conf:
client:
log file: /var/log/ceph/ceph-$name.$pid.log
mon:
debug mon: 1
debug ms: 20
debug paxos: 20
osd default pool size: 2
install:
ceph:
sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
s3tests:
branch: next
workunit:
sha1: 6e6fc23c7e3251a5132d29c46a147bdcef9bfd91
owner: scheduled_teuthology@teuthology
priority: 1000
roles:
- - mon.a
- mon.b
- mds.a
- osd.0
- osd.1
- osd.2
- - osd.3
- osd.4
- osd.5
- mon.c
- - client.0
suite: upgrade:firefly-x:stress-split
suite_branch: master
targets:
ubuntu@plana47.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIDs+Sl9mfdFX0VZy41zfHkMJlD1OtErcQsSPLq3wrQ5HPPflg8IRtS45LHsYBAyBcAl4fmeG1q45jOrHj6yibJhT/HzxO6TOikYKUgr3uWy804NsTAP/A2SJ4wA8Y+1WyiAjBs0EAnISCnIl5Arj17NYPBroDeclKqZEJpL5UbPCmMibqEmjIPNVNulukox/Dh9GJZWlxqD0aKfTRnIOM+6RWHv2h9mCYVtU2ZbMx+QR0sxji5IJzg4ITlGaR9g+U3DnOU95pkxb3x6mE+H8udB58FONty/DHaAC/rXGfjkg056R3iq8FJUpF3HKKjILLRHjLb3ueK3tN1zrEueCt
ubuntu@plana57.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFHCeGWMPGOLyScKFkduv7aJL9bpMUPZQATO9lxpWu1NtzYndPJtWcyUxgWlItu75SJwpXx/l2GhPYcDKrR1Nl37+dbgs5TeDTbr9YdQBuLPbkbIZMQqO4GqUjurEwLU3vFUZ0X7PTlUqn6qwpT+I2YJua19eF2cRQFIGYVZMzaezm47uh67cdKFh0RTA1pSJ2qM/WMn91boRWcsRQrmn4BeOzfpGfSPDRjrHXHiPx3Br4zcOi/3lOxNFcEeoBrA47PMxvxVIlbmxKDfNjHpQQT18VFWb+qcTAzf+zdBy3iDRFFS45fPrqlWjGn9sK74EbRQanDrZlrFkg2a/HIe5T
ubuntu@plana58.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC7Dgv5XfnXOHEuwwOoK+xQtHEZusRHTm9I4oBJ41RYkPq2y5GA5OrXiWVlOrwBoXXdtCeW4ynErDhqiFuL3tYmwNEYzRWqnyqZte4qfsTz93Lhv7UEkageJ2iHNaUNt+H071A8JULR2CRtIxXu6zSSKC8vwmEirxqYj3pPRVm9TCa1iPaj8R3wPmeBjwVD9IU+zAuvIi6oWcqKrxZEdEOciMa72nGO58V7Wo0yICMST6day1jxIBnNaOqGnKafMQSiLAIUSChY+Q544o0LRZO3HW6k9eZlO5yqRUJN1p2H+QxOSG/PicKR2Trode3A/tZmtYqelF2FgOfLjEgBkkKF
tasks:
- internal.lock_machines:
- 3
- plana
- internal.save_config: null
- internal.check_lock: null
- internal.connect: null
- internal.serialize_remote_roles: null
- internal.check_conflict: null
- internal.check_ceph_data: null
- internal.vm_setup: null
- internal.base: null
- internal.archive: null
- internal.coredump: null
- internal.sudo: null
- internal.syslog: null
- internal.timer: null
- chef: null
- clock.check: null
- install:
branch: firefly
- ceph:
fs: xfs
- install.upgrade:
osd.0: null
- ceph.restart:
daemons:
- osd.0
- osd.1
- osd.2
- thrashosds:
chance_pgnum_grow: 1
chance_pgpnum_fix: 1
timeout: 1200
- ceph.restart:
daemons:
- mon.a
wait-for-healthy: false
wait-for-osds-up: true
- workunit:
branch: firefly
clients:
client.0:
- cls/test_cls_rbd.sh
- ceph.restart:
daemons:
- mon.b
wait-for-healthy: false
wait-for-osds-up: true
- workunit:
branch: firefly
clients:
client.0:
- rbd/test_librbd.sh
- install.upgrade:
mon.c: null
- ceph.restart:
daemons:
- mon.c
wait-for-healthy: false
wait-for-osds-up: true
- ceph.wait_for_mon_quorum:
- a
- b
- c
- workunit:
branch: firefly
clients:
client.0:
- rbd/test_librbd_python.sh
- rgw:
client.0:
idle_timeout: 300
- swift:
client.0:
rgw_server: client.0
- rados:
clients:
- client.0
objects: 500
op_weights:
delete: 50
read: 100
rollback: 50
snap_create: 50
snap_remove: 50
write: 100
ops: 4000
teuthology_branch: next
tube: plana
verbose: true
worker_log: /var/lib/teuthworker/archive/worker_logs/worker.plana.11466
description: upgrade:firefly-x:stress-split/{0-cluster/start.yaml 1-firefly-install/firefly.yaml
2-partial-upgrade/firsthalf.yaml 3-thrash/default.yaml 4-mon/mona.yaml 5-workload/rbd-cls.yaml
6-next-mon/monb.yaml 7-workload/rbd_api.yaml 8-next-mon/monc.yaml 9-workload/{rbd-python.yaml
rgw-s3tests.yaml snaps-many-objects.yaml} distros/ubuntu_12.04.yaml}
duration: 3735.700578927994
failure_reason: 'Command failed on plana47 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf
/home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional
-v -a ''!fails_on_rgw''"'
flavor: basic
owner: scheduled_teuthology@teuthology
success: false
Related issues
History
#1 Updated by Yuri Weinstein almost 9 years ago
- Project changed from teuthology to Ceph
#2 Updated by Loïc Dachary almost 9 years ago
- http://pulpito.ceph.com/loic-2014-08-08_12:17:30-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps/406944
- http://pulpito.ceph.com/loic-2014-08-08_12:17:30-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps/406941
- http://pulpito.ceph.com/loic-2014-08-08_12:17:30-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps/406946
- http://pulpito.ceph.com/loic-2014-08-08_12:17:30-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps/406947
#3 Updated by Loïc Dachary almost 9 years ago
The errors for each failure are different and suggests the tests are failing for an independent reason such as the cluster going down before the test complete or something similar.
2014-08-08T11:57:30.497 INFO:teuthology.orchestra.run.vpm146.stderr:====================================================================== 2014-08-08T11:57:30.497 INFO:teuthology.orchestra.run.vpm146.stderr:FAIL: testCreate (test.functional.tests.TestContainerUTF8) 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:---------------------------------------------------------------------- 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:Traceback (most recent call last): 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 447, in testCreate 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr: self.assert_(cont.create()) 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr:AssertionError: False is not true 2014-08-08T11:57:30.498 INFO:teuthology.orchestra.run.vpm146.stderr: 2014-08-08T11:57:30.499 INFO:teuthology.orchestra.run.vpm146.stderr:---------------------------------------------------------------------- 2014-08-08T11:57:30.499 INFO:teuthology.orchestra.run.vpm146.stderr:Ran 137 tests in 2077.215s
2014-08-08T11:38:10.693 INFO:teuthology.orchestra.run.vpm137.stderr:====================================================================== 2014-08-08T11:38:10.693 INFO:teuthology.orchestra.run.vpm137.stderr:ERROR: testStructure (test.functional.tests.TestContainerPaths) 2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:---------------------------------------------------------------------- 2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr:Traceback (most recent call last): 2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp 2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr: cls.env.setUp() 2014-08-08T11:38:10.694 INFO:teuthology.orchestra.run.vpm137.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 687, in setUp 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: 'application/directory'}) 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 739, in write_random 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: if not self.write(data, hdrs=hdrs, parms=parms, cfg=cfg): 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 731, in write 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: raise ResponseError(self.conn.response) 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr:ResponseError: 500: Internal Server Error 2014-08-08T11:38:10.695 INFO:teuthology.orchestra.run.vpm137.stderr: 2014-08-08T11:38:10.989 INFO:teuthology.orchestra.run.vpm137.stderr:---------------------------------------------------------------------- 2014-08-08T11:38:10.990 INFO:teuthology.orchestra.run.vpm137.stderr:Ran 137 tests in 3036.831s
2014-08-08T12:42:51.996 INFO:teuthology.orchestra.run.vpm134.stderr:====================================================================== 2014-08-08T12:42:51.996 INFO:teuthology.orchestra.run.vpm134.stderr:ERROR: testBadHeaders (test.functional.tests.TestFileUTF8) 2014-08-08T12:42:51.997 INFO:teuthology.orchestra.run.vpm134.stderr:---------------------------------------------------------------------- 2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:in_osds: [4, 0, 3, 1, 2] out_osds: [5] dead_osds: [] live_osds: [0, 3, 2, 1, 4, 5] 2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T12:42:52.843 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [4, 0, 3, 1, 2] 2014-08-08T12:42:52.843 INFO:teuthology.orchestra.run.vpm132:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 2' 2014-08-08T12:42:53.228 INFO:teuthology.orchestra.run.vpm132.stderr:marked out osd.2. 2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr:Traceback (most recent call last): 2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 122, in setUp 2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr: super(Base2, self).setUp() 2014-08-08T12:42:53.622 INFO:teuthology.orchestra.run.vpm134.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp 2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr: cls.env.setUp() 2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 796, in setUp 2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr: raise ResponseError(cls.conn.response) 2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr:ResponseError: 500: Internal Server Error 2014-08-08T12:42:53.623 INFO:teuthology.orchestra.run.vpm134.stderr: 2014-08-08T12:42:53.624 INFO:teuthology.orchestra.run.vpm134.stderr:---------------------------------------------------------------------- 2014-08-08T12:42:54.731 INFO:teuthology.orchestra.run.vpm134.stderr:Ran 137 tests in 4019.450s 2014-08-08T12:42:54.731 INFO:teuthology.orchestra.run.vpm134.stderr: 2014-08-08T12:42:55.106 INFO:teuthology.orchestra.run.vpm134.stderr:FAILED (errors=1)
2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:in_osds: [4, 2, 1, 0, 3, 5] out_osds: [] dead_osds: [1] live_osds: [0, 4, 2, 5, 3] 2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:choose_action: min_in 3 min_out 0 min_live 2 min_dead 0 2014-08-08T11:01:54.970 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [4, 2, 1, 0, 3, 5] 2014-08-08T11:01:54.970 INFO:teuthology.orchestra.run.vpm166:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph osd out 0' 2014-08-08T11:01:55.002 INFO:teuthology.orchestra.run.vpm167.stderr:testZeroByteFile (test.functional.tests.TestFileUTF8) ... ok 2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr: 2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr:====================================================================== 2014-08-08T11:01:55.004 INFO:teuthology.orchestra.run.vpm167.stderr:ERROR: testContainerListing (test.functional.tests.TestContainerPaths) 2014-08-08T11:01:55.005 INFO:teuthology.orchestra.run.vpm167.stderr:---------------------------------------------------------------------- 2014-08-08T11:01:55.005 INFO:teuthology.orchestra.run.vpm167.stderr:Traceback (most recent call last): 2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp 2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr: cls.env.setUp() 2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 637, in setUp 2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr: raise ResponseError(cls.conn.response) 2014-08-08T11:01:55.006 INFO:teuthology.orchestra.run.vpm167.stderr:ResponseError: 500: Internal Server Error 2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr: 2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:---------------------------------------------------------------------- 2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr:Ran 137 tests in 1508.034s 2014-08-08T11:01:55.007 INFO:teuthology.orchestra.run.vpm167.stderr: 2014-08-08T11:01:55.008 INFO:teuthology.orchestra.run.vpm167.stderr:FAILED (errors=1)
#4 Updated by Loïc Dachary almost 9 years ago
- File error-log.txt View added
In the attached file, each part separated with ----------------------------- is the output between the last successfull rgw test and the test that fail. Maybe there is a common pattern that would explain why it fails at this point ? error-log.txt
#5 Updated by Loïc Dachary almost 9 years ago
- Project changed from Ceph to rgw
#6 Updated by Loïc Dachary almost 9 years ago
In a few tickets it is suggested that this may be an idle timeout problem. I rescheduled a suite after changing all timeouts from 300 to 1200 to check if that shows improvement.
diff --git a/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml b/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
index e1c223c..b94dc3e 100644
--- a/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
+++ b/suites/rgw/verify/tasks/rgw_s3tests_multiregion.yaml
@@ -36,14 +36,14 @@ tasks:
master zone: r1z1
zones: [r1z1]
client.0:
- idle_timeout: 300
+ idle_timeout: 1200
valgrind: [--tool=memcheck]
system user:
name: client0-system-user
access key: 1te6NH5mcdcq0Tc5i8i2
secret key: 1y4IOauQoL18Gp2zM7lC1vLmoawgqcYPbYGcWfXv
client.1:
- idle_timeout: 300
+ idle_timeout: 1200
valgrind: [--tool=memcheck]
system user:
name: client1-system-user
diff --git a/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml b/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
index c324c03..c2a5eda 100644
--- a/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
+++ b/suites/upgrade/dumpling-x/stress-split/9-workload/rgw-s3tests.yaml
@@ -1,7 +1,7 @@
tasks:
- rgw:
client.0:
- idle_timeout: 300
+ idle_timeout: 1200
- swift:
client.0:
rgw_server: client.0
diff --git a/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml b/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
index c324c03..c2a5eda 100644
--- a/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
+++ b/suites/upgrade/firefly-x/stress-split/9-workload/rgw-s3tests.yaml
@@ -1,7 +1,7 @@
tasks:
- rgw:
client.0:
- idle_timeout: 300
+ idle_timeout: 1200
- swift:
client.0:
rgw_server: client.0
diff --git a/tasks/rgw.py b/tasks/rgw.py
index 5cf72dc..d0ab425 100644
--- a/tasks/rgw.py
+++ b/tasks/rgw.py
@@ -86,7 +86,7 @@ def ship_apache_configs(ctx, config, role_endpoints):
system_type = teuthology.get_system_type(remote)
if not conf:
conf = {}
- idle_timeout = conf.get('idle_timeout', 30)
+ idle_timeout = conf.get('idle_timeout', 1200)
if system_type == 'deb':
mod_path = '/usr/lib/apache2/modules'
print_continue = 'on'
#7 Updated by Loïc Dachary almost 9 years ago
- Status changed from New to Fix Under Review
#8 Updated by Sage Weil almost 9 years ago
- Priority changed from Normal to Urgent
#9 Updated by Sage Weil almost 9 years ago
- Status changed from Fix Under Review to In Progress
#10 Updated by Loïc Dachary almost 9 years ago
To confirm there is a large delay requiring a large idle_timeout:
ubuntu@teuthology:/a/teuthology-2014-08-13_09:42:01-upgrade:firefly-x:stress-split-next---basic-vps/422009$ zgrep 'slow req' remote/*/log/ceph-mon* | sed -ne 's/.* > //p' | sort -n | tail -1 904.913166 secs
#11 Updated by Loïc Dachary almost 9 years ago
#12 Updated by Loïc Dachary almost 9 years ago
#13 Updated by Loïc Dachary almost 9 years ago
Waiting for Shipping apache config with
ubuntu@teuthology:/a/loic-2014-08-14_01:02:11-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping-testing-basic-vps$ grep 'Shipping apache config' */teuthology.log
before inspecting the actual configuration on a running machine.
#14 Updated by Loïc Dachary almost 9 years ago
- Description updated (diff)
#15 Updated by Yuri Weinstein almost 9 years ago
Loic, we had disk failed and possibly due to that suite failed (guessing), I re-started it http://pulpito.front.sepia.ceph.com/teuthology-2014-08-13_18:26:31-upgrade:firefly-x:stress-split-wip-9025-chunk-remapping---basic-vps/
#16 Updated by Loïc Dachary almost 9 years ago
It failed for the same reason. Rescheduled once more, hoping the problem has been fixed
#17 Updated by Loïc Dachary almost 9 years ago
The reason why the suite fails with
Traceback (most recent call last):
File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 27, in nested
vars.append(enter())
File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
return self.gen.next()
File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/rgw.py", line 508, in create_nonregion_pools
(remote,) = ctx.cluster.only(client).remotes.iterkeys()
is because vps.yaml creates entries in the rgw config for clients that are not defined in the role section of the file. The method loops over them and fails when it tries to acquire them
#18 Updated by Loïc Dachary almost 9 years ago
- Status changed from In Progress to Fix Under Review
- Assignee set to Loïc Dachary
#19 Updated by Loïc Dachary almost 9 years ago
the suite runs ok except for one error related to a MDSMap encoding bug that has been fixed in master today.
#20 Updated by Sage Weil almost 9 years ago
- Status changed from Fix Under Review to Resolved