Actions
Bug #11071
closedrgw: mon thrashing makes pool creation time out
Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/teuthology-2015-03-08_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/
Jobs: 795838, 795875
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-08_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/795838/
2015-03-08T18:47:50.905 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2015-03-08T18:47:50.906 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.21:6789 mon_status' 2015-03-08T18:47:51.198 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.29:6789 mon_status' 2015-03-08T18:47:51.495 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.29:6790 mon_status' 2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2015-03-08T18:47:51.793 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2015-03-08T18:47:57.565 INFO:teuthology.orchestra.run.plana65.stderr:testAccountHead (test.functional.tests.TestAccount) ... ERROR
and then:
2015-03-08T18:49:24.510 INFO:teuthology.orchestra.run.plana65.stderr: 2015-03-08T18:49:24.511 INFO:teuthology.orchestra.run.plana65.stderr:====================================================================== 2015-03-08T18:49:24.512 INFO:teuthology.orchestra.run.plana65.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount) 2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:---------------------------------------------------------------------- 2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:Traceback (most recent call last): 2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp 2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr: cls.env.setUp() 2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr: File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp 2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr: raise ResponseError(cls.conn.response) 2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr:ResponseError: 500: Internal Server Error 2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr: 2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr:---------------------------------------------------------------------- 2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr:Ran 137 tests in 116.990s 2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr: 2015-03-08T18:49:24.516 INFO:teuthology.orchestra.run.plana65.stderr:FAILED (errors=1) 2015-03-08T18:49:24.529 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 176, in run_tests args=args, File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 137, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run r.wait() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait label=self.label) CommandFailedError: Command failed on plana65 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.1.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"
Updated by Yuri Weinstein about 9 years ago
Updated by Yuri Weinstein about 9 years ago
Run: http://pulpito.ceph.com/teuthology-2015-03-11_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/
Job: 801327
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-11_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/801327/
2015-03-13T02:56:59.335 INFO:teuthology.orchestra.run.plana22:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.131.18:6789 mon_status' 2015-03-13T02:56:59.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2015-03-13T02:57:03.583 INFO:teuthology.orchestra.run.plana11.stderr:testAccountHead (test.functional.tests.TestAccount) ... ERROR 2015-03-13T02:57:07.147 INFO:teuthology.orchestra.run.plana11.stderr:testContainerListing (test.functional.tests.TestAccount) ... FAIL 2015-03-13T02:57:16.745 INFO:teuthology.orchestra.run.plana11.stderr:testContainerSerializedInfo (test.functional.tests.TestAccount) ... ok 2015-03-13T02:57:16.795 INFO:teuthology.orchestra.run.plana11.stderr:testContainersOrderedByName (test.functional.tests.TestAccount) ... ok 2015-03-13T02:57:16.820 INFO:teuthology.orchestra.run.plana11.stderr:testInvalidAuthToken (test.functional.tests.TestAccount) ... ok 2015-03-13T02:57:16.822 INFO:teuthology.orchestra.run.plana11.stderr:testInvalidUTF8Path (test.functional.tests.TestAccount) ... ok 2015-03-13T02:57:16.826 INFO:teuthology.orchestra.run.plana11.stderr:testLastContainerMarker (test.functional.tests.TestAccount) ... FAIL
Updated by Sage Weil about 9 years ago
mon thrashing can make pool creation time out. need to create pools before starting rgw tests
Updated by Sage Weil about 9 years ago
- Subject changed from "(test.functional.tests.TestAccount) ... ERROR" in upgrade:giant-x-hammer-distro-basic-multi run to rgw: mon thrashing makes pool creation time out
Updated by Orit Wasserman almost 8 years ago
- Priority changed from High to Normal
Actions