Project

General

Profile

Actions
Copy link

Bug #11071

closed

rgw: mon thrashing makes pool creation time out

Added by Yuri Weinstein about 9 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-03-08_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/
Jobs: 795838, 795875
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-08_17:05:01-upgrade:giant-x-hammer-distro-basic-multi/795838/

2015-03-08T18:47:50.905 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum
2015-03-08T18:47:50.906 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.21:6789 mon_status'
2015-03-08T18:47:51.198 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.29:6789 mon_status'
2015-03-08T18:47:51.495 INFO:teuthology.orchestra.run.plana57:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.132.29:6790 mon_status'
2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b']
2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: []
2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b
2015-03-08T18:47:51.792 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b
2015-03-08T18:47:51.793 DEBUG:tasks.ceph.mon.b:waiting for process to exit
2015-03-08T18:47:57.565 INFO:teuthology.orchestra.run.plana65.stderr:testAccountHead (test.functional.tests.TestAccount) ... ERROR

and then:

2015-03-08T18:49:24.510 INFO:teuthology.orchestra.run.plana65.stderr:
2015-03-08T18:49:24.511 INFO:teuthology.orchestra.run.plana65.stderr:======================================================================
2015-03-08T18:49:24.512 INFO:teuthology.orchestra.run.plana65.stderr:ERROR: testAccountHead (test.functional.tests.TestAccount)
2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:----------------------------------------------------------------------
2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:Traceback (most recent call last):
2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
2015-03-08T18:49:24.513 INFO:teuthology.orchestra.run.plana65.stderr:    cls.env.setUp()
2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr:  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 140, in setUp
2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr:    raise ResponseError(cls.conn.response)
2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr:ResponseError: 500: Internal Server Error
2015-03-08T18:49:24.514 INFO:teuthology.orchestra.run.plana65.stderr:
2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr:----------------------------------------------------------------------
2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr:Ran 137 tests in 116.990s
2015-03-08T18:49:24.515 INFO:teuthology.orchestra.run.plana65.stderr:
2015-03-08T18:49:24.516 INFO:teuthology.orchestra.run.plana65.stderr:FAILED (errors=1)
2015-03-08T18:49:24.529 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/swift.py", line 176, in run_tests
    args=args,
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 137, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on plana65 with status 1: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.1.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'"

Related issues 2 (0 open — 2 closed)

Has duplicate rgw - Bug #10925: "(test.functional.tests.TestAccount) ... FAIL" in upgrade:giant-x-hammer-distro-basic-multi runDuplicate02/21/2015

Actions
Has duplicate rgw - Bug #10674: "testAccountHead (test.functional.tests.TestAccount) ... ERROR" in upgrade:firefly-x-next-distro-basic-multiDuplicate01/28/2015

Actions
Actions
Copy link
 #2

Updated by Samuel Just about 9 years ago

  • Project changed from Ceph to rgw
Actions
Copy link
 #3

Updated by Yuri Weinstein about 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-03-11_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/
Job: 801327
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-03-11_18:13:01-upgrade:firefly-x-giant-distro-basic-multi/801327/

2015-03-13T02:56:59.335 INFO:teuthology.orchestra.run.plana22:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph -m 10.214.131.18:6789 mon_status'
2015-03-13T02:56:59.579 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors
2015-03-13T02:57:03.583 INFO:teuthology.orchestra.run.plana11.stderr:testAccountHead (test.functional.tests.TestAccount) ... ERROR
2015-03-13T02:57:07.147 INFO:teuthology.orchestra.run.plana11.stderr:testContainerListing (test.functional.tests.TestAccount) ... FAIL
2015-03-13T02:57:16.745 INFO:teuthology.orchestra.run.plana11.stderr:testContainerSerializedInfo (test.functional.tests.TestAccount) ... ok
2015-03-13T02:57:16.795 INFO:teuthology.orchestra.run.plana11.stderr:testContainersOrderedByName (test.functional.tests.TestAccount) ... ok
2015-03-13T02:57:16.820 INFO:teuthology.orchestra.run.plana11.stderr:testInvalidAuthToken (test.functional.tests.TestAccount) ... ok
2015-03-13T02:57:16.822 INFO:teuthology.orchestra.run.plana11.stderr:testInvalidUTF8Path (test.functional.tests.TestAccount) ... ok
2015-03-13T02:57:16.826 INFO:teuthology.orchestra.run.plana11.stderr:testLastContainerMarker (test.functional.tests.TestAccount) ... FAIL
Actions
Copy link
 #5

Updated by Sage Weil about 9 years ago

mon thrashing can make pool creation time out. need to create pools before starting rgw tests

Actions
Copy link
 #6

Updated by Sage Weil about 9 years ago

  • Status changed from New to 12
Actions
Copy link
 #7

Updated by Sage Weil about 9 years ago

  • Subject changed from "(test.functional.tests.TestAccount) ... ERROR" in upgrade:giant-x-hammer-distro-basic-multi run to rgw: mon thrashing makes pool creation time out
Actions
Copy link
 #8

Updated by Yehuda Sadeh about 9 years ago

  • Assignee set to Alfredo Deza
Actions
Copy link
 #9

Updated by Sage Weil almost 9 years ago

  • Priority changed from Urgent to High
Actions
Copy link
 #10

Updated by Sage Weil over 8 years ago

  • Assignee deleted (Alfredo Deza)
Actions
Copy link
 #11

Updated by Orit Wasserman almost 8 years ago

  • Priority changed from High to Normal
Actions
Copy link
 #12

Updated by Patrick Donnelly over 4 years ago

  • Status changed from 12 to New
Actions
Copy link
 #13

Updated by Casey Bodley about 4 years ago

  • Status changed from New to Closed
Actions
Copy link

Also available in: Atom PDF