Project

General

Profile

Actions
Copy link

Bug #17872

closed

all swift tests fail with '[Errno 111] Connection refused' because radosgw instance isn't started

Added by Casey Bodley over 7 years ago. Updated over 7 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

From http://qa-proxy.ceph.com/teuthology/cbodley-2016-11-11_12:15:00-rgw:verify-wip-rgw-compression-type-random---basic-mira/540939/teuthology.log

The radosgw instance is started here at time 18:42:08:

2016-11-11T18:42:08.101 INFO:tasks.rgw.client.0:Restarting daemon
2016-11-11T18:42:08.102 INFO:teuthology.orchestra.run.mira035:Running: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/client.0.log --time-stamp=yes --tool=memcheck radosgw --rgw-frontends 'civetweb port=7280' -n client.0 -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.client.0.stdout 2>&1" 
2016-11-11T18:42:08.106 INFO:tasks.rgw.client.0:Started

Swift tests start almost a minute later, at 18:43:05:

2016-11-11T18:43:05.863 INFO:teuthology.orchestra.run.mira035:Running: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" 
testAccountHead (test.functional.tests.TestAccount) ... ERROR
testContainerListing (test.functional.tests.TestAccount) ... ERROR
testContainerSerializedInfo (test.functional.tests.TestAccount) ... ERROR
testContainersOrderedByName (test.functional.tests.TestAccount) ... ERROR
testInvalidAuthToken (test.functional.tests.TestAccount) ... ERROR
testInvalidUTF8Path (test.functional.tests.TestAccount) ... ERROR
testLastContainerMarker (test.functional.tests.TestAccount) ... ERROR
testListingLimit (test.functional.tests.TestAccount) ... ERROR
testMarkerLimitContainerList (test.functional.tests.TestAccount) ... ERROR
testPUT (test.functional.tests.TestAccount) ... ERROR
testGetRequest (test.functional.tests.TestAccountNoContainers) ... ERROR
testGetRequest (test.functional.tests.TestAccountNoContainersUTF8) ... ERROR
testAccountHead (test.functional.tests.TestAccountUTF8) ... ERROR

...

======================================================================
ERROR: testAccountHead (test.functional.tests.TestAccount)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp
    cls.env.setUp()
  File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 131, in setUp
    cls.conn.authenticate()
  File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 131, in authenticate
    connection.request('GET', path, '', headers)
  File "/usr/lib64/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request
    self.endheaders(body)
  File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders
    self._send_output(message_body)
  File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output
    self.send(msg)
  File "/usr/lib64/python2.7/httplib.py", line 826, in send
    self.connect()
  File "/usr/lib64/python2.7/httplib.py", line 807, in connect
    self.timeout, self.source_address)
  File "/usr/lib64/python2.7/socket.py", line 571, in create_connection
    raise err
error: [Errno 111] Connection refused

...

The contents of the radosgw log rgw.client.0.log indicate that it didn't actually start until 18:43:09, over 60 seconds after it was invoked (it also doesn't include debug_rgw=20 or debug_ms=1 as configured):

2016-11-11 18:43:09.042131 8c83c80  0 framework: civetweb
2016-11-11 18:43:09.048401 8c83c80  0 framework conf key: port, val: 7280
2016-11-11 18:43:09.152807 8c83c80  0 ceph version 11.0.2-1369-g9768205 (976820570198a98d79affb12b1c9fa5c1bb8d7bf), process memcheck-amd64-, pid 9109
Actions
Copy link
 #1

Updated by Casey Bodley over 7 years ago

Later in that teuthology.log:

2016-11-11T18:43:12.196 INFO:teuthology.misc:Shutting down rgw daemons...
2016-11-11T18:43:12.196 DEBUG:tasks.rgw.client.0:waiting for process to exit
2016-11-11T18:43:12.196 INFO:teuthology.orchestra.run:waiting for 300
2016-11-11T18:43:12.246 INFO:tasks.rgw.client.0.mira035.stderr:daemon-helper: command crashed with signal 9
2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- WARNING: unhandled syscall: 306
2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- You may be able to write your own handler.
2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:12.583 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- WARNING: unhandled syscall: 306
2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- You may be able to write your own handler.
2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- WARNING: unhandled syscall: 306
2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- You may be able to write your own handler.
2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- WARNING: unhandled syscall: 306
2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- You may be able to write your own handler.
2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- WARNING: unhandled syscall: 306
2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- You may be able to write your own handler.
2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:14.287 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:14.287 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- WARNING: unhandled syscall: 306
2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- You may be able to write your own handler.
2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- Read the file README_MISSING_SYSCALL_OR_IOCTL.
2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- Nevertheless we consider this a bug.  Please report
2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- it at http://valgrind.org/support/bug_reports.html.
2016-11-11T18:43:18.197 INFO:tasks.rgw.client.0:Stopped
Actions
Copy link
 #2

Updated by Yehuda Sadeh over 7 years ago

  • Status changed from New to Resolved

resolved by changes to the qa suite

Actions
Copy link

Also available in: Atom PDF