Actions
Bug #17872
closedall swift tests fail with '[Errno 111] Connection refused' because radosgw instance isn't started
Status:
Resolved
Priority:
High
Assignee:
-
Target version:
-
% Done:
0%
Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
The radosgw instance is started here at time 18:42:08:
2016-11-11T18:42:08.101 INFO:tasks.rgw.client.0:Restarting daemon 2016-11-11T18:42:08.102 INFO:teuthology.orchestra.run.mira035:Running: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term valgrind --trace-children=no --child-silent-after-fork=yes --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/client.0.log --time-stamp=yes --tool=memcheck radosgw --rgw-frontends 'civetweb port=7280' -n client.0 -k /etc/ceph/ceph.client.0.keyring --log-file /var/log/ceph/rgw.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.client.0.stdout 2>&1" 2016-11-11T18:42:08.106 INFO:tasks.rgw.client.0:Started
Swift tests start almost a minute later, at 18:43:05:
2016-11-11T18:43:05.863 INFO:teuthology.orchestra.run.mira035:Running: "SWIFT_TEST_CONFIG_FILE=/home/ubuntu/cephtest/archive/testswift.client.0.conf /home/ubuntu/cephtest/swift/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/swift/test/functional -v -a '!fails_on_rgw'" testAccountHead (test.functional.tests.TestAccount) ... ERROR testContainerListing (test.functional.tests.TestAccount) ... ERROR testContainerSerializedInfo (test.functional.tests.TestAccount) ... ERROR testContainersOrderedByName (test.functional.tests.TestAccount) ... ERROR testInvalidAuthToken (test.functional.tests.TestAccount) ... ERROR testInvalidUTF8Path (test.functional.tests.TestAccount) ... ERROR testLastContainerMarker (test.functional.tests.TestAccount) ... ERROR testListingLimit (test.functional.tests.TestAccount) ... ERROR testMarkerLimitContainerList (test.functional.tests.TestAccount) ... ERROR testPUT (test.functional.tests.TestAccount) ... ERROR testGetRequest (test.functional.tests.TestAccountNoContainers) ... ERROR testGetRequest (test.functional.tests.TestAccountNoContainersUTF8) ... ERROR testAccountHead (test.functional.tests.TestAccountUTF8) ... ERROR ... ====================================================================== ERROR: testAccountHead (test.functional.tests.TestAccount) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 104, in setUp cls.env.setUp() File "/home/ubuntu/cephtest/swift/test/functional/tests.py", line 131, in setUp cls.conn.authenticate() File "/home/ubuntu/cephtest/swift/test/functional/swift.py", line 131, in authenticate connection.request('GET', path, '', headers) File "/usr/lib64/python2.7/httplib.py", line 1017, in request self._send_request(method, url, body, headers) File "/usr/lib64/python2.7/httplib.py", line 1051, in _send_request self.endheaders(body) File "/usr/lib64/python2.7/httplib.py", line 1013, in endheaders self._send_output(message_body) File "/usr/lib64/python2.7/httplib.py", line 864, in _send_output self.send(msg) File "/usr/lib64/python2.7/httplib.py", line 826, in send self.connect() File "/usr/lib64/python2.7/httplib.py", line 807, in connect self.timeout, self.source_address) File "/usr/lib64/python2.7/socket.py", line 571, in create_connection raise err error: [Errno 111] Connection refused ...
The contents of the radosgw log rgw.client.0.log indicate that it didn't actually start until 18:43:09, over 60 seconds after it was invoked (it also doesn't include debug_rgw=20 or debug_ms=1 as configured):
2016-11-11 18:43:09.042131 8c83c80 0 framework: civetweb 2016-11-11 18:43:09.048401 8c83c80 0 framework conf key: port, val: 7280 2016-11-11 18:43:09.152807 8c83c80 0 ceph version 11.0.2-1369-g9768205 (976820570198a98d79affb12b1c9fa5c1bb8d7bf), process memcheck-amd64-, pid 9109
Updated by Casey Bodley over 7 years ago
Later in that teuthology.log:
2016-11-11T18:43:12.196 INFO:teuthology.misc:Shutting down rgw daemons... 2016-11-11T18:43:12.196 DEBUG:tasks.rgw.client.0:waiting for process to exit 2016-11-11T18:43:12.196 INFO:teuthology.orchestra.run:waiting for 300 2016-11-11T18:43:12.246 INFO:tasks.rgw.client.0.mira035.stderr:daemon-helper: command crashed with signal 9 2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- WARNING: unhandled syscall: 306 2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- You may be able to write your own handler. 2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:12.535 INFO:tasks.ceph.osd.5.mira028.stderr:--00:00:10:06.610 15962-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:12.583 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- WARNING: unhandled syscall: 306 2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- You may be able to write your own handler. 2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:12.584 INFO:tasks.ceph.osd.4.mira028.stderr:--00:00:10:06.662 15961-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- WARNING: unhandled syscall: 306 2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- You may be able to write your own handler. 2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:12.727 INFO:tasks.ceph.osd.3.mira028.stderr:--00:00:10:06.808 15960-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- WARNING: unhandled syscall: 306 2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- You may be able to write your own handler. 2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:13.357 INFO:tasks.ceph.osd.2.mira035.stderr:--00:00:10:07.337 23475-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- WARNING: unhandled syscall: 306 2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- You may be able to write your own handler. 2016-11-11T18:43:14.286 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:14.287 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:14.287 INFO:tasks.ceph.osd.1.mira035.stderr:--00:00:10:08.272 23474-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- WARNING: unhandled syscall: 306 2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- You may be able to write your own handler. 2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- Read the file README_MISSING_SYSCALL_OR_IOCTL. 2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- Nevertheless we consider this a bug. Please report 2016-11-11T18:43:15.380 INFO:tasks.ceph.osd.0.mira035.stderr:--00:00:10:09.368 23473-- it at http://valgrind.org/support/bug_reports.html. 2016-11-11T18:43:18.197 INFO:tasks.rgw.client.0:Stopped
Updated by Yehuda Sadeh over 7 years ago
- Status changed from New to Resolved
resolved by changes to the qa suite
Actions