Actions
Bug #16727
opens3tests-test-readwrite failed in rados run
% Done:
0%
Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Run: http://pulpito.ceph.com/yuriw-2016-07-15_16:58:50-rados-wip-yuri-testing2_2016_7_15-distro-basic-smithi/
Job: 316606
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-07-15_16:58:50-rados-wip-yuri-testing2_2016_7_15-distro-basic-smithi/316606/teuthology.log
2016-07-15T21:56:33.934 INFO:teuthology.orchestra.run.smithi057:Running: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops' 2016-07-15T21:56:33.938 INFO:teuthology.orchestra.run.smithi057.stderr:Traceback (most recent call last): 2016-07-15T21:56:33.938 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite", line 9, in <module> 2016-07-15T21:56:33.938 INFO:teuthology.orchestra.run.smithi057.stderr: load_entry_point('s3tests', 'console_scripts', 's3tests-test-readwrite')() 2016-07-15T21:56:33.938 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 256, in main 2016-07-15T21:56:33.939 INFO:teuthology.orchestra.run.smithi057.stderr: trace=temp_dict['error']['traceback']) 2016-07-15T21:56:33.939 INFO:teuthology.orchestra.run.smithi057.stderr:Exception: exception: 2016-07-15T21:56:33.939 INFO:teuthology.orchestra.run.smithi057.stderr: S3ResponseError: 403 Forbidden 2016-07-15T21:56:33.939 INFO:teuthology.orchestra.run.smithi057.stderr:<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><BucketName>rwtest</BucketName><RequestId>tx000000000000000000069-005789bd55-101a-default</RequestId><HostId>101a-default-default</HostId></Error> 2016-07-15T21:56:33.940 INFO:teuthology.orchestra.run.smithi057.stderr: Traceback (most recent call last): 2016-07-15T21:56:33.940 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/s3tests/readwrite.py", line 85, in writer 2016-07-15T21:56:33.940 INFO:teuthology.orchestra.run.smithi057.stderr: key.set_contents_from_file(fp) 2016-07-15T21:56:33.940 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 1293, in set_contents_from_file 2016-07-15T21:56:33.941 INFO:teuthology.orchestra.run.smithi057.stderr: chunked_transfer=chunked_transfer, size=size) 2016-07-15T21:56:33.941 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 750, in send_file 2016-07-15T21:56:33.941 INFO:teuthology.orchestra.run.smithi057.stderr: chunked_transfer=chunked_transfer, size=size) 2016-07-15T21:56:33.941 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 951, in _send_file_internal 2016-07-15T21:56:33.942 INFO:teuthology.orchestra.run.smithi057.stderr: query_args=query_args 2016-07-15T21:56:33.942 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 668, in make_request 2016-07-15T21:56:33.942 INFO:teuthology.orchestra.run.smithi057.stderr: retry_handler=retry_handler 2016-07-15T21:56:33.942 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1071, in make_request 2016-07-15T21:56:33.943 INFO:teuthology.orchestra.run.smithi057.stderr: retry_handler=retry_handler) 2016-07-15T21:56:33.943 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 940, in _mexe 2016-07-15T21:56:33.943 INFO:teuthology.orchestra.run.smithi057.stderr: request.body, request.headers) 2016-07-15T21:56:33.943 INFO:teuthology.orchestra.run.smithi057.stderr: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/key.py", line 884, in sender 2016-07-15T21:56:33.944 INFO:teuthology.orchestra.run.smithi057.stderr: response.status, response.reason, body) 2016-07-15T21:56:33.944 INFO:teuthology.orchestra.run.smithi057.stderr:S3ResponseError: S3ResponseError: 403 Forbidden 2016-07-15T21:56:33.944 INFO:teuthology.orchestra.run.smithi057.stderr:<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><BucketName>rwtest</BucketName><RequestId>tx000000000000000000069-005789bd55-101a-default</RequestId><HostId>101a-default-default</HostId></Error> 2016-07-15T21:56:33.944 INFO:teuthology.orchestra.run.smithi057.stderr: 2016-07-15T21:56:33.945 INFO:tasks.ceph.osd.2.smithi057.stderr:2016-07-16 04:56:34.885234 7f0444ee5700 -1 received signal: Hangup from PID: 19253 task name: /usr/bin/python UID: 0 2016-07-15T21:56:33.945 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/var/lib/teuthworker/src/ceph-qa-suite_master/tasks/s3readwrite.py", line 229, in run_tests stdin=conf, File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 196, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 394, in run r.wait() File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 121, in wait label=self.label) CommandFailedError: Command failed on smithi057 with status 1: '/home/ubuntu/cephtest/s3-tests/virtualenv/bin/s3tests-test-readwrite'
Updated by Matt Benjamin over 7 years ago
The second log doesn't return 403/forbidden (so looks like different issue)?
Updated by David Zafman over 7 years ago
Also, seen on a run based on Jewel:
http://pulpito.ceph.com/dzafman-2016-10-05_15:06:22-rados-wip-zafman-testing-jewel-distro-basic-smithi/456535
http://pulpito.ceph.com/dzafman-2016-10-05_15:06:22-rados-wip-zafman-testing-jewel-distro-basic-smithi/456626
http://pulpito.ceph.com/dzafman-2016-10-05_15:06:22-rados-wip-zafman-testing-jewel-distro-basic-smithi/456765
Actions