Bug #12192: s3tests.functional failed in upgrade:hammer-hammer---basic-vps run - rgw - Ceph

Bug #12192

s3tests.functional failed in upgrade:hammer-hammer---basic-vps run

Added by Yuri Weinstein over 8 years ago. Updated almost 4 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Q/A

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

upgrade/hammer

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Run: http://pulpito.ceph.com/teuthology-2015-06-30_14:04:21-upgrade:hammer-hammer---basic-vps/
Job: 955580
Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-06-30_14:04:21-upgrade:hammer-hammer---basic-vps/955580/teuthology.log

Maybe an issue with the tests?

Command:

teuthology-suite -v -c hammer -m vps -s upgrade/hammer --filter "upgrade:hammer/newer/{0-cluster/start.yaml 1-install/v0.94.2.yaml 2-workload/s3tests.yaml 3-upgrade-sequence/upgrade-mon-osd-mds.yaml 4-final/{monthrash.yaml osdthrash.yaml testrgw.yaml} distros/ubuntu_12.04.yaml}" --suite-branch wip-11570 -p 90

2015-06-30T14:22:23.939 INFO:teuthology.orchestra.run.vpm159.stderr:s3tests.fuzz.test.test_fuzzer.test_null_child ... ok
2015-06-30T14:22:23.947 INFO:teuthology.orchestra.run.vpm159.stderr:s3tests.fuzz.test.test_fuzzer.test_weighted_set ... ok
2015-06-30T14:22:23.947 INFO:teuthology.orchestra.run.vpm159.stderr:s3tests.fuzz.test.test_fuzzer.test_header_presence ... ok
2015-06-30T14:22:23.948 INFO:teuthology.orchestra.run.vpm159.stderr:s3tests.fuzz.test.test_fuzzer.test_duplicate_header ... ok
2015-06-30T14:22:23.951 INFO:teuthology.orchestra.run.vpm159.stderr:s3tests.fuzz.test.test_fuzzer.test_expand_headers ... ok
2015-06-30T14:22:23.951 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.951 INFO:teuthology.orchestra.run.vpm159.stderr:======================================================================
2015-06-30T14:22:23.951 INFO:teuthology.orchestra.run.vpm159.stderr:ERROR: test suite for <module 's3tests.functional' from '/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py'>
2015-06-30T14:22:23.951 INFO:teuthology.orchestra.run.vpm159.stderr:----------------------------------------------------------------------
2015-06-30T14:22:23.952 INFO:teuthology.orchestra.run.vpm159.stderr:Traceback (most recent call last):
2015-06-30T14:22:23.952 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/suite.py", line 209, in run
2015-06-30T14:22:23.952 INFO:teuthology.orchestra.run.vpm159.stderr:    self.setUp()
2015-06-30T14:22:23.952 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/suite.py", line 292, in setUp
2015-06-30T14:22:23.952 INFO:teuthology.orchestra.run.vpm159.stderr:    self.setupContext(ancestor)
2015-06-30T14:22:23.953 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/suite.py", line 315, in setupContext
2015-06-30T14:22:23.953 INFO:teuthology.orchestra.run.vpm159.stderr:    try_run(context, names)
2015-06-30T14:22:23.953 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/util.py", line 470, in try_run
2015-06-30T14:22:23.953 INFO:teuthology.orchestra.run.vpm159.stderr:    return func()
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 329, in setup
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:    nuke_prefixed_buckets(prefix=prefix)
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 87, in nuke_prefixed_buckets
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:    nuke_prefixed_buckets_on_conn(prefix, name, conn)
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/__init__.py", line 62, in nuke_prefixed_buckets_on_conn
2015-06-30T14:22:23.954 INFO:teuthology.orchestra.run.vpm159.stderr:    for bucket in conn.get_all_buckets():
2015-06-30T14:22:23.955 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 436, in get_all_buckets
2015-06-30T14:22:23.955 INFO:teuthology.orchestra.run.vpm159.stderr:    response = self.make_request('GET', headers=headers)
2015-06-30T14:22:23.955 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/s3/connection.py", line 664, in make_request
2015-06-30T14:22:23.955 INFO:teuthology.orchestra.run.vpm159.stderr:    retry_handler=retry_handler
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1070, in make_request
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:    retry_handler=retry_handler)
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:  File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/boto/connection.py", line 1027, in _mexe
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:    raise BotoServerError(response.status, response.reason, body)
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:BotoServerError: BotoServerError: 500 Internal Server Error
2015-06-30T14:22:23.956 INFO:teuthology.orchestra.run.vpm159.stderr:None
2015-06-30T14:22:23.957 INFO:teuthology.orchestra.run.vpm159.stderr:-------------------- >> begin captured stdout << ---------------------
2015-06-30T14:22:23.957 INFO:teuthology.orchestra.run.vpm159.stderr:Deleting buckets on alt
2015-06-30T14:22:23.957 INFO:teuthology.orchestra.run.vpm159.stderr:Cleaning buckets from connection alt prefix 'test-client.0-fen3h9hwktk9gpy-'.
2015-06-30T14:22:23.957 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.957 INFO:teuthology.orchestra.run.vpm159.stderr:--------------------- >> end captured stdout << ----------------------
2015-06-30T14:22:23.958 INFO:teuthology.orchestra.run.vpm159.stderr:-------------------- >> begin captured logging << --------------------
2015-06-30T14:22:23.958 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Using access key provided by client.
2015-06-30T14:22:23.958 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Using secret key provided by client.
2015-06-30T14:22:23.958 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Using access key provided by client.
2015-06-30T14:22:23.958 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Using secret key provided by client.
2015-06-30T14:22:23.959 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: path=/
2015-06-30T14:22:23.959 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: auth_path=/
2015-06-30T14:22:23.959 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Method: GET
2015-06-30T14:22:23.959 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Path: /
2015-06-30T14:22:23.959 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Data:
2015-06-30T14:22:23.960 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Headers: {}
2015-06-30T14:22:23.960 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Host: vpm159.front.sepia.ceph.com:7280
2015-06-30T14:22:23.960 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Port: 7280
2015-06-30T14:22:23.960 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Params: {}
2015-06-30T14:22:23.960 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: establishing HTTP connection: kwargs={'port': 7280, 'timeout': 30}
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.961 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.962 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.962 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.962 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.962 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.962 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:20:46 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 0.0 seconds
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.963 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:20:46 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.964 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 1.6 seconds
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.965 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:20:48 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 1.7 seconds
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.966 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.967 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:20:50 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 7.2 seconds
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.968 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.969 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:20:57 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 15.2 seconds
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.970 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.971 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:21:12 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 9.6 seconds
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Token: None
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: StringToSign:
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:GET
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.972 INFO:teuthology.orchestra.run.vpm159.stderr:Tue, 30 Jun 2015 21:20:46 GMT
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:/
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Signature:
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Final headers: {'Date': 'Tue, 30 Jun 2015 21:20:46 GMT', 'Content-Length': 0, 'Authorization': u'AWS YQTJYABLTMLHNLWSOHDG:nQZgzfVjG87FUgec/REt9TCBEoA=', 'User-Agent': 'Boto/2.35.1 Python/2.7.3 Linux/3.2.0-58-virtual'}
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Response headers: [('date', 'Tue, 30 Jun 2015 21:21:22 GMT'), ('content-length', '538'), ('content-type', 'text/html; charset=iso-8859-1'), ('connection', 'close'), ('server', 'Apache/2.2.22 (Ubuntu) mod_fastcgi/mod_fastcgi-SNAP-0910052141')]
2015-06-30T14:22:23.973 INFO:teuthology.orchestra.run.vpm159.stderr:boto: DEBUG: Received 500 response.  Retrying in 60.0 seconds
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:--------------------- >> end captured logging << ---------------------
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:----------------------------------------------------------------------
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:Ran 29 tests in 99.141s
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:
2015-06-30T14:22:23.974 INFO:teuthology.orchestra.run.vpm159.stderr:FAILED (errors=1)
2015-06-30T14:22:23.975 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/var/lib/teuthworker/src/ceph-qa-suite_wip-11570/tasks/s3tests.py", line 347, in run_tests
    label="s3 tests against rgw" 
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/cluster.py", line 64, in run
    return [remote.run(**kwargs) for remote in remotes]
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed (s3 tests against rgw) on vpm159 with status 1: "S3TEST_CONF=/home/ubuntu/cephtest/archive/s3-tests.client.0.conf BOTO_CONFIG=/home/ubuntu/cephtest/boto.cfg /home/ubuntu/cephtest/s3-tests/virtualenv/bin/nosetests -w /home/ubuntu/cephtest/s3-tests -v -a '!fails_on_rgw'

Related issues

History

#1 Updated by Kefu Chai over 8 years ago

i am also experiencing the same failure at http://pulpito.ceph.com/ubuntu-2015-06-29_18:58:24-rados-wip-kefu-testing---basic-multi/ , for example http://qa-proxy.ceph.com/teuthology/ubuntu-2015-06-29_18:58:24-rados-wip-kefu-testing---basic-multi/954628/teuthology.log

in /a/ubuntu-2015-06-29_18:58:24-rados-wip-kefu-testing---basic-multi/954628/remote/burnupi10/apache.client.0/error.log:

[Tue Jun 30 13:03:46.357512 2015] [:warn] [pid 6190:tid 140061230585600] FastCGI: 10.214.134.32 PUT http://burnupi10.front.sepia.ceph.com:7280/rwtest/ auth AWS ZVMLPSPYMPZDKJCRQULW:d3Uip2f2J2FkLBbINrPyyQm5B80=
[Tue Jun 30 13:03:46.357621 2015] [fastcgi:error] [pid 6190:tid 140061230585600] (2)No such file or directory: [client 10.214.134.32:33867] FastCGI: failed to connect to server "/home/ubuntu/cephtest/apache/htdocs.client.0/rgw.fcgi": connect() failed

in teuthology.log, we did prepare the rgw.fcgi script:

2015-06-30T13:03:01.154 INFO:tasks.rgw:Apache is configured to use mod_fastcgi
2015-06-30T13:03:01.155 INFO:teuthology.orchestra.run.burnupi10:Running: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/apache/apache.client.0.conf'
2015-06-30T13:03:01.178 INFO:teuthology.orchestra.run.burnupi10:Running: 'python -c \'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))\' /home/ubuntu/cephtest/apache/htdocs.client.0/rgw.fcgi'
2015-06-30T13:03:01.287 INFO:teuthology.orchestra.run.burnupi10:Running: 'chmod a=rx /home/ubuntu/cephtest/apache/htdocs.client.0/rgw.fcgi'

in remote/burnupi10/log/rgw.client.0.log.gz

2015-06-30 13:03:02.055906 7fa6953f1840  0 couldn't find old data placement pools config, setting up new ones for the zone
2015-06-30 13:03:08.158967 7fa6953f1840  0 framework: fastcgi
2015-06-30 13:03:08.158980 7fa6953f1840  0 framework conf key: socket_port, val: 9000
2015-06-30 13:03:08.158983 7fa6953f1840  0 framework conf key: socket_host, val: 0.0.0.0
2015-06-30 13:03:08.158986 7fa6953f1840  0 starting handler: fastcgi
2015-06-30 13:03:09.405850 7fa6803f4700  0 -- 10.214.134.32:0/1006186 >> 10.214.134.32:6800/5235 pipe(0x7fa63400c050 sd=32 :36081 s=2 pgs=3 cs=1 l=1 c=0x7fa63400a8d0).injecting socket failure
2015-06-30 13:03:09.447368 7fa6805f6700  0 -- 10.214.134.32:0/1006186 >> 10.214.134.32:6804/5233 pipe(0x7fa668015ca0 sd=32 :55511 s=2 pgs=6 cs=1 l=1 c=0x7fa668019f40).injecting socket failure
2015-06-30 13:03:10.206469 7fa66cff9700 -1 failed to list objects pool_iterate returned r=-2
2015-06-30 13:03:10.206483 7fa66cff9700  0 ERROR: lists_keys_next(): ret=-2
2015-06-30 13:03:10.206489 7fa66cff9700  0 ERROR: sync_all_users() returned ret=-2
2015-06-30 13:03:20.046223 7fa682ffd700  0 monclient: hunting for new mon
2015-06-30 13:03:49.277115 7fa6884ca700  0 -- 10.214.134.32:0/1006186 >> 10.214.134.32:6806/5234 pipe(0x7fa6680369b0 sd=11 :0 s=1 pgs=0 cs=0 l=1 c=0x7fa668025720).fault

where -2 means -ENOENT.