Actions
Bug #7837
closeds3tests test_object_metadata_replaced_on_put fails on an erasure coded pool
% Done:
0%
Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Using this configuration:
roles: - [mon.a, osd.0, osd.1, client.0] tasks: - chef: - install: branch: wip-7676-firefly - ceph: - rgw: ec-data-pool: true client.0: - s3tests: client.0: server: client.0
s3tests fails with:
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: ====================================================================== 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: FAIL: s3tests.functional.test_s3.test_object_metadata_replaced_on_put 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: ---------------------------------------------------------------------- 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Traceback (most recent call last): 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case. py", line 197, in runTest 2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: self.test(*self.arg) 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 1026, in test_obje ct_metadata_replaced_on_put 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: assert got is None, "did not expect to see metadata: %r" % got 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AssertionError: did not expect to see metadata: u'bar' 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: -------------------- >> begin captured logging << -------------------- 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/ 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/ 2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/ 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data: 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {} 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {} 2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign: 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/ 2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature: 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:u7Ck3/fDSu0gpf7pjzpL4GjByQw= 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data: 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {'Content-Length': '3', 'Expect': '100-Continue', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'x-amz-meta-meta1': 'bar', 'User-Agent': 'Boto/2.27.0 Python/2.7.3 Linux/3.13.0-rc3-ceph-00048-gf48db1e'} 2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {} 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign: 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: N7UdGUp1E+RbVvZSTy1R8g== 2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: application/octet-stream 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: x-amz-meta-meta1:bar 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature: 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:XINeEewEUapC/kk9DAB9puJ8TZs= 2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data: 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {'Content-Length': '3', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Type': 'application/octet-stream', 'Expect': '100-Continue', 'User-Agent': 'Boto/2.27.0 Python/2.7.3 Linux/3.13.0-rc3-ceph-00048-gf48db1e'} 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280 2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {} 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign: 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: N7UdGUp1E+RbVvZSTy1R8g== 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: application/octet-stream 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT 2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature: 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:exGVUc8M6RzNH4qUKp/9kPOmJCY= 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: HEAD 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data: 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {} 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {} 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign: 2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: HEAD 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature: 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:JPv1exi9RHUK0o6iNB0hY0Tk9L8= 2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: --------------------- >> end captured logging << ---------------------
This test passes with a replicated data pool.
Updated by Yehuda Sadeh about 10 years ago
- Assignee changed from Yehuda Sadeh to Samuel Just
This looks like an EC related regression, was working before.
The test first creates an object + xattrs. Then overwrites it completely with new xattrs. It is expected that at this point we don't hold the old xattrs. This is the relevant rados operation that overrides the object. Note the delete op.
2014-03-25 11:48:20.354168 7f0df0ff9700 1 -- 192.168.106.226:0/1016231 --> 192.168.106.226:6814/13647 -- osd_op(client.4198.0:153 default.4198.1_foo [create 0~0,delete,setxattr user.rgw.idtag (15),writefull 0~3,setxattr user.rgw.manifest (407),setxattr user.rgw.acl (133),setxattr user.rgw.content_type (25),setxattr user.rgw.etag (33)] 3.ee3a09ea ondisk+write e49) v4 -- ?+0 0xd09830 con 0xcd74d0 2014-03-25 11:48:20.389305 7f0e024d7700 1 -- 192.168.106.226:0/1016231 <== osd.1 192.168.106.226:6814/13647 15 ==== osd_op_reply(153 default.4198.1_foo [create 0~0,delete,setxattr (15),writefull 0~3,setxattr (407),setxattr (133),setxattr (25),setxattr (33)] v49'2 uv2 ondisk = 0) v6 ==== 479+0+0 (41840841 0 0) 0xd09830 con 0xcd74d0
Updated by Josh Durgin about 10 years ago
I've got osd and rgw logs on metropolis:/home/joshd/teuthology/archive/rgw-ec-pool-7837
Actions