Project

General

Profile

Actions
Copy link

Bug #7837

closed

s3tests test_object_metadata_replaced_on_put fails on an erasure coded pool

Added by Josh Durgin about 10 years ago. Updated about 10 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Samuel Just
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Using this configuration:

roles:
- [mon.a, osd.0, osd.1, client.0]
tasks:
- chef:
- install:
    branch: wip-7676-firefly
- ceph:
- rgw:
    ec-data-pool: true
    client.0:
- s3tests:
    client.0:
      server: client.0

s3tests fails with:

2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: ======================================================================
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: FAIL: s3tests.functional.test_s3.test_object_metadata_replaced_on_put
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: ----------------------------------------------------------------------
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Traceback (most recent call last):
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]:   File "/home/ubuntu/cephtest/s3-tests/virtualenv/local/lib/python2.7/site-packages/nose/case.
py", line 197, in runTest
2014-03-24T18:11:15.941 INFO:teuthology.orchestra.run.err:[10.214.132.112]:     self.test(*self.arg)
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]:   File "/home/ubuntu/cephtest/s3-tests/s3tests/functional/test_s3.py", line 1026, in test_obje
ct_metadata_replaced_on_put
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]:     assert got is None, "did not expect to see metadata: %r" % got
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AssertionError: did not expect to see metadata: u'bar'
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: -------------------- >> begin captured logging << --------------------
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/
2014-03-24T18:11:15.942 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data:
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {}
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {}
2014-03-24T18:11:15.943 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign:
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/
2014-03-24T18:11:15.944 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature:
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:u7Ck3/fDSu0gpf7pjzpL4GjByQw=
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data:
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {'Content-Length': '3', 'Expect': '100-Continue', 'Content-Type': 'application/octet-stream', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'x-amz-meta-meta1': 'bar', 'User-Agent': 'Boto/2.27.0 Python/2.7.3 Linux/3.13.0-rc3-ceph-00048-gf48db1e'}
2014-03-24T18:11:15.945 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {}
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign:
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: N7UdGUp1E+RbVvZSTy1R8g==
2014-03-24T18:11:15.946 INFO:teuthology.orchestra.run.err:[10.214.132.112]: application/octet-stream
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: x-amz-meta-meta1:bar
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature:
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:XINeEewEUapC/kk9DAB9puJ8TZs=
2014-03-24T18:11:15.947 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: PUT
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data:
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {'Content-Length': '3', 'Content-MD5': 'N7UdGUp1E+RbVvZSTy1R8g==', 'Content-Type': 'application/octet-stream', 'Expect': '100-Continue', 'User-Agent': 'Boto/2.27.0 Python/2.7.3 Linux/3.13.0-rc3-ceph-00048-gf48db1e'}
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280
2014-03-24T18:11:15.948 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {}
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign:
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: PUT
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: N7UdGUp1E+RbVvZSTy1R8g==
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: application/octet-stream
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT
2014-03-24T18:11:15.949 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature:
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:exGVUc8M6RzNH4qUKp/9kPOmJCY=
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: auth_path=/test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Method: HEAD
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Path: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.950 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Data:
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Headers: {}
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Host: localhost:7280
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Port: 7280
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Params: {}
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Token: None
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: StringToSign:
2014-03-24T18:11:15.951 INFO:teuthology.orchestra.run.err:[10.214.132.112]: HEAD
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: 
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: Tue, 25 Mar 2014 01:10:32 GMT
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: /test-client.0-h3ceuuliyahlyse-103/foo
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: boto: DEBUG: Signature:
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: AWS GFDWTMNJFKKKABQWFSLX:JPv1exi9RHUK0o6iNB0hY0Tk9L8=
2014-03-24T18:11:15.952 INFO:teuthology.orchestra.run.err:[10.214.132.112]: --------------------- >> end captured logging << ---------------------

This test passes with a replicated data pool.

Actions
Copy link
 #1

Updated by Yehuda Sadeh about 10 years ago

  • Assignee changed from Yehuda Sadeh to Samuel Just

This looks like an EC related regression, was working before.

The test first creates an object + xattrs. Then overwrites it completely with new xattrs. It is expected that at this point we don't hold the old xattrs. This is the relevant rados operation that overrides the object. Note the delete op.

2014-03-25 11:48:20.354168 7f0df0ff9700  1 -- 192.168.106.226:0/1016231 --> 192.168.106.226:6814/13647 -- osd_op(client.4198.0:153 default.4198.1_foo [create 0~0,delete,setxattr user.rgw.idtag (15),writefull 0~3,setxattr user.rgw.manifest (407),setxattr user.rgw.acl (133),setxattr user.rgw.content_type (25),setxattr user.rgw.etag (33)] 3.ee3a09ea ondisk+write e49) v4 -- ?+0 0xd09830 con 0xcd74d0
2014-03-25 11:48:20.389305 7f0e024d7700  1 -- 192.168.106.226:0/1016231 <== osd.1 192.168.106.226:6814/13647 15 ==== osd_op_reply(153 default.4198.1_foo [create 0~0,delete,setxattr (15),writefull 0~3,setxattr (407),setxattr (133),setxattr (25),setxattr (33)] v49'2 uv2 ondisk = 0) v6 ==== 479+0+0 (41840841 0 0) 0xd09830 con 0xcd74d0

Actions
Copy link
 #2

Updated by Josh Durgin about 10 years ago

I've got osd and rgw logs on metropolis:/home/joshd/teuthology/archive/rgw-ec-pool-7837

Actions
Copy link
 #3

Updated by Samuel Just about 10 years ago

  • Status changed from New to 12

looking

Actions
Copy link
 #4

Updated by Samuel Just about 10 years ago

wip-7837

Actions
Copy link
 #5

Updated by Samuel Just about 10 years ago

  • Status changed from 12 to 7
Actions
Copy link
 #6

Updated by Sage Weil about 10 years ago

  • Status changed from 7 to Resolved
Actions
Copy link

Also available in: Atom PDF