https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2014-07-29T09:38:36ZCeph rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=386222014-07-29T09:38:36ZSage Weilsage@newdream.net
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>12</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=400052014-08-26T09:24:22ZIan Colleicolle@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Luis Pabon</i></li></ul> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=411082014-09-15T06:50:58ZLuis Pabon
<ul></ul><p>Starting to look into this...</p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=427062014-10-07T12:52:14ZLuis Pabon
<ul><li><strong>% Done</strong> changed from <i>0</i> to <i>80</i></li></ul><p>I have added a test to s3-test to check for EntityTooSmall and it <strong>passes</strong> on the current code. According to AWS an S3 server can return HTTP code 400 with the error code set to EntityTooSmall, and that is what the current RadosGW does.</p>
<p>Please let me know if I am mistaken, but it seems that the code returns the correct information. I will add the new test to s3-test.</p>
<p>For your information, here is the patch I am going to be sending to s3-test:<br /><pre>
diff --git a/s3tests/functional/test_s3.py b/s3tests/functional/test_s3.py
index 44db1f1..31a0f0d 100644
--- a/s3tests/functional/test_s3.py
+++ b/s3tests/functional/test_s3.py
@@ -4113,14 +4113,12 @@ def transfer_part(bucket, mp_id, mp_keyname, i, part):
part_out = StringIO(part)
mp.upload_part_from_file(part_out, i+1)
-def generate_random(size):
+def generate_random(size, part_size=5*1024*1024):
"""
- Generate the specified number of megabytes of random data.
+ Generate the specified number random data.
(actually each MB is a repetition of the first KB)
"""
- mb = 1024 * 1024
chunk = 1024
- part_size = 5 * mb
allowed = string.ascii_letters
for x in range(0, size, part_size):
strpart = ''.join([allowed[random.randint(0, len(allowed) - 1)] for _ in xrange(chunk)])
@@ -4133,14 +4131,14 @@ def generate_random(size):
if (x == size):
return
-def _multipart_upload(bucket, s3_key_name, size, do_list=None, headers=None, metadata=None):
+def _multipart_upload(bucket, s3_key_name, size, part_size=5*1024*1024, do_list=None, headers=None, metadata=None):
"""
generate a multi-part upload for a random file of specifed size,
if requested, generate a list of the parts
return the upload descriptor
"""
upload = bucket.initiate_multipart_upload(s3_key_name, headers=headers, metadata=metadata)
- for i, part in enumerate(generate_random(size)):
+ for i, part in enumerate(generate_random(size, part_size)):
transfer_part(bucket, upload.id, upload.key_name, i, part)
if do_list is not None:
@@ -4196,6 +4194,19 @@ def test_multipart_upload_multiple_sizes():
@attr(resource='object')
@attr(method='put')
+@attr(operation='check failure on multiple multi-part upload with size too small')
+@attr(assertion='fails 400')
+def test_multipart_upload_size_too_small():
+ bucket = get_new_bucket()
+ key="mymultipart"
+ upload = _multipart_upload(bucket, key, 100 * 1024, part_size=10*1024)
+ e = assert_raises(boto.exception.S3ResponseError, upload.complete_upload)
+ eq(e.status, 400)
+ eq(e.error_code, u'EntityTooSmall')
+
+@attr(resource='object')
+@attr(method='put')
@attr(operation='check contents of multi-part upload')
@attr(assertion='successful')
def test_multipart_upload_contents():
</pre></p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=427072014-10-07T12:56:39ZYehuda Sadehyehuda@redhat.com
<ul></ul><p>Maybe the problem is that we don't send the xml body with the appropriate error?</p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=427492014-10-07T17:30:34ZLuis Pabon
<ul></ul><p>Here is the response from the gateway:<br /><pre>
======================================================================
ERROR: s3tests.functional.test_s3.test_multipart_upload_size_too_small
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/lpabon/git/ceph/s3-tests/virtualenv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/lpabon/git/ceph/s3-tests/s3tests/functional/test_s3.py", line 4203, in test_multipart_upload_size_too_small
upload.complete_upload()
File "/home/lpabon/git/ceph/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/multipart.py", line 319, in complete_upload
self.id, xml)
File "/home/lpabon/git/ceph/s3-tests/virtualenv/lib/python2.7/site-packages/boto/s3/bucket.py", line 1806, in complete_multipart_upload
response.status, response.reason, body)
S3ResponseError: S3ResponseError: 400 Bad Request
<?xml version="1.0" encoding="UTF-8"?><Error><Code>EntityTooSmall</Code></Error>
</pre></p>
<p>According to <a class="external" href="http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html">http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html</a> the error above seems to be fine. It can also be parsed correctly by boto.</p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=427502014-10-07T17:48:02ZJosh Durgin
<ul></ul><p>The issue was reported on firefly - does it have the same behavior as master, or is there something that should be backported?</p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=427582014-10-07T22:01:48ZLuis Pabon
<ul></ul><p>Josh Durgin wrote:</p>
<blockquote>
<p>The issue was reported on firefly - does it have the same behavior as master, or is there something that should be backported?</p>
</blockquote>
<p>Good question, I'll take a look.</p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=429912014-10-13T07:22:44ZLuis Pabon
<ul></ul><p>I have submitted the following patches:</p>
<p>Update s3-tests with the new small size multipart tests:<br /><a class="external" href="https://github.com/ceph/s3-tests/pull/21">https://github.com/ceph/s3-tests/pull/21</a></p>
<p>Update vstart.sh to setup the users for s3-test in ceph master branch:<br /><a class="external" href="https://github.com/ceph/ceph/pull/2712">https://github.com/ceph/ceph/pull/2712</a></p> rgw - Bug #8766: multipart minimum size error should be EntityTooSmallhttps://tracker.ceph.com/issues/8766?journal_id=440692014-11-05T13:23:39ZYehuda Sadehyehuda@redhat.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Resolved</i></li></ul><p>Tested on firefly, seem to work.</p>