https://tracker.ceph.com/https://tracker.ceph.com/favicon.ico2019-03-14T17:44:18ZCeph rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1320312019-03-14T17:44:18ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Assignee</strong> set to <i>Casey Bodley</i></li><li><strong>Priority</strong> changed from <i>Normal</i> to <i>High</i></li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1322712019-03-18T21:49:43ZCasey Bodleycbodley@redhat.com
<ul></ul><p>A similar report coming from Dan Smith via [ceph-users] Rados Gateway using S3 Api does not store file correctly:</p>
<blockquote>
<p>The file is 92MB in size. I have stored files much larger and much smaller. If I store the file WITHOUT using the Customer Provided 256-bit AES key using Server Side encryption, the file stores and retrieves just fine (SHA256 hashes match).</p>
<p>If I store the file USING the 256-bit AES key using Server Side encryption, the file stores without error, however, when I retrieve the file and compare the hash of the file I retrieve from ceph against the hash of the original file, the hashes differ.</p>
<p>I am using the AWSSDK.S3 nuget package version 3.3.31.24, with ceph version "ceph version 12.2.10-551-gbb089269ea (bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)"</p>
</blockquote>
<p>Perhaps this sanitized header is useful:<br /><pre>
PUT [redacted]/delete-me?partNumber=18&uploadId=2~2dt4pYGY3vfKxBb9FcbVlAnbz_z3HTV HTTP/1.1
Expect: 100-continue
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: [redacted]
x-amz-server-side-encryption-customer-key-MD5: [redacted]
User-Agent: aws-sdk-dotnet-coreclr/3.3.31.24 aws-sdk-dotnet-core/3.3.32.2 .NET_Core/4.6.27317.07 OS/Microsoft_Windows_10.0.17763 ClientAsync TransferManager/MultipartUploadCommand
Host: [redacted]
X-Amz-Date: [redacted]
X-Amz-Decoded-Content-Length: 5242880
X-Amz-Content-SHA256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
Authorization: [redacted]
Content-Length: 5248726
Content-Type: text/plain
14000;chunk-signature=[redacted]
[payload here]
</pre></p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1322722019-03-18T21:54:19ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>New</i> to <i>12</i></li></ul><p>I'm able to reproduce the issue with our s3tests case test_encryption_sse_c_multipart_upload() by adding 1 to the 5M part size:<br /><pre><code class="diff syntaxhl"><span class="CodeRay"><span class="line comment">diff --git a/s3tests/functional/test_s3.py b/s3tests/functional/test_s3.py</span>
<span class="line comment">index f2deb8e..a067ba4 100644</span>
<span class="line head"><span class="head">--- </span><span class="filename">a/s3tests/functional/test_s3.py</span></span>
<span class="line head"><span class="head">+++ </span><span class="filename">b/s3tests/functional/test_s3.py</span></span>
<span class="change"><span class="change">@@</span> -8789,7 +8789,7 <span class="change">@@</span></span> <span class="keyword">def</span> <span class="function">test_encryption_sse_c_multipart_upload</span>():
<span class="string"><span class="delimiter">'</span><span class="content">x-amz-server-side-encryption-customer-key-md5</span><span class="delimiter">'</span></span>: <span class="string"><span class="delimiter">'</span><span class="content">DWygnHRtgiJ77HCm+1rvHw==</span><span class="delimiter">'</span></span>,
<span class="string"><span class="delimiter">'</span><span class="content">Content-Type</span><span class="delimiter">'</span></span>: content_type
}
<span class="line delete"><span class="delete">-</span> (upload, data) = _multipart_upload_enc(bucket, key, objlen,</span>
<span class="line insert"><span class="insert">+</span> (upload, data) = _multipart_upload_enc(bucket, key, objlen,<span class="eyecatcher"> part_size=<span class="integer">1</span>+<span class="integer">5</span>*<span class="integer">1024</span>*<span class="integer">1024</span>,</span></span>
init_headers=enc_headers, part_headers=enc_headers,
metadata={<span class="string"><span class="delimiter">'</span><span class="content">foo</span><span class="delimiter">'</span></span>: <span class="string"><span class="delimiter">'</span><span class="content">bar</span><span class="delimiter">'</span></span>})
upload.complete_upload()
</span></code></pre><br /><pre>
======================================================================
FAIL: s3tests.functional.test_s3.test_encryption_sse_c_multipart_upload
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/cbodley/s3-tests/virtualenv/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/cbodley/s3-tests/s3tests/functional/test_s3.py", line 8807, in test_encryption_sse_c_multipart_upload
eq(data, test_string)
AssertionError: 'dVZduWXPJyzErZCgbML[...]rDqRyNOPBIdnsm' != 'dVZduWXPJyzErZCgbML[...]\xc1tS\r\x04\nJ!'
</pre></p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1322972019-03-19T10:22:06ZLászló van den Hoek
<ul></ul><blockquote>
<p>A similar report coming from Dan Smith via [ceph-users] Rados Gateway using S3 Api does not store file correctly</p>
</blockquote>
<p>Link: <a class="external" href="http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033833.html">http://lists.ceph.com/pipermail/ceph-users-ceph.com/2019-March/033833.html</a></p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1325612019-03-22T13:53:44ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Backport</strong> set to <i>nautilus,mimic,luminous</i></li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1325922019-03-22T15:33:35ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>12</i> to <i>Fix Under Review</i></li><li><strong>Pull request ID</strong> set to <i>27130</i></li></ul><p>I'm testing a fix for this at <a class="external" href="https://github.com/ceph/ceph/pull/27130">https://github.com/ceph/ceph/pull/27130</a>. It looks like the issue is only on the decrypt side, so existing encrypted data is not corrupted and can be recovered.</p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1325962019-03-22T16:14:22ZCasey Bodleycbodley@redhat.com
<ul></ul><p>s3tests in <a class="external" href="https://github.com/ceph/s3-tests/pull/263">https://github.com/ceph/s3-tests/pull/263</a> will also need backports</p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1330832019-03-28T17:31:45ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>Fix Under Review</i> to <i>7</i></li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1333062019-04-01T15:27:16ZCasey Bodleycbodley@redhat.com
<ul><li><strong>Status</strong> changed from <i>7</i> to <i>Pending Backport</i></li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1333092019-04-01T15:50:10ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/39068">Backport #39068</a>: nautilus: silent corruption using SSE-C on multi-part upload to S3 with non-default part size</i> added</li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1333112019-04-01T15:50:18ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/39069">Backport #39069</a>: mimic: silent corruption using SSE-C on multi-part upload to S3 with non-default part size</i> added</li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1333132019-04-01T15:50:27ZAbhishek Lekshmananabhishek.lekshmanan@gmail.com
<ul><li><strong>Copied to</strong> <i><a class="issue tracker-9 status-3 priority-4 priority-default closed" href="/issues/39070">Backport #39070</a>: luminous: silent corruption using SSE-C on multi-part upload to S3 with non-default part size</i> added</li></ul> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1333312019-04-01T16:54:51ZCasey Bodleycbodley@redhat.com
<ul></ul><p>test cases in <a class="external" href="https://github.com/ceph/s3-tests/pull/266">https://github.com/ceph/s3-tests/pull/266</a> can be backported as well</p> rgw - Bug #38700: silent corruption using SSE-C on multi-part upload to S3 with non-default part sizehttps://tracker.ceph.com/issues/38700?journal_id=1337652019-04-08T12:08:56ZNathan Cutlerncutler@suse.cz
<ul><li><strong>Status</strong> changed from <i>Pending Backport</i> to <i>Resolved</i></li></ul>