Ceph » rgw

I think I've found a similar problem around object size = 16. I think it's likely to have the same cause. Object size 15 worked fine for me, object size 17 "worked" - but processed things one byte at a time. I'll put more here once I learn more what's happening.

Unfortunately this PR doesn't solve the problem for us. We attached the entire client and server log (running with your patch), and the wireshark dump from the client side.

The client starting the chunked upload and first two chunks:

STREAMING-AWS4-HMAC-SHA256-PAYLOAD
nhpAWS4Authentication: string-to-sign
AWS4-HMAC-SHA256
20170816T115603Z
20170816/us-east-1/s3/aws4_request
6a93f48bcc8d274117f48a50dc84c5cfd812b3fb7edc5be7d763459647423c62
nhp:  > Authorization: <hidden>
nhp:  >
nhp:  > write 16384 bytes
nhpAWS4EncWrite: string-to-sign
AWS4-HMAC-SHA256-PAYLOAD
20170816T115603Z
20170816/us-east-1/s3/aws4_request
f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
b3d650f1ee6a4f131076f307ac5eb6ff9563ab3714eacf963e093238733990d4
nhp: 2017-08-16 13:56:03.569239 send enter 0x7fd8353ec740 2677
nhp: 2017-08-16 13:56:03.569292 send return 2677 0 0 elapsed +0 00:00:00.000053
nhp: 2017-08-16 13:56:03.569300 send enter 0x7fd8353ec740 4
nhp: 2017-08-16 13:56:03.569310 send return 4 0 0 elapsed +0 00:00:00.000010
nhp: 2017-08-16 13:56:03.569315 send enter 0x7fd8353ec740 87
nhp: 2017-08-16 13:56:03.569328 send return 87 0 0 elapsed +0 00:00:00.000013
nhp: 2017-08-16 13:56:03.569333 send enter 0x7fd8353ec740 2
nhp: 2017-08-16 13:56:03.569343 send return 2 0 0 elapsed +0 00:00:00.000010
nhp: 2017-08-16 13:56:03.569349 send enter 0x7fd8353ec740 6
nhp: 2017-08-16 13:56:03.569357 send return 6 0 0 elapsed +0 00:00:00.000008
nhp: 2017-08-16 13:56:03.569362 send enter 0x7fd8353ec740 16384
nhp: 2017-08-16 13:56:03.569381 send return 16384 0 0 elapsed +0 00:00:00.000019
nhp: 2017-08-16 13:56:03.569387 send enter 0x7fd8353ec740 2
nhp: 2017-08-16 13:56:03.569393 send return 2 0 0 elapsed +0 00:00:00.000006
nhp:  > write 16384 bytes
nhpAWS4EncWrite: string-to-sign
AWS4-HMAC-SHA256-PAYLOAD
20170816T115603Z
20170816/us-east-1/s3/aws4_request
d61f8acc559fdf781581b6eda77de720360a2786f1d8449aa10654cf73997487
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
e33f24140499430b048f6600af4f41f3ccb0cb766d9f7661124cf8ba4b827523
nhp: 2017-08-16 13:56:03.569633 send enter 0x7fd8353ec740 4
nhp: 2017-08-16 13:56:03.569654 send return 4 0 0 elapsed +0 00:00:00.000021
nhp: 2017-08-16 13:56:03.569659 send enter 0x7fd8353ec740 89
nhp: 2017-08-16 13:56:03.569675 send return 89 0 0 elapsed +0 00:00:00.000016
nhp: 2017-08-16 13:56:03.569680 send enter 0x7fd8353ec740 2
nhp: 2017-08-16 13:56:03.569690 send return 2 0 0 elapsed +0 00:00:00.000010
nhp: 2017-08-16 13:56:03.569695 send enter 0x7fd8353ec740 6
nhp: 2017-08-16 13:56:03.569709 send return 6 0 0 elapsed +0 00:00:00.000014
nhp: 2017-08-16 13:56:03.569714 send enter 0x7fd8353ec740 16384
nhp: 2017-08-16 13:56:03.569731 send return 16384 0 0 elapsed +0 00:00:00.000017
nhp: 2017-08-16 13:56:03.569737 send enter 0x7fd8353ec740 2
nhp: 2017-08-16 13:56:03.569742 send return 2 0 0 elapsed +0 00:00:00.000005

The wireshark dump from the client:

Authorization: AWS4-HMAC-SHA256 Credential=5QB12EADJ3HTU0PR5TQZ/20170816/us-east-1/s3/aws4_request, SignedHeaders=content-encoding;content-type;date;host;x-amz-content-sha256;x-amz-date;x-amz-decoded-content-length;x-amz-meta-blocksize;x-amz-meta-chunkidformat;x-amz-meta-chunkprefix;x-amz-meta-chunks;x-amz-meta-chunksize;x-amz-meta-chunktype;x-amz-meta-copynumber;x-amz-meta-dbid;x-amz-meta-dbname;x-amz-meta-filename;x-amz-meta-filesize;x-amz-meta-filetype;x-amz-meta-incarnation;x-amz-meta-libraryname;x-amz-meta-node;x-amz-meta-opentime;x-amz-meta-sbtrequest;x-amz-meta-sbtretrycount;x-amz-meta-sbtversion;x-amz-meta-sessionid;x-amz-meta-status;x-amz-meta-system;x-amz-meta-tickloc;x-amz-meta-user, Signature=f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0

57
4000;chunk-signature=d61f8acc559fdf781581b6eda77de720360a2786f1d8449aa10654cf73997487

4000
.....

Then here's rgw detecting the chunked upload:

STREAMING-AWS4-HMAC-SHA256-PAYLOAD
2017-08-16 13:56:03.570349 7f4bdc4f5700 10 canonical request hash = 6a93f48bcc8d274117f48a50dc84c5cfd812b3fb7edc5be7d763459647423c62
2017-08-16 13:56:03.570358 7f4bdc4f5700 10 string to sign = AWS4-HMAC-SHA256
20170816T115603Z
20170816/us-east-1/s3/aws4_request
6a93f48bcc8d274117f48a50dc84c5cfd812b3fb7edc5be7d763459647423c62
2017-08-16 13:56:03.570387 7f4bdc4f5700 10 body content detected in multiple chunks
2017-08-16 13:56:03.570388 7f4bdc4f5700 10 aws4 seed signature ok... delaying v4 auth
2017-08-16 13:56:03.570503 7f4bdc4f5700 10 date_k    = aab84507d9ebbbeebff8608f793ad0cfc4c07b757c28829bc331a9866ea04ee7
2017-08-16 13:56:03.570515 7f4bdc4f5700 10 region_k  = 301517aefddf7433ec49a49f2ba5a690e7a36a1c5176b4b02ffd324ba6202d85
2017-08-16 13:56:03.570518 7f4bdc4f5700 10 service_k = 5ee8c521c9d09722fed63a9435c3b00c69b5ac29566cf95d8edcaac9d037515c
2017-08-16 13:56:03.570528 7f4bdc4f5700 10 signing_k = ec979262faf07db88a9631f4fc75bb3954823ea84749c05fb8834088a65cd841
2017-08-16 13:56:03.570552 7f4bdc4f5700 10 generated signature = f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0
2017-08-16 13:56:03.570557 7f4bdc4f5700 15 string_to_sign=AWS4-HMAC-SHA256
20170816T115603Z
20170816/us-east-1/s3/aws4_request
6a93f48bcc8d274117f48a50dc84c5cfd812b3fb7edc5be7d763459647423c62
2017-08-16 13:56:03.570569 7f4bdc4f5700 15 server signature=f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0
2017-08-16 13:56:03.570570 7f4bdc4f5700 15 client signature=f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0
2017-08-16 13:56:03.570571 7f4bdc4f5700 15 compare=0
2017-08-16 13:56:03.570618 7f4bdc4f5700 10 date_k    = aab84507d9ebbbeebff8608f793ad0cfc4c07b757c28829bc331a9866ea04ee7
2017-08-16 13:56:03.570627 7f4bdc4f5700 10 region_k  = 301517aefddf7433ec49a49f2ba5a690e7a36a1c5176b4b02ffd324ba6202d85
2017-08-16 13:56:03.570631 7f4bdc4f5700 10 service_k = 5ee8c521c9d09722fed63a9435c3b00c69b5ac29566cf95d8edcaac9d037515c
2017-08-16 13:56:03.570634 7f4bdc4f5700 10 signing_k = ec979262faf07db88a9631f4fc75bb3954823ea84749c05fb8834088a65cd841

And failing on the first chunk:

2017-08-16 13:56:03.570755 7f4bdc4f5700  2 req 12:0.001047:s3:PUT /oracle-data-miropoto-1/file_chunk/0/SBTDB/unknown/2017-08-16/crap/ZyH3tH16jobK/0000000001:put_obj:executing
2017-08-16 13:56:03.570939 7f4bdc4f5700 20 parsed new chunk; signature=d61f8acc559fdf781581b6eda77de720360a2786f1d8449aa10654cf73997487, data_length=87, data_starts_in_stream=91
2017-08-16 13:56:03.570952 7f4bdc4f5700 30 AWSv4ComplMulti: stream_pos_was=91, to_extract=87
2017-08-16 13:56:03.570954 7f4bdc4f5700 30 AWSv4ComplMulti: to_extract=87, data_len=10
2017-08-16 13:56:03.570962 7f4bdc4f5700 30 AWSv4ComplMulti: to_extract=77, received=77
2017-08-16 13:56:03.570964 7f4bdc4f5700 20 AWSv4ComplMulti: filled=87
2017-08-16 13:56:03.571023 7f4bdc4f5700 20 AWSv4ComplMulti: string_to_sign=
AWS4-HMAC-SHA256-PAYLOAD
20170816T115603Z
20170816/us-east-1/s3/aws4_request
f69993822618a847e0e101d9fff756471f3faed67808890e1b6fb536c80c98c0
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
3a79b501bbeaa09e83a00e71ecb07b3acd43acba55b805f393f543edf6c09376
2017-08-16 13:56:03.571054 7f4bdc4f5700 20 AWSv4ComplMulti: ERROR: chunk signature mismatch
2017-08-16 13:56:03.571058 7f4bdc4f5700 20 AWSv4ComplMulti: declared signature=d61f8acc559fdf781581b6eda77de720360a2786f1d8449aa10654cf73997487
2017-08-16 13:56:03.571059 7f4bdc4f5700 20 AWSv4ComplMulti: calculated signature=bd1a20b0df8a661c5a736bad4703dfc0e0be28ac77c224fd71a9dd44d91697ce

Attached is a wireshark dump of the response to a restore test.
We noticed the response has

Content-Encoding: aws-chunked

Ok, I see that the original patch doesn't solve the full issue yet, changing from backport -> in progress