rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail.
As of 1.11 aws-sdk-java uses v4 signatures by default, and optionally does
chunked transfers. When it decides to do chunked transfers may depend
in part on how large the file is (7M reported to be a good size for this),
but whenever it tries to do a chunked transfer, the transfer fails
on the first chunk. It is possible to force chunked transfers always by
when constructing S3ClientOptions. With that option, I can see the
behavior with a 576 byte file.
aws-sdk-java 1.10 and previous did not use v4 signatures, so did not fail.
The problem appears to be in ceph's
where it uses the string AWS4-HMAC-SHA256 it appears that Java is using
this string instead, AWS4-HMAC-SHA256-PAYLOAD .
I'm including a sample java program that can be used to exercise the bug. usage:
p5 file-to-read-from bucket objectname-to-write-to
I'm including a patch that can be applied against aws-sdk-java 1.11.152-1-g4a90256 to add debug output for
per-chunk signature generation.
And I'm including a sample fix for ceph - with that fix, the sample java program here works.