Bug #20447
closedrgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail.
0%
Description
As of 1.11 aws-sdk-java uses v4 signatures by default, and optionally does
chunked transfers. When it decides to do chunked transfers may depend
in part on how large the file is (7M reported to be a good size for this),
but whenever it tries to do a chunked transfer, the transfer fails
on the first chunk. It is possible to force chunked transfers always by
.setPayloadSigningEnabled(true)
when constructing S3ClientOptions. With that option, I can see the
behavior with a 576 byte file.
aws-sdk-java 1.10 and previous did not use v4 signatures, so did not fail.
The problem appears to be in ceph's
src/rgw/rgw_auth_s3.cc
AWSv4ComplMulti::calc_chunk_signature
where it uses the string AWS4-HMAC-SHA256 it appears that Java is using
this string instead, AWS4-HMAC-SHA256-PAYLOAD .
I'm including a sample java program that can be used to exercise the bug. usage:
p5 file-to-read-from bucket objectname-to-write-to
I'm including a patch that can be applied against aws-sdk-java 1.11.152-1-g4a90256 to add debug output for
per-chunk signature generation.
And I'm including a sample fix for ceph - with that fix, the sample java program here works.
Files
Updated by Marcus Watts almost 7 years ago
I've submitted a PR with the patch I posted here. https://github.com/ceph/ceph/pull/15965
Updated by Casey Bodley over 6 years ago
- Status changed from 7 to Pending Backport
Updated by Nathan Cutler over 6 years ago
- Copied to Backport #20827: kraken: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. added
Updated by Nathan Cutler over 6 years ago
- Copied to Backport #20828: jewel: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. added
Updated by Marcus Watts over 6 years ago
Turns out this issue does not appear in Jewel: it was introduced in kraken. So no jewel backport necessary. I've noted this in the jewel backport tracker issue accordingly.
Updated by Nathan Cutler over 6 years ago
- Status changed from Pending Backport to Resolved