Project

General

Profile

Bug #20447

rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail.

Added by Marcus Watts almost 7 years ago. Updated over 6 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
kraken, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

As of 1.11 aws-sdk-java uses v4 signatures by default, and optionally does
chunked transfers. When it decides to do chunked transfers may depend
in part on how large the file is (7M reported to be a good size for this),
but whenever it tries to do a chunked transfer, the transfer fails
on the first chunk. It is possible to force chunked transfers always by
.setPayloadSigningEnabled(true)
when constructing S3ClientOptions. With that option, I can see the
behavior with a 576 byte file.

aws-sdk-java 1.10 and previous did not use v4 signatures, so did not fail.

The problem appears to be in ceph's
src/rgw/rgw_auth_s3.cc
AWSv4ComplMulti::calc_chunk_signature
where it uses the string AWS4-HMAC-SHA256 it appears that Java is using
this string instead, AWS4-HMAC-SHA256-PAYLOAD .

I'm including a sample java program that can be used to exercise the bug. usage:
p5 file-to-read-from bucket objectname-to-write-to

I'm including a patch that can be applied against aws-sdk-java 1.11.152-1-g4a90256 to add debug output for
per-chunk signature generation.

And I'm including a sample fix for ceph - with that fix, the sample java program here works.

p5.java View (4.3 KB) Marcus Watts, 06/28/2017 07:52 AM

aws-sdk-java-debug-1 (3.34 KB) Marcus Watts, 06/28/2017 07:52 AM

ceph-fix-chunked-sig-1 (920 Bytes) Marcus Watts, 06/28/2017 07:53 AM


Related issues

Copied to rgw - Backport #20827: kraken: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. Rejected
Copied to rgw - Backport #20828: jewel: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. Rejected

History

#1 Updated by Marcus Watts over 6 years ago

I've submitted a PR with the patch I posted here. https://github.com/ceph/ceph/pull/15965

#2 Updated by Yehuda Sadeh over 6 years ago

  • Backport set to kraken, jewel

#3 Updated by Matt Benjamin over 6 years ago

  • Status changed from New to 7

#4 Updated by Casey Bodley over 6 years ago

  • Status changed from 7 to Pending Backport

#5 Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #20827: kraken: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. added

#6 Updated by Nathan Cutler over 6 years ago

  • Copied to Backport #20828: jewel: rgw: S3 object uploads using aws-sdk-java 1.11 to do chunked transfers fail. added

#7 Updated by Marcus Watts over 6 years ago

Turns out this issue does not appear in Jewel: it was introduced in kraken. So no jewel backport necessary. I've noted this in the jewel backport tracker issue accordingly.

#8 Updated by Nathan Cutler over 6 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF