Bug #38785: prefix begin with space got 403 error - rgw - Ceph

Actions

Copy link

Bug #38785

open

prefix begin with space got 403 error

Added by liang sibin about 5 years ago. Updated about 5 years ago.

Status:

Triaged

Priority:

Normal

Assignee:

Matt Benjamin

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

Ceph - v12.2.5

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

aws s3 sdk version 1.11.517
ceph version 12.2.5
when listobjects setting prefix begin with space will get 403 error.
if space not at beginning,it will success.

String bucketName = "0000";
    ListObjectsRequest listObjectsRequest = new ListObjectsRequest();
    listObjectsRequest.setPrefix(" abc/");
    listObjectsRequest.setDelimiter("/");
    listObjectsRequest.setBucketName(bucketName);
    ObjectListing listing = s3.listObjects(listObjectsRequest);
    List<S3ObjectSummary> s3ObjectSummaries = listing.getObjectSummaries();
    for (S3ObjectSummary s3ObjectSummary : s3ObjectSummaries) {
        System.out.println(s3ObjectSummary.getKey());
    }

https://github.com/aws/aws-sdk-java/issues/1942

Actions

Copy link

Updated by liang sibin about 5 years ago

https://github.com/aws/aws-sdk-java/issues/1733
Another problem maybe is the same problem.

Actions

Copy link

Updated by Matt Benjamin about 5 years ago

Status changed from New to Triaged
Assignee set to Matt Benjamin

Actions

Copy link

Updated by Matt Benjamin about 5 years ago

I don't reproduce this behavior on master.

What I tested more specifically was, given the following keys:

keys = ['abc/one' , 'abc/two', 'foo/one', 'foo/two']

the following using boto3:

prefix = ' abc/'
delim = '/'

res1 = s3.list_objects(Bucket=bname, Prefix=prefix, Delimiter=delim)
print res1

succeeds and prints:

{u'Name': 'buck1', 'ResponseMetadata': {'HTTPStatusCode': 200, 'RetryAttempts': 0, 'HostId': '', 'RequestId': 'tx000000000000000000081-005c90c024-1027-default', 'HTTPHeaders': {'transfer-encoding': 'chunked', 'date': 'Tue, 19 Mar 2019 10:10:44 GMT', 'connection': 'Keep-Alive', 'x-amz-request-id': 'tx000000000000000000081-005c90c024-1027-default', 'content-type': 'application/xml'}}, u'Delimiter': u'/', u'MaxKeys': 1000, u'Prefix': ' abc/', u'Marker': u'', u'EncodingType': 'url', u'IsTruncated': False}

Actions

Copy link

Updated by Matt Benjamin about 5 years ago

I get the same, no-403 result running the above test against our downstream ceph rgw, based on 12.2.8.

Please review my reproducer steps in case I'm missing something important.

Matt

Actions

Copy link

Updated by liang sibin about 5 years ago

but use aws s3 java sdk will get 403 error.

Actions

Copy link

Updated by Matt Benjamin about 5 years ago

hi,

I can investigate with the sdk; that said, do you have any idea what is wrong the the leading space at the S3 protocol level? I.e., doing the same thing with boto3 and getting success is normally an indication that there's nothing wrong with the protocol implementation; if the java sdk is mishandling the leading space, that would be an sdk bug...

Matt

Actions

Copy link

Updated by Matt Benjamin about 5 years ago

(I'll verify that the desired prefix--including leading space--is reaching rgw)

Actions

Copy link

Updated by liang sibin about 5 years ago

https://github.com/aws/aws-sdk-java/issues/1733
Maybe the way Ceph calculates the SigV4 'CanonicalURI' parameter in this edge case differs from the way that actual-S3 does (which, for better and for worse, differs from the SigV4 spec). That'd cause SignatureDoesNotMatch, because the SDK matches actual-S3's implementation.

Actions

Copy link