Project

General

Profile

Bug #20463

rgw: AWSv4 encoding/signature problem, can happen with listobjects marker.

Added by Marcus Watts over 1 year ago. Updated 9 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
06/28/2017
Due date:
% Done:

0%

Source:
Tags:
Backport:
jewel kraken
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

For buckets containing object names that contain / and %, it is
possible in some circumstances (at least with boto) for "listobjects"
operations to attempt to fetch additional objects using 'marker=' and
a value containing both / and %. When this happens, when using AWSv4,
radosgw then returns a signature validation error.

It is not actually necessary have a bucket with lots of strange objects
to order to reproduce this. It can be more simply done on any bucket
by constructing a marker value.

Following is a way to do this with boto.

export S3_USE_SIGV4=1
python
import boto
import boto.s3.connection
access_key = '40JV0D93WB9U33YNQ0L4'
secret_key = '8GjwnD4iocK3gUuOAbIj9G/9QFeXgIt+n9rJk06O'

conn = boto.connect_s3(
aws_access_key_id = access_key,
aws_secret_access_key = secret_key,
host = 'hybodus', port = 80,
is_secure=False, calling_format = boto.s3.connection.OrdinaryCallingFormat(), )

conn._auth_handler.region_name = "default"

bucket = conn.get_bucket('my-new-bucket')
#these fail:
s=bucket.get_all_keys(marker='level1/8e%25%25FAH3')
s=bucket.get_all_keys(marker='level18e%25/FAH3')
#these work:
s=bucket.get_all_keys(marker='level1/3')
s=bucket.get_all_keys(marker='level18e%25FAH3')

I'm including a patch to ceph here that makes the above work.

ceph-fix-badmarker-1 (599 Bytes) Marcus Watts, 06/28/2017 10:22 PM


Related issues

Copied to rgw - Backport #20824: jewel: rgw: AWSv4 encoding/signature problem, can happen with listobjects marker. Resolved
Copied to rgw - Backport #20825: kraken: rgw: AWSv4 encoding/signature problem, can happen with listobjects marker. Rejected

History

#1 Updated by Marcus Watts over 1 year ago

I've made this PR for ceph: https://github.com/ceph/ceph/pull/15998 - contains just the patch I posted above.

#2 Updated by Orit Wasserman over 1 year ago

do we need a fix for jewel?

#3 Updated by Orit Wasserman over 1 year ago

  • Status changed from New to Testing

#4 Updated by Matt Benjamin over 1 year ago

  • Assignee set to Marcus Watts

#5 Updated by Marcus Watts over 1 year ago

Jewel has the same behavior - so yes it probably needs the same fix.

#7 Updated by Matt Benjamin over 1 year ago

  • Status changed from Testing to Pending Backport
  • Backport set to jewel kraken

#8 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #20824: jewel: rgw: AWSv4 encoding/signature problem, can happen with listobjects marker. added

#9 Updated by Nathan Cutler over 1 year ago

  • Copied to Backport #20825: kraken: rgw: AWSv4 encoding/signature problem, can happen with listobjects marker. added

#10 Updated by Nathan Cutler 9 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF