Project

General

Profile

Bug #22729

rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object

Added by Malcolm Lee about 1 year ago. Updated 10 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
-
Start date:
01/18/2018
Due date:
% Done:

0%

Source:
Tags:
Backport:
luminous, jewel
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:

Description

When use copy part without http header "x-amz-copy-source-range",rgw will use copy_obj to handle it. But according to aws s3 doc https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html:

"This request header is not required when copying an entire source object."

So, "x-amz-copy-source-range" is a optional header, and it can't be a mark to distinguish "copy part" from "copy object".

The issue can be reproduced with Jewel 10.2.10.

2018-01-17 17:49:05.077322 7f07b9f83700 1 ====== starting new request req=0x7f07b9f7d790 =====
2018-01-17 17:49:05.077344 7f07b9f83700 2 req 13:0.000022::PUT /found1/ttm1::initializing for trans_id = tx00000000000000000000d-005a5f1c11-108a-default
2018-01-17 17:49:05.077354 7f07b9f83700 10 rgw api priority: s3=5 s3website=4
2018-01-17 17:49:05.077356 7f07b9f83700 10 host=10.71.84.73
2018-01-17 17:49:05.077358 7f07b9f83700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2018-01-17 17:49:05.077360 7f07b9f83700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/found1/ttm1
2018-01-17 17:49:05.077369 7f07b9f83700 10 meta>> HTTP_X_AMZ_COPY_SOURCE
2018-01-17 17:49:05.077375 7f07b9f83700 10 x>> x-amz-copy-source:found1/t1
2018-01-17 17:49:05.077397 7f07b9f83700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2018-01-17 17:49:05.077403 7f07b9f83700 10 handler=22RGWHandler_REST_Obj_S3
2018-01-17 17:49:05.077405 7f07b9f83700 2 req 13:0.000084:s3:PUT /found1/ttm1::getting op 1
2018-01-17 17:49:05.077413 7f07b9f83700 10 op=22RGWCopyObj_ObjStore_S3
2018-01-17 17:49:05.077414 7f07b9f83700 2 req 13:0.000093:s3:PUT /found1/ttm1:copy_obj:authorizing
2018-01-17 17:49:05.077449 7f07b9f83700 10 get_canon_resource(): dest=/found1/ttm1?partNumber=1&uploadId=2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w
2018-01-17 17:49:05.077452 7f07b9f83700 10 auth_hdr:
PUT

Wed, 17 Jan 2018 09:49:05 UTC
x-amz-copy-source:found1/t1
/found1/ttm1?partNumber=1&uploadId=2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w
2018-01-17 17:49:05.077501 7f07b9f83700 15 calculated digest=jn/oxkhT1vvzg07acKRtrqjoNRY=
2018-01-17 17:49:05.077502 7f07b9f83700 15 auth_sign=jn/oxkhT1vvzg07acKRtrqjoNRY=
2018-01-17 17:49:05.077503 7f07b9f83700 15 compare=0
2018-01-17 17:49:05.077505 7f07b9f83700 2 req 13:0.000184:s3:PUT /found1/ttm1:copy_obj:normalizing buckets and tenants
2018-01-17 17:49:05.077508 7f07b9f83700 10 s->object=ttm1 s->bucket=found1
2018-01-17 17:49:05.077510 7f07b9f83700 2 req 13:0.000189:s3:PUT /found1/ttm1:copy_obj:init permissions
2018-01-17 17:49:05.077542 7f07b9f83700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>liang</ID><DisplayName>lee</DisplayName></Owner><AccessControlList><Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>liang</ID><DisplayName>lee</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-01-17 17:49:05.077551 7f07b9f83700 2 req 13:0.000230:s3:PUT /found1/ttm1:copy_obj:recalculating target
2018-01-17 17:49:05.077553 7f07b9f83700 2 req 13:0.000232:s3:PUT /found1/ttm1:copy_obj:reading permissions
2018-01-17 17:49:05.077555 7f07b9f83700 2 req 13:0.000234:s3:PUT /found1/ttm1:copy_obj:init op
2018-01-17 17:49:05.077556 7f07b9f83700 2 req 13:0.000235:s3:PUT /found1/ttm1:copy_obj:verifying op mask
2018-01-17 17:49:05.077557 7f07b9f83700 20 required_mask= 2 user.op_mask=7
2018-01-17 17:49:05.077558 7f07b9f83700 2 req 13:0.000237:s3:PUT /found1/ttm1:copy_obj:verifying op permissions
2018-01-17 17:49:05.077576 7f07b9f83700 20 get_obj_state: rctx=0x7f07b9f7d6a0 obj=found1:_multipart_t1.2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w.meta state=0x559252006cd8 s->prefetch_data=0
2018-01-17 17:49:05.078871 7f07b9f83700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>liang</ID><DisplayName>lee</DisplayName></Owner><AccessControlList><Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>liang</ID><DisplayName>lee</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-01-17 17:49:05.078883 7f07b9f83700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2
2018-01-17 17:49:05.079002 7f07b9f83700 2 req 13:0.001680:s3:PUT /found1/ttm1:copy_obj:op status=-2
2018-01-17 17:49:05.079006 7f07b9f83700 2 req 13:0.001685:s3:PUT /found1/ttm1:copy_obj:http status=404
2018-01-17 17:49:05.079011 7f07b9f83700 1 ====== req done req=0x7f07b9f7d790 op status=-2 http_status=404 ======
2018-01-17 17:49:05.079026 7f07b9f83700 20 process_request() returned -2
2018-01-17 17:49:05.079077 7f07b9f83700 1 civetweb: 0x559252072200: 10.71.84.73 - - [17/Jan/2018:17:49:05 +0800] "PUT /found1/ttm1 HTTP/1.1" 404 0 - Go-http-client/1.1


Related issues

Copied to rgw - Backport #22903: luminous: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object Resolved
Copied to rgw - Backport #22904: jewel: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object Resolved

History

#2 Updated by Matt Benjamin about 1 year ago

Hi Malcolm,

It's early and I havne't had coffee. Can you clarify why RGW's behavior is semantically wrong--meaning, how it produces an incorrect end result? Because I don't think the header being optional in AWS by itself forbids RGW to choose a different strategy for performing an operation if the end result is valid.

Matt

#3 Updated by Malcolm Lee about 1 year ago

Matt Benjamin wrote:

Hi Malcolm,

It's early and I havne't had coffee. Can you clarify why RGW's behavior is semantically wrong--meaning, how it produces an incorrect end result? Because I don't think the header being optional in AWS by itself forbids RGW to choose a different strategy for performing an operation if the end result is valid.

Matt

Hi Matt,
For example, I initiate multipart, and then use copy part to copy a existing source object as a part one without "x-amz-copy-source-range"(I want to copy an entire source object), but rgw will return "HTTP 404". In rgw log,

2018-01-17 17:49:05.077551 7f07b9f83700 2 req 13:0.000230:s3:PUT /found1/ttm1:copy_obj:recalculating target
2018-01-17 17:49:05.077553 7f07b9f83700 2 req 13:0.000232:s3:PUT /found1/ttm1:copy_obj:reading permissions
2018-01-17 17:49:05.077555 7f07b9f83700 2 req 13:0.000234:s3:PUT /found1/ttm1:copy_obj:init op
2018-01-17 17:49:05.077556 7f07b9f83700 2 req 13:0.000235:s3:PUT /found1/ttm1:copy_obj:verifying op mask
2018-01-17 17:49:05.077557 7f07b9f83700 20 required_mask= 2 user.op_mask=7
2018-01-17 17:49:05.077558 7f07b9f83700 2 req 13:0.000237:s3:PUT /found1/ttm1:copy_obj:verifying op permissions
2018-01-17 17:49:05.077576 7f07b9f83700 20 get_obj_state: rctx=0x7f07b9f7d6a0
2018-01-17 17:49:05.078883 7f07b9f83700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2
2018-01-17 17:49:05.079002 7f07b9f83700 2 req 13:0.001680:s3:PUT /found1/ttm1:copy_obj:op status=-2
2018-01-17 17:49:05.079006 7f07b9f83700 2 req 13:0.001685:s3:PUT /found1/ttm1:copy_obj:http status=404

Radowsgw use copy_obj to handle copy part request, while copying part should be handled with put_obj in normal request with header "x-amz-copy-source-range".

#4 Updated by Malcolm Lee about 1 year ago

And the syntax of "Copy Object" is

PUT /destinationObject HTTP/1.1
Host: destinationBucket.s3.amazonaws.com
x-amz-copy-source: /source_bucket/sourceObject
x-amz-metadata-directive: metadata_directive
x-amz-copy-source-if-match: etag
x-amz-copy-source-if-none-match: etag
x-amz-copy-source-if-unmodified-since: time_stamp
x-amz-copy-source-if-modified-since: time_stamp
<request metadata>
Authorization: authorization string (see Authenticating Requests (AWS Signature Version
        4))
Date: date

Copy Part is
PUT /ObjectName?partNumber=PartNumber&uploadId=UploadId HTTP/1.1
Host: BucketName.s3.amazonaws.com
x-amz-copy-source: /source_bucket/sourceObject
x-amz-copy-source-range:bytes=first-last
x-amz-copy-source-if-match: etag
x-amz-copy-source-if-none-match: etag
x-amz-copy-source-if-unmodified-since: time_stamp
x-amz-copy-source-if-modified-since: time_stamp
Date: date
Authorization: authorization string

These two requests params also different.

#5 Updated by Matt Benjamin about 1 year ago

I see, sorry.

#6 Updated by Abhishek Lekshmanan about 1 year ago

  • Status changed from New to In Progress

#7 Updated by Matt Benjamin about 1 year ago

  • Status changed from In Progress to Pending Backport
  • Assignee set to Matt Benjamin
  • Backport set to luminous, jewel

#8 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #22903: luminous: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object added

#9 Updated by Nathan Cutler about 1 year ago

  • Copied to Backport #22904: jewel: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object added

#10 Updated by Nathan Cutler 10 months ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF