Bug #22729
closedrgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object
0%
Description
When use copy part without http header "x-amz-copy-source-range"，rgw will use copy_obj to handle it. But according to aws s3 doc https://docs.aws.amazon.com/AmazonS3/latest/API/mpUploadUploadPartCopy.html:
"This request header is not required when copying an entire source object."
So, "x-amz-copy-source-range" is a optional header, and it can't be a mark to distinguish "copy part" from "copy object".
The issue can be reproduced with Jewel 10.2.10.
2018-01-17 17:49:05.077322 7f07b9f83700 1 ====== starting new request req=0x7f07b9f7d790 =====
2018-01-17 17:49:05.077344 7f07b9f83700 2 req 13:0.000022::PUT /found1/ttm1::initializing for trans_id = tx00000000000000000000d-005a5f1c11-108a-default
2018-01-17 17:49:05.077354 7f07b9f83700 10 rgw api priority: s3=5 s3website=4
2018-01-17 17:49:05.077356 7f07b9f83700 10 host=10.71.84.73
2018-01-17 17:49:05.077358 7f07b9f83700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2018-01-17 17:49:05.077360 7f07b9f83700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/found1/ttm1
2018-01-17 17:49:05.077369 7f07b9f83700 10 meta>> HTTP_X_AMZ_COPY_SOURCE
2018-01-17 17:49:05.077375 7f07b9f83700 10 x>> x-amz-copy-source:found1/t1
2018-01-17 17:49:05.077397 7f07b9f83700 20 get_handler handler=22RGWHandler_REST_Obj_S3
2018-01-17 17:49:05.077403 7f07b9f83700 10 handler=22RGWHandler_REST_Obj_S3
2018-01-17 17:49:05.077405 7f07b9f83700 2 req 13:0.000084:s3:PUT /found1/ttm1::getting op 1
2018-01-17 17:49:05.077413 7f07b9f83700 10 op=22RGWCopyObj_ObjStore_S3
2018-01-17 17:49:05.077414 7f07b9f83700 2 req 13:0.000093:s3:PUT /found1/ttm1:copy_obj:authorizing
2018-01-17 17:49:05.077449 7f07b9f83700 10 get_canon_resource(): dest=/found1/ttm1?partNumber=1&uploadId=2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w
2018-01-17 17:49:05.077452 7f07b9f83700 10 auth_hdr:
PUT
Wed, 17 Jan 2018 09:49:05 UTC
x-amz-copy-source:found1/t1
/found1/ttm1?partNumber=1&uploadId=2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w
2018-01-17 17:49:05.077501 7f07b9f83700 15 calculated digest=jn/oxkhT1vvzg07acKRtrqjoNRY=
2018-01-17 17:49:05.077502 7f07b9f83700 15 auth_sign=jn/oxkhT1vvzg07acKRtrqjoNRY=
2018-01-17 17:49:05.077503 7f07b9f83700 15 compare=0
2018-01-17 17:49:05.077505 7f07b9f83700 2 req 13:0.000184:s3:PUT /found1/ttm1:copy_obj:normalizing buckets and tenants
2018-01-17 17:49:05.077508 7f07b9f83700 10 s->object=ttm1 s->bucket=found1
2018-01-17 17:49:05.077510 7f07b9f83700 2 req 13:0.000189:s3:PUT /found1/ttm1:copy_obj:init permissions
2018-01-17 17:49:05.077542 7f07b9f83700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>liang</ID><DisplayName>lee</DisplayName></Owner><AccessControlList><Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>liang</ID><DisplayName>lee</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-01-17 17:49:05.077551 7f07b9f83700 2 req 13:0.000230:s3:PUT /found1/ttm1:copy_obj:recalculating target
2018-01-17 17:49:05.077553 7f07b9f83700 2 req 13:0.000232:s3:PUT /found1/ttm1:copy_obj:reading permissions
2018-01-17 17:49:05.077555 7f07b9f83700 2 req 13:0.000234:s3:PUT /found1/ttm1:copy_obj:init op
2018-01-17 17:49:05.077556 7f07b9f83700 2 req 13:0.000235:s3:PUT /found1/ttm1:copy_obj:verifying op mask
2018-01-17 17:49:05.077557 7f07b9f83700 20 required_mask= 2 user.op_mask=7
2018-01-17 17:49:05.077558 7f07b9f83700 2 req 13:0.000237:s3:PUT /found1/ttm1:copy_obj:verifying op permissions
2018-01-17 17:49:05.077576 7f07b9f83700 20 get_obj_state: rctx=0x7f07b9f7d6a0 obj=found1:_multipart_t1.2~Izxdi3NbETxcZPnnHmS3dj2ryVVhj4w.meta state=0x559252006cd8 s->prefetch_data=0
2018-01-17 17:49:05.078871 7f07b9f83700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>liang</ID><DisplayName>lee</DisplayName></Owner><AccessControlList><Grant>
<Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>liang</ID><DisplayName>lee</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-01-17 17:49:05.078883 7f07b9f83700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2
2018-01-17 17:49:05.079002 7f07b9f83700 2 req 13:0.001680:s3:PUT /found1/ttm1:copy_obj:op status=-2
2018-01-17 17:49:05.079006 7f07b9f83700 2 req 13:0.001685:s3:PUT /found1/ttm1:copy_obj:http status=404
2018-01-17 17:49:05.079011 7f07b9f83700 1 ====== req done req=0x7f07b9f7d790 op status=-2 http_status=404 ======
2018-01-17 17:49:05.079026 7f07b9f83700 20 process_request() returned -2
2018-01-17 17:49:05.079077 7f07b9f83700 1 civetweb: 0x559252072200: 10.71.84.73 - - [17/Jan/2018:17:49:05 +0800] "PUT /found1/ttm1 HTTP/1.1" 404 0 - Go-http-client/1.1
Updated by Matt Benjamin over 6 years ago
Hi Malcolm,
It's early and I havne't had coffee. Can you clarify why RGW's behavior is semantically wrong--meaning, how it produces an incorrect end result? Because I don't think the header being optional in AWS by itself forbids RGW to choose a different strategy for performing an operation if the end result is valid.
Matt
Updated by Malcolm Lee over 6 years ago
Matt Benjamin wrote:
Hi Malcolm,
It's early and I havne't had coffee. Can you clarify why RGW's behavior is semantically wrong--meaning, how it produces an incorrect end result? Because I don't think the header being optional in AWS by itself forbids RGW to choose a different strategy for performing an operation if the end result is valid.
Matt
Hi Matt,
For example, I initiate multipart, and then use copy part to copy a existing source object as a part one without "x-amz-copy-source-range"(I want to copy an entire source object), but rgw will return "HTTP 404". In rgw log,
2018-01-17 17:49:05.077551 7f07b9f83700 2 req 13:0.000230:s3:PUT /found1/ttm1:copy_obj:recalculating target
2018-01-17 17:49:05.077553 7f07b9f83700 2 req 13:0.000232:s3:PUT /found1/ttm1:copy_obj:reading permissions
2018-01-17 17:49:05.077555 7f07b9f83700 2 req 13:0.000234:s3:PUT /found1/ttm1:copy_obj:init op
2018-01-17 17:49:05.077556 7f07b9f83700 2 req 13:0.000235:s3:PUT /found1/ttm1:copy_obj:verifying op mask
2018-01-17 17:49:05.077557 7f07b9f83700 20 required_mask= 2 user.op_mask=7
2018-01-17 17:49:05.077558 7f07b9f83700 2 req 13:0.000237:s3:PUT /found1/ttm1:copy_obj:verifying op permissions
2018-01-17 17:49:05.077576 7f07b9f83700 20 get_obj_state: rctx=0x7f07b9f7d6a0
2018-01-17 17:49:05.078883 7f07b9f83700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2
2018-01-17 17:49:05.079002 7f07b9f83700 2 req 13:0.001680:s3:PUT /found1/ttm1:copy_obj:op status=-2
2018-01-17 17:49:05.079006 7f07b9f83700 2 req 13:0.001685:s3:PUT /found1/ttm1:copy_obj:http status=404
Radowsgw use copy_obj to handle copy part request, while copying part should be handled with put_obj in normal request with header "x-amz-copy-source-range".
Updated by Malcolm Lee over 6 years ago
And the syntax of "Copy Object" is
PUT /destinationObject HTTP/1.1
Host: destinationBucket.s3.amazonaws.com
x-amz-copy-source: /source_bucket/sourceObject
x-amz-metadata-directive: metadata_directive
x-amz-copy-source-if-match: etag
x-amz-copy-source-if-none-match: etag
x-amz-copy-source-if-unmodified-since: time_stamp
x-amz-copy-source-if-modified-since: time_stamp
<request metadata>
Authorization: authorization string (see Authenticating Requests (AWS Signature Version
4))
Date: date
Copy Part is
PUT /ObjectName?partNumber=PartNumber&uploadId=UploadId HTTP/1.1
Host: BucketName.s3.amazonaws.com
x-amz-copy-source: /source_bucket/sourceObject
x-amz-copy-source-range:bytes=first-last
x-amz-copy-source-if-match: etag
x-amz-copy-source-if-none-match: etag
x-amz-copy-source-if-unmodified-since: time_stamp
x-amz-copy-source-if-modified-since: time_stamp
Date: date
Authorization: authorization string
These two requests params also different.
Updated by Abhishek Lekshmanan over 6 years ago
- Status changed from New to In Progress
Updated by Matt Benjamin about 6 years ago
- Status changed from In Progress to Pending Backport
- Assignee set to Matt Benjamin
- Backport set to luminous, jewel
Updated by Nathan Cutler about 6 years ago
- Copied to Backport #22903: luminous: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object added
Updated by Nathan Cutler about 6 years ago
- Copied to Backport #22904: jewel: rgw: copying part without http header "x-amz-copy-source-range" will be mistaken for copying object added
Updated by Nathan Cutler almost 6 years ago
- Status changed from Pending Backport to Resolved