Bug #16712: multisite: 400-error with certain complete multipart upload requests - rgw - Ceph

Actions

Copy link

Bug #16712

closed

multisite: 400-error with certain complete multipart upload requests

Added by Abhishek Lekshmanan almost 8 years ago. Updated over 7 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Abhishek Lekshmanan

Target version:

% Done:

Source:

other

Tags:

Backport:

jewel

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

Crash signature (v1):

Crash signature (v2):

Description

Sometimes complete multipart upload returns a 400 error, saying incomplete parts, even though the client has got a 200 status for each of the multipart upload in a remote site. For eg. in a multipart upload with 7 parts, with eacho f the 7 parts getting a 200 success, complete_multipart request returns a 400, The rados objects on disk look like the following when (& after) the complete multipart is issued,

$ ./bin/rados -c ./run/c3/ceph.conf -p us-1.rgw.buckets.non-ec listomapkeys 3d3b1eaa-83d7-4670-ac72-cb3f29aae547.4237.4__multipart_object_4.2~pmDIYYCpr6S1QptPFjKxfwllVFBy3AD.meta
part.00000002
part.00000003
part.00000004
part.00000005
part.00000006
part.00000007

$./bin/rados -c ./run/c3/ceph.conf -p us-3.rgw.buckets.non-ec listomapkeys 3d3b1eaa-83d7-4670-ac72-cb3f29aae547.4237.4__multipart_object_4.2~pmDIYYCpr6S1QptPFjKxfwllVFBy3AD.meta
part.00000001

the multipart upload only finds 6 parts because the buckets non-ec pool in the third zone only sees the parts 2-7 and doesn't see part1

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Abhishek Lekshmanan almost 8 years ago

Subject changed from mulisite: issue with certain multipart uploads to mulisite: 400-error with certain complete multipart upload requests

Actions

Copy link

Updated by Abhishek Lekshmanan almost 8 years ago

Subject changed from mulisite: 400-error with certain complete multipart upload requests to multisite: 400-error with certain complete multipart upload requests

Actions

Copy link

Updated by Casey Bodley almost 8 years ago

Abhishek Lekshmanan wrote:

the multipart upload only finds 6 parts because the buckets non-ec pool in the third zone only sees the parts 2-7 and doesn't see part1

Presumably, us-3.rgw.buckets.non-ec is the correct pool for zone us-3 to be writing its parts to. The us-1.rgw.buckets.non-ec pool should belong to zone us-1 and only exist within cluster c1, right? So it looks like part1 is placed correctly, while the rest are not. Would it be possible to provide a log for the gateway in question?

Actions

Copy link

Updated by Abhishek Lekshmanan almost 8 years ago

uploaded via ceph-post-file at fbb3cff0-2dcb-42c3-ba12-68e18472415c

Actions

Copy link

Updated by Abhishek Lekshmanan almost 8 years ago

looking at the rgw.meta pool object for the corresponding bucket, we actually store the data_extra_pool as the primary zone's data_extra pool, not sure who/how this value was changed

Actions

Copy link

Updated by Abhishek Lekshmanan almost 8 years ago

also the value of bucket instance at this point (in domain root pool) also says the data extra pool as us-1.rgw.buckets.non-ec (ie. the primary zone non-ec pool)

Actions

Copy link