Bug #57980: rgw/cloud-transition: transition fails when using MCG Azure Namespacestore with a pre-created bucket - rgw - Ceph

Actions

Copy link

Bug #57980

open

rgw/cloud-transition: transition fails when using MCG Azure Namespacestore with a pre-created bucket

Added by Soumya Koduri over 1 year ago. Updated 11 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Soumya Koduri

Target version:

% Done:

Source:

Tags:

backport_processed

Backport:

quincy reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

50098

Crash signature (v1):

Crash signature (v2):

Description

Reported by - dparkes@redhat.com

Found Errors during cloud transition when using MCG Azure Namespacestore with a pre-created bucket.

If I use pre-created buckets with AWS S3, or with an MCG AWS S3 Namespacestore, the transitions work fine, but when using an MCG Azure Namespacestore I get the following errors, and no objects are transitioned:

022-11-03T08:42:29.675+0000 7fa1bffe7640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.675+0000 7fa1bffe7640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1111 , ret = -13
2022-11-03T08:42:29.675+0000 7fa1bffe7640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to create target bucket: rgwtransition, ret:-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to create target bucket on the cloud endpoint ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to transfer object(azns/aznss1115) to the cloud endpoint(https://s3-openshift-storage.apps.ocp410.077dazopenshift.com) ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to transition obj(key:azns/aznss1115) to cloud (r=-39)
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: remove_expired_obj :mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1115 (39) Directory not empty wp_thrd: 2, 2
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 20 lifecycle: ERROR: orule.process() returned ret=-39thread:wp_thrd: 2, 2
2022-11-03T08:42:29.707+0000 7fa1c07e8640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.707+0000 7fa1c07e8640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1113 , ret = -13
2022-11-03T08:42:29.707+0000 7fa1c07e8640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1211 , ret = -13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.756+0000 7fa1bf7e6640 0 lifecycle: ERROR: fail to complete request, ret = -22

Version-Release number of selected component (if applicable):

How reproducible:

Every time

Steps to Reproduce:
1.Create in ODF a Azure namespace store
2.Create an OBC, and use the credentials of the OBC and name of the bucket in the RGW SC configuration
3.Add a LC to the bucket that uses the SC
4. Check RGW logs for failure.

Actual results:

Expected results:

For MCG Azure namespace buckets to work with RGW cloud transitions

<<<<

Related issues 2 (1 open — 1 closed)

Actions

Copy link

Updated by Soumya Koduri over 1 year ago

Few observations:

- 2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13

^^^ This above error is returned when using only MCG (Noobaa) endpoint. But succeeds when transitioning directly to AWS endpoint.

- 2022-11-03T08:42:29.756+0000 7fa1bf7e6640 0 lifecycle: ERROR: fail to complete request, ret = -22

^^^ whereas the object transfer is failing only for the namespace bucket on Azure but succeeds on Namespace/AWS

From the http packets -> MCG is returning 403 for HEAD request and 400 for PUT request (failed). Maybe the header fields are not proper in case of (Azure) namespace buckets.. still investigating

Actions

Copy link

Updated by Soumya Koduri over 1 year ago

Status changed from New to Fix Under Review
Pull request ID set to 48761

From the http packets -> MCG is returning 403 for HEAD request and 400 for PUT request (failed). Maybe the header fields are not proper in case of (Azure) namespace buckets..

Issue1: MCG is returning 403 for HEAD request

This seems to be an issue with MCG server and is being addressed in Noobaa codebase.

Issue2: PUT request to Azure Namespace bucket fails

as per naming conventions used by Azure store (https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata, https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata), the metadata attributes names cannot contain special characters, otherwise the server fails the request as Invalid

Actions

Copy link

Updated by Soumya Koduri over 1 year ago

Pull request ID deleted (~~48761~~)

Actions

Copy link

Updated by Casey Bodley over 1 year ago

Status changed from Fix Under Review to Need More Info

@Soumya M S is there a real bug here, or can this be closed?

Actions

Copy link

Updated by Soumya Koduri over 1 year ago

Status changed from Need More Info to Closed

Casey Bodley wrote:

@Soumya M S is there a real bug here, or can this be closed?

yes. As you stated earlier, the issues mentioned in this tracker needs to be addressed by MCG. Closing the bug. Thanks!

Actions

Copy link

Updated by Soumya Koduri about 1 year ago

Status changed from Closed to Fix Under Review
Pull request ID set to 50098

In order to address this issue and help MCG determine the source and map the s3 style attrs, below two new attrs are being added to the objects transitioned to the cloud -

x-rgw-cloud : true/false
    (set to "true" if the object is being transitioned/synced from RGW)

x-rgw-cloud-keep-attrs : true/false

if set to default "true" , the cloud service should store all the x-amz-meta-rgwx-* attrs. If cannot be mapped/stored, the operation should fail
    if set to "false", the destination cloud can ignore such attrs and just store the object data being sent.

Actions

Copy link