Project

General

Profile

Bug #57980

rgw/cloud-transition: transition fails when using MCG Azure Namespacestore with a pre-created bucket

Added by Soumya Koduri 3 months ago. Updated 18 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Soumya Koduri
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Reported by -

Found Errors during cloud transition when using MCG Azure Namespacestore with a pre-created bucket.

If I use pre-created buckets with AWS S3, or with an MCG AWS S3 Namespacestore, the transitions work fine, but when using an MCG Azure Namespacestore I get the following errors, and no objects are transitioned:

022-11-03T08:42:29.675+0000 7fa1bffe7640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.675+0000 7fa1bffe7640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1111 , ret = -13
2022-11-03T08:42:29.675+0000 7fa1bffe7640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to create target bucket: rgwtransition, ret:-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to create target bucket on the cloud endpoint ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to transfer object(azns/aznss1115) to the cloud endpoint(https://s3-openshift-storage.apps.ocp410.077dazopenshift.com) ret=-39
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to transition obj(key:azns/aznss1115) to cloud (r=-39)
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 0 lifecycle: ERROR: remove_expired_obj :mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1115 (39) Directory not empty wp_thrd: 2, 2
2022-11-03T08:42:29.682+0000 7fa1bf7e6640 20 lifecycle: ERROR: orule.process() returned ret=-39thread:wp_thrd: 2, 2
2022-11-03T08:42:29.707+0000 7fa1c07e8640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.707+0000 7fa1c07e8640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1113 , ret = -13
2022-11-03T08:42:29.707+0000 7fa1c07e8640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 20 lifecycle: ERROR: cloud_tier_get_object(): conn.complete_request() returned ret=-13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to fetch HEAD from cloud for obj=:mixtrans[d9c4f708-5598-4c44-9d36-849552a08c4d.174272.2]):azns/aznss1211 , ret = -13
2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13
2022-11-03T08:42:29.756+0000 7fa1bf7e6640 0 lifecycle: ERROR: fail to complete request, ret = -22

Version-Release number of selected component (if applicable):

How reproducible:

Every time

Steps to Reproduce:
1.Create in ODF a Azure namespace store
2.Create an OBC, and use the credentials of the OBC and name of the bucket in the RGW SC configuration
3.Add a LC to the bucket that uses the SC
4. Check RGW logs for failure.

Actual results:

Expected results:

For MCG Azure namespace buckets to work with RGW cloud transitions

<<<<

History

#1 Updated by Soumya Koduri 3 months ago

Few observations:

- 2022-11-03T08:42:29.718+0000 7fa1bf7e6640 0 lifecycle: ERROR: failed to check object on the cloud endpoint ret=-13

^^^ This above error is returned when using only MCG (Noobaa) endpoint. But succeeds when transitioning directly to AWS endpoint.

- 2022-11-03T08:42:29.756+0000 7fa1bf7e6640 0 lifecycle: ERROR: fail to complete request, ret = -22

^^^ whereas the object transfer is failing only for the namespace bucket on Azure but succeeds on Namespace/AWS

From the http packets -> MCG is returning 403 for HEAD request and 400 for PUT request (failed). Maybe the header fields are not proper in case of (Azure) namespace buckets.. still investigating

#2 Updated by Soumya Koduri 3 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 48761

From the http packets -> MCG is returning 403 for HEAD request and 400 for PUT request (failed). Maybe the header fields are not proper in case of (Azure) namespace buckets..

Issue1: MCG is returning 403 for HEAD request

This seems to be an issue with MCG server and is being addressed in Noobaa codebase.

Issue2: PUT request to Azure Namespace bucket fails

as per naming conventions used by Azure store (https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata, https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blob-properties-metadata), the metadata attributes names cannot contain special characters, otherwise the server fails the request as Invalid

#3 Updated by Soumya Koduri 2 months ago

  • Pull request ID deleted (48761)

#4 Updated by Casey Bodley 18 days ago

  • Status changed from Fix Under Review to Need More Info

@Soumya is there a real bug here, or can this be closed?

#5 Updated by Soumya Koduri 18 days ago

  • Status changed from Need More Info to Closed

Casey Bodley wrote:

@Soumya is there a real bug here, or can this be closed?

yes. As you stated earlier, the issues mentioned in this tracker needs to be addressed by MCG. Closing the bug. Thanks!

Also available in: Atom PDF