Bug #63438: RGW: Cloud sync module fails to sync folders - rgw - Ceph

Actions

Copy link

Bug #63438

open

RGW: Cloud sync module fails to sync folders

Added by Gabriel Adrian Samfira 6 months ago. Updated 3 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Target version:

% Done:

Source:

Tags:

cloud-sync backport_processed

Backport:

reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

54311

Crash signature (v1):

Crash signature (v2):

Description

We've stumbled across this bug while trying to sync a bucket with folders inside it, against a MinIO
instance. The logs showed that the requests being made to the primary rados gateway instance, included
an object path, where the slash was being escaped into %2F. This resulted in a 404 error which bubbled
up into an ENOENT error in the cloud sync module. The relevant snippet of the log is as follows:

2023-11-02T15:36:25.061+0000 7f3e10fae640 20 http_client[GET/https://10.10.10.112:443/17e3030a7fd04956b29d4f95e09f752c%3Agsamfira-test3/testfolder%2Fvirtio-win10-prewhql-0.0-100-sources.zip?rgwx-zonegroup=119cc5ef-a452-44de-a895-4b9de478540d&rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-man
ifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt]sign_request_v4(): sigv4 header: x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
2023-11-02T15:36:25.061+0000 7f3e10fae640 20 http_client[GET/https://10.10.10.112:443/17e3030a7fd04956b29d4f95e09f752c%3Agsamfira-test3/testfolder%2Fvirtio-win10-prewhql-0.0-100-sources.zip?rgwx-zonegroup=119cc5ef-a452-44de-a895-4b9de478540d&rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-man
ifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt]sign_request_v4(): sigv4 header: x-amz-date: 20231102T153625Z
2023-11-02T15:36:25.061+0000 7f3e10fae640 20 sending request to https://10.10.10.112:443/17e3030a7fd04956b29d4f95e09f752c%3Agsamfira-test3/testfolder%2Fvirtio-win10-prewhql-0.0-100-sources.zip?rgwx-zonegroup=119cc5ef-a452-44de-a895-4b9de478540d&rgwx-prepend-metadata=true&rgwx-stat=true&rgwx-sync-
manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt
2023-11-02T15:36:25.061+0000 7f3e10fae640 20 register_request mgr=0x5556785d3200 req_data->id=2, curl_handle=0x55567d5b5880
2023-11-02T15:36:25.061+0000 7f3e167b9640 20 link_request req_data=0x55567d53e5a0 req_data->id=2, curl_handle=0x55567d5b5880
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c0d5cc0:op=0x55567ca7ca00:27RGWReadRemoteDataLogShardCR: operate()
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c0d5cc0:op=0x55567d5f5e00:21RGWDataIncSyncShardCR: operate()
2023-11-02T15:36:25.065+0000 7f3df977f640 20 RGW-SYNC:data:sync:shard[120]: shard_id=120 sync_marker=00000000000000000000:00000000000000000685 next_marker= truncated=0
2023-11-02T15:36:25.065+0000 7f3df977f640 20 run: stack=0x55567c0d5cc0 is io blocked
2023-11-02T15:36:25.065+0000 7f3e10fae640  0 rgw async rados processor: store->stat_remote_obj() returned r=-2
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c9f5540:op=0x55567d430000:18RGWStatRemoteObjCR: operate()
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c9f5540:op=0x55567d430000:18RGWStatRemoteObjCR: operate() returned r=-2
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c9f5540:op=0x555695084800:23RGWAWSHandleRemoteObjCR: operate()
2023-11-02T15:36:25.065+0000 7f3df977f640 10 rgw rados thread: RGWStatRemoteObjCR() returned -2
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c9f5540:op=0x555695084800:23RGWAWSHandleRemoteObjCR: operate() returned r=-2
2023-11-02T15:36:25.065+0000 7f3df977f640 20 rgw rados thread: cr:s=0x55567c9f5540:op=0x55568379a000:26RGWBucketSyncSingleEntryCRINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE11rgw_obj_keyE: operate()
2023-11-02T15:36:25.065+0000 7f3df977f640 10 RGW-SYNC:data:sync:shard[45]:entry[17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7[0]]:bucket_sync_sources[source=17e3030a7fd04956b29d4f95e09f752c:gsamfira-test3[dae6f793-a9b9-4933-9d65-3e516ef20753.63322
61.1]):7:source_zone=dae6f793-a9b9-4933-9d65-3e516ef20753]:bucket[17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1<-17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7]:inc_sync[17e3030a7fd04956b29d4f95e09f7
52c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7]:entry[testfolder/virtio-win10-prewhql-0.0-100-sources.zip]: failed, retcode=-2 ((2) No such file or directory)
2023-11-02T15:36:25.065+0000 7f3df977f640 20 RGW-SYNC:data:sync:shard[45]:entry[17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7[0]]:bucket_sync_sources[source=17e3030a7fd04956b29d4f95e09f752c:gsamfira-test3[dae6f793-a9b9-4933-9d65-3e516ef20753.63322
61.1]):7:source_zone=dae6f793-a9b9-4933-9d65-3e516ef20753]:bucket[17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1<-17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7]:inc_sync[17e3030a7fd04956b29d4f95e09f7
52c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7]: updating marker marker_oid=bucket.sync-status.dae6f793-a9b9-4933-9d65-3e516ef20753:17e3030a7fd04956b29d4f95e09f752c/gsamfira-test3:dae6f793-a9b9-4933-9d65-3e516ef20753.6332261.1:7 marker=00000000001.2.6 timestamp=2023-11-02T15:3
5:56.032657+0000

The bug seems to be in the send_prepare_convert() function which is called by RGWRESTStreamRWRequest and is cause by the fact that slashes are being url encoded.

Related issues 1 (1 open — 0 closed)