Ceph » rgw

Perhaps a known issue but couldn't find this in the tracker. As the title suggests, if a user uploads an object with the key "." (i.e., a single dot/period character) or "..", it won't get replicated to the secondary zone. Other cases, like a key more than 2 dots or any other key starts with dot (e.g., ".file"), are fine.

It's easily reproducible:

$ aws s3api put-object --key=. --bucket=<bucket> --body <file>
$ aws s3api put-object --key=.. --bucket=<bucket> --body <file>

create the objects properly on the primary site but these objects won't get replicated to the secondary site. User can download the objects from the primary site using aws cli with no issues.

Secondary site would emit events like

...RGW-SYNC:data:sync:shard[..] ... entry[.]: ERROR: failed to sync object: <bucket_instance>:<datalog_shard>/.

and sync status would show the impacted shard as recovering

$ sudo radosgw-admin sync status
  ...
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: ...
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        1 shards are recovering
                        recovering shards: [90]

After some further investigation, I believe this is an issue with the curl library ceph uses so not directly a ceph issue. It can even be reproduced using curl tool as well:

# curl strips of single dot in the end in the uri
$ curl http://<rgw_ip>/.
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

# curl strips of double dots in the end in the uri
$ curl http://127.0.0.1:8101/..
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

3 dots are not removed, though:

$ curl http://<rgw_ip>/...
<?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message></Message><BucketName>...</BucketName><RequestId>tx000000e4e811a4ec2317c-0065c53177-4183-zg1-1</RequestId><HostId>4183-zg1-1-zg1</HostId></Error>

https://github.com/curl/curl/issues/716 seems to be describing the ~same issue.

I was initially suspicious that ceph would be trimming off the "dot" character (either at the sender/client side or at the receiver/master side) but uri is correctly formed before handing it off the curl library for delivery and tcp dump shows that "dot" char is trimmed off by the sender side.

(gdb) n                                                                                                                 409       r = (*req)->send(nullptr);                                                                                                                (gdb) list 409
404
405       if (!send) {
406         return 0;
407       }
408
409       r = (*req)->send(nullptr);
410       if (r < 0) {
411         goto done_err;
412       }
413       return 0;

(gdb) p (*req)->url
$172 = "http://localhost:8101/u1b1/.?rgwx-zonegroup=73adcd35-2eb3-4189-aabb-e54e03e14376&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=4afef25"...

tcpdump shows that "." is dropped at the delivery:

CÐ:<93>CÐ^_(GET /u1b1/?rgwx-zonegroup=73adcd35-2eb3-4189-aabb-e54e03e14376&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-sync-cloudtiered&rgwx-skip-decrypt&rgwx-if-not-replicated-to=4afef25f-55b0-4e1e-af93-88bc18eae9f3%3Au1b1%3A44ada735-7336-4e43-b03d-2d0840e947de.4183.1 HTTP/1.1^M
Host: localhost:8101^M
Accept: */*^M
Authorization: AWS4-HMAC-SHA256 Credential=1234567890/20240208/zg1/s3/aws4_request,SignedHeaders=date;host;x-amz-content-sha256;x-amz-date,Signature=9111ecebfdb73a110b3d55446323dc326d96f0f62c3e4d8609662d239964e02e^M
Date: Thu, 08 Feb 2024 17:16:10 +0000^M
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855^M
X-Amz-Date: 20240208T171610Z^M