Ceph » rgw

For Server side copy object s3 request, primary zone rgw copies the object from source bucket, however the secondary zone rgw while replicating the object, instead of copying it from the destination bucket in secondary zone, it pulls the entire object from primary zone.

Its a common practice for Hadoop and Spark workflows to perform many server side copy operations with the file size being huge, as a result the secondary zone replication lag increases and also coz of pulling object from primary zone, it often causes inter zone link saturation for large objects.

Below are the ops log that show the operations, where the copy object during replication is pulling the object from primary zone (bytes_sent field for the original copy-object PUT request and replication request highlights the object is being pulled form primary zone)

**** ORIGINAL COPY OBJECT REQUEST ******
{"bucket":"server-copy2","time":"2023-06-26T12:17:29.562045Z","time_local":"2023-06-26T08:17:29.562045-0400","remote_addr":"****","user":"cosbench-001","operation":"copy_obj","uri":"PUT /server-copy2/test.img HTTP/1.1","http_status":"200","error_code":"","bytes_sent":253,"bytes_received":0,"object_size":0,"total_time":250,"user_agent":"aws-cli/2.12.3 Python/3.11.4 Linux/3.10.0-1160.59.1.el7.x86_64 exe/x86_64.rhel.7 prompt/off command/s3api.copy-object","referrer":"","trans_id":"tx000008268141cf462b70a-00649981d9-1553-dev-zone-bcc-master","authentication_type":"Local","access_key_id":"THISISBCSTESTUSER001","temp_url":false}

**** Replication user getting the DATALOG for the server side copy object ******
{"bucket":"server-copy2","time":"2023-06-26T12:17:49.084243Z","time_local":"2023-06-26T08:17:49.084243-0400","remote_addr":"****","user":"repl-user","operation":"list_bucket","uri":"GET /server-copy2?versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=0e62e7b1-0cc2-4095-8389-cfceb7f2b507?versions&format=json&objs-container=true&key-marker&version-id-marker&rgwx-zonegroup=0e62e7b1-0cc2-4095-8389-cfceb7f2b507 HTTP/1.1","http_status":"200","error_code":"","bytes_sent":538,"bytes_received":0,"object_size":0,"total_time":11,"user_agent":"","referrer":"","trans_id":"tx00000e540cd076da9d7f5-00649981ed-178f-dev-zone-bcc-master","authentication_type":"Local","access_key_id":"THISISBCSREPLUSER","temp_url":false}

**** Replication user getting the server side copy object from PRIMARY zone ******
{"bucket":"server-copy2","time":"2023-06-26T12:17:49.098243Z","time_local":"2023-06-26T08:17:49.098243-0400","remote_addr":"****","object_owner":"cosbench-001","user":"repl-user","operation":"get_obj","uri":"GET /server-copy2/test.img?rgwx-zonegroup=0e62e7b1-0cc2-4095-8389-cfceb7f2b507&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt?rgwx-zonegroup=0e62e7b1-0cc2-4095-8389-cfceb7f2b507&rgwx-prepend-metadata=true&rgwx-sync-manifest&rgwx-skip-decrypt HTTP/1.1","http_status":"200","error_code":"","bytes_sent":1073743817,"bytes_received":0,"object_size":1073741824,"total_time":4367,"user_agent":"","referrer":"","trans_id":"tx000002f1dabb4cd71f4f2-00649981ed-1559-dev-zone-bcc-master","authentication_type":"Local","access_key_id":"THISISBCSREPLUSER","temp_url":false}