Project

General

Profile

Bug #47529

In multi-site and versioned scenarios, operations with null versionId cannot be deleted synchronously

Added by yp dai over 2 years ago. Updated over 2 years ago.

Status:
Fix Under Review
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

1.first,complete the multisite configuration, rgw1 in master zone, rgw2 in secondary zone.
2.create a bucket named bk01 in the master zone.
3.upload an object with key aa to bk01.
4.enable bk01 version, eg: aws s3api put-bucket-versioning --versioning-configuration Status=Enabled --endpoint=http://rgw1:9000 --bucket bk01.
5.upload an object with key aa to bk01 again.
6.delete version id is null obj in master zone, eg: aws s3api delete-object --endpoint=http://rgw1:9000 --bucket bk01 --key aa --version-id null
7.view in master zone: aws s3api list-object-versions --endpoint=http://rgw1:9000 --bucket bk01

{
    "Versions": [
        {
            "LastModified": "2020-09-18T05:59:55.691Z", 
            "VersionId": "67bzuCK-i73JVtPsn0U8hjPHfwgqnD3", 
            "ETag": "\"f86cb82ff889919256be6f2911c31eb8\"", 
            "StorageClass": "STANDARD", 
            "Key": "aa", 
            "Owner": {
                "DisplayName": "dyp01", 
                "ID": "dyp01" 
            }, 
            "IsLatest": true, 
            "Size": 1276
        }
    ]
}

8.view in secondary zone: aws s3api list-object-versions --endpoint=http://rgw2:9000 --bucket bk01

{
    "Versions": [
        {
            "LastModified": "2020-09-18T05:59:55.691Z", 
            "VersionId": "67bzuCK-i73JVtPsn0U8hjPHfwgqnD3", 
            "ETag": "\"f86cb82ff889919256be6f2911c31eb8\"", 
            "StorageClass": "STANDARD", 
            "Key": "aa", 
            "Owner": {
                "DisplayName": "dyp01", 
                "ID": "dyp01" 
            }, 
            "IsLatest": true, 
            "Size": 1276
        }, 
        {
            "LastModified": "2020-09-18T05:59:32.306Z", 
            "VersionId": "null", 
            "ETag": "\"f86cb82ff889919256be6f2911c31eb8\"", 
            "StorageClass": "STANDARD", 
            "Key": "aa", 
            "Owner": {
                "DisplayName": "dyp01", 
                "ID": "dyp01" 
            }, 
            "IsLatest": false, 
            "Size": 1276
        }
   ]
}

However, their output should be consistent.

History

#1 Updated by lei cao over 2 years ago

could you provide rgw log in secondary zone, there must be error log where rgw try delete the null version. Or, is there bilog for op deleting the null version in master zone?

#2 Updated by yp dai over 2 years ago

lei cao wrote:

could you provide rgw log in secondary zone, there must be error log where rgw try delete the null version. Or, is there bilog for op deleting the null version in master zone?

in the master zone, objects with ‘null’ version id can be successfully deleted.
it is also possible to manually delete objects whose version id is 'null' in the secondary zone.

the problem is that deleting the version id of'null' in the master zone will not automatically synchronize to the secondary zone.

if you can have an environment there, you can try to see if you have this problem.

all my operations here are using the same rgw user.

#3 Updated by yp dai over 2 years ago

I have tested, delete the versionId is not 'null' can be synchronized to the secondary zone.

#4 Updated by Abhishek Lekshmanan over 2 years ago

  • Assignee set to Abhishek Lekshmanan

#5 Updated by yp dai over 2 years ago

I tried to submit a PR for this problem:
https://github.com/ceph/ceph/pull/37276

#6 Updated by lei cao over 2 years ago

I find the reason, When deleting the null-versionId object in a versioned bucket of master zone, bilog could be recorded as a delete-marker like this {
"op_id": "00000000006.18143.11",
"op_tag": "000000005f7054efp6hn0sm708vfbhjs",
"op": "link_olh_del",
"object": "cl-test",
"instance": "t-4KWRqy5IEx2DeJ8N3eZI5WyYeuHMM",
"state": "complete",
"index_ver": 6,
"timestamp": "2020-09-27 09:01:35.222086215Z",
"ver": {
"pool": -1,
"epoch": 3
},
"bilog_flags": 1,
"versioned": true,
"owner": "wds",
"owner_display_name": "wds",
"zones_trace": [
"51f3e8d5-43f6-41e1-8635-6c143ba5f3da"
]
},
The instance t-4KWRqy5IEx2DeJ8N3eZI5WyYeuHMM is not a actual versionId. Instead, randomly generated for object OLH. When datasync take place in second zone, the instance is treated as versionId of object, Of course this object cann't be found. At last, operations with null versionId cannot be deleted synchronously in second zone.

#7 Updated by lei cao over 2 years ago

submit a PR for this problem too:
https://github.com/ceph/ceph/pull/37430

#8 Updated by Nathan Cutler over 2 years ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 37430

Also available in: Atom PDF