Bug #59106: rgw/archive: Sync logs are not trimmed - rgw - Ceph

Actions

Copy link

Bug #59106

open

rgw/archive: Sync logs are not trimmed

Added by Soumya Koduri about 1 year ago. Updated 12 months ago.

Status:

Pending Backport

Priority:

Normal

Assignee:

Soumya Koduri

Target version:

% Done:

Source:

Tags:

archive backport_processed

Backport:

pacific quincy reef

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

50676

Crash signature (v1):

Crash signature (v2):

Description

The sync logs generated by the archive zone (which is not a source zone) are never trimmed. However this issue doesn't seem to arise for the other zones excluded (as source/dest) from the sync using the Sync Policy rules.

In addition, "bucket sync status" command incorrectly shows as 'shards behind' from the archive zone (though the sync is disabled).

[root@localhost build]# ./bin/radosgw-admin sync status  --rgw-zone=default
    2023-03-20T09:11:13.085+0530 7f87fe43c9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-20T09:11:13.091+0530 7f87fe43c9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-20T09:11:13.091+0530 7f87fe43c9c0 -1 WARNING: all dangerous and experimental features are enabled.
          realm d9011ad1-d38c-4335-a1f5-4693a119711a (multisite)
          zonegroup d6fc5b57-b338-4748-a57a-fbb81f1ad8f1 (default)
           zone 6d71e705-d117-4ee7-9a42-cf7d8294be69 (default)
       current time 2023-03-20T03:41:13Z
    zonegroup features enabled: resharding
      metadata sync no sync (zone is master)
          data sync source: cd15cd31-c06f-4fa8-beac-f1b044f4acf8 (archive-zone)
                        not syncing from zone
                source: ec8571ac-fa84-44e9-976d-87e3d5f1833f (multisite-zone)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source
    [root@localhost build]#

[root@localhost build]# ./bin/radosgw-admin bucket sync status --bucket=bucket1 --rgw-zone=default
    2023-03-20T09:10:01.864+0530 7f4a71a0d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-20T09:10:01.870+0530 7f4a71a0d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-20T09:10:01.871+0530 7f4a71a0d9c0 -1 WARNING: all dangerous and experimental features are enabled.
          realm d9011ad1-d38c-4335-a1f5-4693a119711a (multisite)
          zonegroup d6fc5b57-b338-4748-a57a-fbb81f1ad8f1 (default)
           zone 6d71e705-d117-4ee7-9a42-cf7d8294be69 (default)
         bucket :bucket1[6d71e705-d117-4ee7-9a42-cf7d8294be69.14193.1])
       current time 2023-03-20T03:40:01Z

source zone cd15cd31-c06f-4fa8-beac-f1b044f4acf8 (archive-zone)
      source bucket :bucket1[6d71e705-d117-4ee7-9a42-cf7d8294be69.14193.1])
                incremental sync on 11 shards
                bucket is behind on 2 shards
                behind shards: [2,10]
        source zone ec8571ac-fa84-44e9-976d-87e3d5f1833f (multisite-zone)
      source bucket :bucket1[6d71e705-d117-4ee7-9a42-cf7d8294be69.14193.1])
                incremental sync on 11 shards
                bucket is caught up with source
    [root@localhost build]#

Related issues 3 (1 open — 2 closed)

Actions

Copy link

Updated by Soumya Koduri about 1 year ago

Status changed from New to In Progress
Assignee set to Soumya Koduri

Actions

Copy link

Updated by Soumya Koduri about 1 year ago

Status changed from In Progress to Fix Under Review
Pull request ID set to 50676

The fix posted (https://github.com/ceph/ceph/pull/50676) is to disable logging for the archive sync module as it is not used as source for the incremental sync. Also it seems right not to create sync pipe from the module with logging disabled to other zones. This fixes the sync info and bucket sync status cmd output.

{
        "id": "cd3ef980-ac0f-4cee-b0d7-8642a267d367",
        "name": "archive-zone",
        "endpoints": [
            "http://localhost:8002" 
        ],
        "log_meta": false,
        "log_data": false,
        "bucket_index_max_shards": 11,
        "read_only": false,
        "tier_type": "archive",
        "sync_from_all": true,
        "sync_from": [],
        "redirect_zone": "",
        "supported_features": [
            "resharding" 
        ]
    },

./bin/radosgw-admin bucket sync status --bucket=bucket1 --rgw-zone=archive-zone
      2023-03-26T19:21:00.455+0530 7fe3373689c0 -1 WARNING: all dangerous and experimental features are enabled.
      2023-03-26T19:21:00.460+0530 7fe3373689c0 -1 WARNING: all dangerous and experimental features are enabled.
      2023-03-26T19:21:00.461+0530 7fe3373689c0 -1 WARNING: all dangerous and experimental features are enabled.
                realm 0d99a168-6363-450b-adef-40b91058b95d (multisite)
            zonegroup 1b25bc3b-03fb-4303-99a0-bdf4a11e4aac (default)
                 zone cd3ef980-ac0f-4cee-b0d7-8642a267d367 (archive-zone)
               bucket :bucket1[ed4df705-5157-4302-aa1f-1e5264d8904f.14139.1])
         current time 2023-03-26T13:51:00Z      
          source zone c21ca1fe-2804-4392-a002-df7b285e2c69 (multisite-zone)
        source bucket :bucket1[ed4df705-5157-4302-aa1f-1e5264d8904f.14139.1])
                      incremental sync on 11 shards
                      bucket is caught up with source
          source zone ed4df705-5157-4302-aa1f-1e5264d8904f (default)
        source bucket :bucket1[ed4df705-5157-4302-aa1f-1e5264d8904f.14139.1])
                      incremental sync on 11 shards
                      bucket is caught up with source

./bin/radosgw-admin bucket sync status --bucket=bucket1 --rgw-zone=default
      2023-03-26T19:19:23.655+0530 7fd4bbf599c0 -1 WARNING: all dangerous and experimental features are enabled.
      2023-03-26T19:19:23.661+0530 7fd4bbf599c0 -1 WARNING: all dangerous and experimental features are enabled.
      2023-03-26T19:19:23.661+0530 7fd4bbf599c0 -1 WARNING: all dangerous and experimental features are enabled.
                realm 0d99a168-6363-450b-adef-40b91058b95d (multisite)
            zonegroup 1b25bc3b-03fb-4303-99a0-bdf4a11e4aac (default)
                 zone ed4df705-5157-4302-aa1f-1e5264d8904f (default)
               bucket :bucket1[ed4df705-5157-4302-aa1f-1e5264d8904f.14139.1])
         current time 2023-03-26T13:49:23Z      
          source zone c21ca1fe-2804-4392-a002-df7b285e2c69 (multisite-zone)
        source bucket :bucket1[ed4df705-5157-4302-aa1f-1e5264d8904f.14139.1])
                      incremental sync on 11 shards
                      bucket is caught up with source

./bin/radosgw-admin sync info --rgw-zone=default 
    2023-03-26T18:59:07.969+0530 7f3ebff6d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-26T18:59:07.974+0530 7f3ebff6d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-26T18:59:07.974+0530 7f3ebff6d9c0 -1 WARNING: all dangerous and experimental features are enabled.
{
        "sources": [
    {
            "id": "all",
            "source": {
                "zone": "multisite-zone",
                "bucket": "*" 
            },
            "dest": {
                "zone": "default",
                "bucket": "*" 
            },
            "params": {
                "source": {
                    "filter": {
                        "tags": []
                    }
                },
                "dest": {},
                "priority": 0,
                "mode": "system",
                "user": "" 
            }
        }
        ],
        "dests": [
    {
            "id": "all",
            "source": {
                "zone": "default",
                "bucket": "*" 
            },
            "dest": {
                "zone": "multisite-zone",
                "bucket": "*" 
            },
            "params": {
                "source": {
                    "filter": {
                        "tags": []
                    }
                },
                "dest": {},
                "priority": 0,
                "mode": "system",
                "user": "" 
            }
        },
    {
            "id": "all",
            "source": {
                "zone": "default",
                "bucket": "*" 
            },
            "dest": {
                "zone": "archive-zone",
                "bucket": "*" 
            },
            "params": {
                "source": {
                    "filter": {
                        "tags": []
                    }
                },
                "dest": {},
                "priority": 0,
                "mode": "system",
                "user": "" 
            }
        }
        ],
        "hints": {
        "sources": [],
        "dests": []
        },
        "resolved-hints-1": {
        "sources": [],
        "dests": []
        },
        "resolved-hints": {
        "sources": [],
        "dests": []
        }
    }

./bin/radosgw-admin sync info --rgw-zone=archive-zone
    2023-03-26T19:12:53.235+0530 7fbc5f20d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-26T19:12:53.253+0530 7fbc5f20d9c0 -1 WARNING: all dangerous and experimental features are enabled.
    2023-03-26T19:12:53.254+0530 7fbc5f20d9c0 -1 WARNING: all dangerous and experimental features are enabled.
{
        "sources": [
    {
            "id": "all",
            "source": {
                "zone": "multisite-zone",
                "bucket": "*" 
            },
            "dest": {
                "zone": "archive-zone",
                "bucket": "*" 
            },
            "params": {
                "source": {
                    "filter": {
                        "tags": []
                    }
                },
                "dest": {},
                "priority": 0,
                "mode": "system",
                "user": "" 
            }
        },
    {
            "id": "all",
            "source": {
                "zone": "default",
                "bucket": "*" 
            },
            "dest": {
                "zone": "archive-zone",
                "bucket": "*" 
            },
            "params": {
                "source": {
                    "filter": {
                        "tags": []
                    }
                },
                "dest": {},
                "priority": 0,
                "mode": "system",
                "user": "" 
            }
        }
        ],
        "dests": [],
        "hints": {
        "sources": [],
        "dests": []
        },
        "resolved-hints-1": {
        "sources": [],
        "dests": []
        },
        "resolved-hints": {
        "sources": [],
        "dests": []
        }
    }