Ceph » rgw

We are observing lifecycle policy for certain buckets remains stuck in the UNINITIAL state, when the LC transition policy is applied on multiple buckets.

There are a total of buckets 428 buckets in the LC list, out of which 5 are stuck in the UNINITIAL state for a few days. All belong to the same lc shard of lc.4

-------------- snippet of the buckets stuck ----------

"status": "COMPLETE" 
    },
{
        "bucket": ":lucileg.356-bucky-4188-12:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.208655.6",
        "shard": "lc.4",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL" 
    },
{
        "bucket": ":lucileg.356-bucky-4188-31:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.218596.16",
        "shard": "lc.4",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL" 
    },
{
        "bucket": ":lucileg.356-bucky-4188-42:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.208655.24",
        "shard": "lc.4",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL" 
    },
{
        "bucket": ":lucileg.356-bucky-4188-63:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.218596.32",
        "shard": "lc.4",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL" 
    },
{
        "bucket": ":lucileg.356-bucky-4188-95:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.218596.47",
        "shard": "lc.4",
        "started": "Thu, 01 Jan 1970 00:00:00 GMT",
        "status": "UNINITIAL" 
    },
{
        "bucket": ":maryx.267-bucky-4525-37:1e614482-6c1c-49cb-81ef-8fb70c1ff4e8.133978.25",
        "shard": "lc.4",
        "started": "Wed, 21 Feb 2024 11:08:55 GMT",
        "status": "COMPLETE" 
    }
How reproducible:

Seen intermittently when LC is applied simultaneously on multiple buckets (100 at a time.)

Reproducibility: 2/3 times.

Steps to Reproduce:

1. Create around 100 buckets and upload 10 objects per bucket.
2. Apply an LC rule on all the buckets to transition the objects to a 'cold' storage class.
3. Repeat the test (steps 1 and 2) around 2-3 times.

Actual results:

We observed that 5 buckets are stuck in a UNINITIAL state for a few days.

Expected results:

LC process should not be stuck in a UNINITIAL state.