Project

General

Profile

Bug #36512

Lifecycle doesn't remove delete markers

Added by Sean Purdy almost 3 years ago. Updated 6 months ago.

Status:
Resolved
Priority:
High
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
nautilus, mimic, luminous
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

I have a versioned bucket with a lifecycle. Files get deleted and delete markers added. A lifecycle configuration deletes the non-current versions, but doesn't remove the delete markers.

Lifecycle configuration:

{ 
  "Rules": [
    {
      "ID": "Test expiry 2",
      "Status": "Enabled",
      "Prefix": "",
      "NoncurrentVersionExpiration": {
        "NoncurrentDays": 1
      },
      "Expiration": {
        "ExpiredObjectDeleteMarker": true
      }
    }
  ]
}

I delete a file in the bucket. Delete marker is added:

{
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "Sean Test", 
                "ID": "seantest" 
            }, 
            "IsLatest": true, 
            "VersionId": "H7pb8jPght20pKQd.b-m3nZPEvdW.Vd", 
            "Key": "00008000.txt", 
            "LastModified": "2018-10-16T15:29:48.105Z" 
        }
    ], 
    "Versions": [
        {
            "LastModified": "2018-10-16T14:36:54.717Z", 
            "VersionId": "bZEYkUaJLcgVFEZCWAOZwvpPrCHBhRt", 
            "ETag": "\"f262eaa559613b0eaaf53983ff880ec9\"", 
            "StorageClass": "STANDARD", 
            "Key": "00008000.txt", 
            "Owner": {
                "DisplayName": "Sean Test", 
                "ID": "seantest" 
            }, 
            "IsLatest": false, 
            "Size": 243
        }
    ]
}

When the lifecycle triggers, I get this:

{
    "DeleteMarkers": [
        {
            "Owner": {
                "DisplayName": "Sean Test", 
                "ID": "seantest" 
            }, 
            "IsLatest": true, 
            "VersionId": "H7pb8jPght20pKQd.b-m3nZPEvdW.Vd", 
            "Key": "00008000.txt", 
            "LastModified": "2018-10-16T15:29:48.105Z" 
        }
    ]
}

So why didn't the lifecycle remove the delete markers? The non-current version was removed.

Tested on 12.2.7 on debian stretch and 12.2.8 on debian jessie.

Thanks,

Sean Purdy


Related issues

Copied to rgw - Backport #38882: mimic: Lifecycle doesn't remove delete markers Rejected
Copied to rgw - Backport #38883: nautilus: Lifecycle doesn't remove delete markers Resolved
Copied to rgw - Backport #38884: luminous: Lifecycle doesn't remove delete markers Resolved

History

#1 Updated by Casey Bodley almost 3 years ago

  • Priority changed from Normal to High

#2 Updated by Sean Purdy almost 3 years ago

More specifically, it seems this behaviour happens only if you have autosharding enabled and a reshard has happened.

#3 Updated by Abhishek Lekshmanan over 2 years ago

  • Assignee set to Abhishek Lekshmanan

#4 Updated by Matt Benjamin over 2 years ago

  • Status changed from New to Triaged

#5 Updated by Abhishek Lekshmanan over 2 years ago

  • Status changed from Triaged to Fix Under Review

#6 Updated by Casey Bodley over 2 years ago

  • Status changed from Fix Under Review to 7

#7 Updated by Sage Weil over 2 years ago

  • Backport set to nautilus

#8 Updated by Abhishek Lekshmanan over 2 years ago

  • Backport changed from nautilus to nautilus, mimic, luminous

#9 Updated by Casey Bodley over 2 years ago

  • Status changed from 7 to Pending Backport

#10 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #38882: mimic: Lifecycle doesn't remove delete markers added

#11 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #38883: nautilus: Lifecycle doesn't remove delete markers added

#12 Updated by Nathan Cutler over 2 years ago

  • Copied to Backport #38884: luminous: Lifecycle doesn't remove delete markers added

#13 Updated by Nathan Cutler over 2 years ago

  • Pull request ID set to 26564

#14 Updated by Nathan Cutler about 2 years ago

Backporting note from https://github.com/ceph/ceph/pull/29122

Since it's almost not possible to backport the entire PR #26564. The dependencies make it difficult and not proper to back port all of them.
I only backported the 2 commits that fixes the lifecycle not taking effect after resharding.
I think this is the main pain for ceph user, as you can't expect lifecycle to work with dynamic resharding enabled.

The other part of PR #26564, is about adding a command to radosgw-admin "radosgw-admin lc reshard fix" which can re-add the lifecycle to the old bucket.
This part is not proper for backport.
And user can always reset their lifecycle to the old bucket via commands like "s3cmd setlifecycle" as a workaround. I think we can leave this part for now.

#15 Updated by Nathan Cutler 6 months ago

  • Status changed from Pending Backport to Resolved

While running with --resolve-parent, the script "backport-create-issue" noticed that all backports of this issue are in status "Resolved" or "Rejected".

Also available in: Atom PDF