Project

General

Profile

Feature #54476

rgw: allow S3 delete-marker behavior to be restored via config

Added by J. Eric Ivancich 9 months ago. Updated 8 months ago.

Status:
Fix Under Review
Priority:
Normal
Target version:
% Done:

0%

Source:
Tags:
Backport:
quincy,pacific,octopus
Reviewed:
Affected Versions:
Pull request ID:

Description

According to Amazon's S3 documentation on versioned buckets, when the most recent version of an object is a delete-marker, a delete operation should add another delete-marker and make it the latest version. See: https://docs.aws.amazon.com/AmazonS3/latest/userguide/ManagingDelMarkers.html .

It's unclear what advantage this offers as opposed to simply not altering the bucket index at all. Adding delete-markers increases the size of the bucket index, a distinct disadvantage. We're aware that some users of RGW have hundreds of thousands of consecutive delete-markers, likely due to tooling that issues deletes ("just in case"?).

Q: Can anyone come up with an advantage to this way of handling delete-markers?

If the user of a bucket is willing to clean up older versions using lifecycle, then these "redundant" delete-markers will be cleaned up as part of the process (. But otherwise Ceph RGW does not currently offer any tooling to do so.

External scripting could be created to handle the clean-up (e.g., see: https://wasabi-support.zendesk.com/hc/en-us/articles/360060003551-How-do-I-mass-delete-Delete-Markers-inside-a-bucket-).

So should Ceph RGW offer a configurable option so as not to create these "redundant" delete-markers? And if so, which default behavior should Ceph RGW exhibit.

Secondarily, other than external scripting, should Ceph RGW offer any support for this? `radosgw-admin bucket check --fix...` does clean up bucket indices, but should it handle a situation such as this as well?


Related issues

Related to rgw - Bug #40393: Lifecycle expiration action generates delete marker continuously Resolved

History

#1 Updated by J. Eric Ivancich 9 months ago

I thought I'd see if anyone knows why this is S3's behavior and can think of any adverse behavior that would result from not creating a delete-marker on top of another delete-marker.

#2 Updated by J. Eric Ivancich 9 months ago

  • Subject changed from rgw: optionally diverge from S3's delete-marker behavior to rgw: optionally diverge from S3's "stacked" delete-marker behavior

#3 Updated by Casey Bodley 9 months ago

  • Related to Bug #40393: Lifecycle expiration action generates delete marker continuously added

#4 Updated by Casey Bodley 9 months ago

J. Eric Ivancich wrote:

I thought I'd see if anyone knows why this is S3's behavior and can think of any adverse behavior that would result from not creating a delete-marker on top of another delete-marker.

i'm not exactly sure. but one consideration is that the s3 delete op returns the version ID of the delete marker it creates, allowing the client to delete that particular delete marker later. in this way, a client can temporarily 'hide' the presence of an object

and given the current design, several clients could coordinate this way, such that the object is only visible if no client is currently hiding it. that behavior might be useful in some application

We're aware that some users of RGW have hundreds of thousands of consecutive delete-markers, likely due to tooling that issues deletes ("just in case"?).

i was under the impression that these cases were specific to a lifecycle bug, fixed in https://tracker.ceph.com/issues/40393. it might be useful to distinguish these two cases (lifecycle expirations vs. s3 deletes) in this discussion

regarding lifecycle, i believe it gives sufficient control over what happens to delete markers through https://docs.aws.amazon.com/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html and the ExpiredObjectDeleteMarker field of https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleExpiration.html

Secondarily, other than external scripting, should Ceph RGW offer any support for this? `radosgw-admin bucket check --fix...` does clean up bucket indices, but should it handle a situation such as this as well?

in the case of 'bucket check' tooling, i don't think we can distinguish between the delete markers that were created by that lifecycle bug vs. those created intentionally via s3 deletes

but i think it's much easier to choose a sensible default in this tooling, than it is to change how delete markers work globally. i think it'd be fine for 'bucket check --fix' to remove all but the most recent delete marker

#5 Updated by Soumya Koduri 8 months ago

I do not see this behaviour in the latest master. I tested using both s3 delete and by setting below lifecycle policy -
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>id1</ID>
<Prefix>s</Prefix>
<Status>Enabled</Status>
<Expiration>
<Date>2022-01-01</Date>
</Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Rule>
</LifecycleConfiguration>

(or)


<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>id1</ID>
<Prefix>s</Prefix>
<Status>Enabled</Status>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<NoncurrentVersionExpiration>
<NoncurrentDays>1</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>

Probably fixed by bug#40393?

#6 Updated by J. Eric Ivancich 8 months ago

  • Subject changed from rgw: optionally diverge from S3's "stacked" delete-marker behavior to rgw: allow S3 delete-marker behavior to be restored via config
  • Backport changed from yes, tbd to quincy,pacific,octopus
  • Pull request ID set to 45754

#7 Updated by J. Eric Ivancich 8 months ago

After reading feedback and discussing with Matt, decided to make this configurable.

Note: commit 69d7589fb1305b7d202ffd126c3c835e7cd0dda3 added the behavior to diverge from S3 documented behavior. This tracker/PR allows config to restore S3 documented behavior.

#8 Updated by J. Eric Ivancich 8 months ago

  • Status changed from Need More Info to Fix Under Review

#9 Updated by J. Eric Ivancich 8 months ago

Soumya Koduri wrote:

I do not see this behaviour in the latest master. I tested using both s3 delete and by setting below lifecycle policy -

I just tested it using the s3cmd client and it was working, Soumya. I'm not sure why it wouldn't be working under lifecycle.

Also available in: Atom PDF