Ceph » rgw

I thought I'd see if anyone knows why this is S3's behavior and can think of any adverse behavior that would result from not creating a delete-marker on top of another delete-marker.

J. Eric Ivancich wrote:

I thought I'd see if anyone knows why this is S3's behavior and can think of any adverse behavior that would result from not creating a delete-marker on top of another delete-marker.

i'm not exactly sure. but one consideration is that the s3 delete op returns the version ID of the delete marker it creates, allowing the client to delete that particular delete marker later. in this way, a client can temporarily 'hide' the presence of an object

and given the current design, several clients could coordinate this way, such that the object is only visible if no client is currently hiding it. that behavior might be useful in some application

We're aware that some users of RGW have hundreds of thousands of consecutive delete-markers, likely due to tooling that issues deletes ("just in case"?).

i was under the impression that these cases were specific to a lifecycle bug, fixed in https://tracker.ceph.com/issues/40393. it might be useful to distinguish these two cases (lifecycle expirations vs. s3 deletes) in this discussion

regarding lifecycle, i believe it gives sufficient control over what happens to delete markers through https://docs.aws.amazon.com/AmazonS3/latest/API/API_NoncurrentVersionExpiration.html and the ExpiredObjectDeleteMarker field of https://docs.aws.amazon.com/AmazonS3/latest/API/API_LifecycleExpiration.html

Secondarily, other than external scripting, should Ceph RGW offer any support for this? `radosgw-admin bucket check --fix...` does clean up bucket indices, but should it handle a situation such as this as well?

in the case of 'bucket check' tooling, i don't think we can distinguish between the delete markers that were created by that lifecycle bug vs. those created intentionally via s3 deletes

but i think it's much easier to choose a sensible default in this tooling, than it is to change how delete markers work globally. i think it'd be fine for 'bucket check --fix' to remove all but the most recent delete marker

I do not see this behaviour in the latest master. I tested using both s3 delete and by setting below lifecycle policy -
<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/&quot;>
<Rule>
<ID>id1</ID>
<Prefix>s</Prefix>
<Status>Enabled</Status>
<Expiration>
<Date>2022-01-01</Date>
</Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Rule>
</LifecycleConfiguration>

(or)

<LifecycleConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/&quot;>
<Rule>
<ID>id1</ID>
<Prefix>s</Prefix>
<Status>Enabled</Status>
<Expiration>
<ExpiredObjectDeleteMarker>true</ExpiredObjectDeleteMarker>
</Expiration>
<NoncurrentVersionExpiration>
<NoncurrentDays>1</NoncurrentDays>
</NoncurrentVersionExpiration>
</Rule>
</LifecycleConfiguration>

Probably fixed by bug#40393?

After reading feedback and discussing with Matt, decided to make this configurable.

Note: commit 69d7589fb1305b7d202ffd126c3c835e7cd0dda3 added the behavior to diverge from S3 documented behavior. This tracker/PR allows config to restore S3 documented behavior.

Soumya Koduri wrote:

I do not see this behaviour in the latest master. I tested using both s3 delete and by setting below lifecycle policy -

I just tested it using the s3cmd client and it was working, Soumya. I'm not sure why it wouldn't be working under lifecycle.