Feature #42750
openswift segments don't expire
0%
Description
Objects uploaded through the swift API as segments don't expire correctly.
When doing a swift segment upload to a bucket swift leaves a 0-byte file for the upload and then all the data is stored in segments in the <bucket>_segments bucket. The hints for the object correctly include the expiration time. At the time of expiration the 0-byte file is removed and the segments are then orphaned.
This behavior has been verified on Luminous and Nautilus.
Thoughts: create hints, for each segment file as they're created so the expiration process or update the expiration process to discover the segments and process them
Updated by Casey Bodley over 4 years ago
- Status changed from New to 4
- Tags set to swift slo expiration
Does this work in openstack's swift? My understanding is that expiration and slo are both implemented as middleware, and I don't see any documentation about how they interact. Looking at the swift-object-expirer source at https://github.com/openstack/swift/blob/master/swift/obj/expirer.py#L481, it appears to be sending normal DELETE requests without the ?multipart-manifest=delete parameter that would also delete its segments.
Are you uploading with a swift client command similar to this?
swift upload container obj -S 4M -H "X-Delete-At:..."
This command only adds the expiration header to the final request that writes the manifest, and not for any of the segment uploads. That's why radosgw's object expirer doesn't consider the segments for expiration.
And while the swift client generally uploads and deletes the segments transparently, the underlying SLO api is more flexible - for example, nothing prevents two different object manifests from referring to the same segment objects, or from referring to segments that were uploaded as normal swift objects. So if the X-Delete-At header on one of those manifests expired its segments by default, that would corrupt unrelated objects.
On the other hand, if an environment uses the swift client exclusively to manage these segments, it should be safe to provide them the option to expire them along with their manifest. Maybe a new header such as 'x-rgw-expire-segments' would be a reasonable way for clients to express this intent?
Updated by Peter Eisch over 4 years ago
The environment does use the swift client exclusively and the introduction of 'x-rgw-expire-segments' would work. Openstack's swift command we use with the swift servers is:
swift upload <bucket>/<dir> -S 4294967296 --header X-Delete-After:2419200 --changed \
--skip-identical --segment-threads 20 <1.5TB-tarball>
Swift appears to protect the segments from multiple manifests by constructing a path for the segments:
... <bucket>_segments/<file>/1570778123.364884/1073741824/5368709/00000197 <bucket>_segments/<file>/1570778123.364884/1073741824/5368709/00000198 <bucket>_segments/<file>/1570778123.364884/1073741824/5368709/00000199 <bucket>_segments/<file>/1570778123.364884/1073741824/5368709/00000200 ...
Through this they appear to protect the segments from multiple manifests overlapping the segments. (Note how they use append the bucket name, include a timestamp and two other unique properties in the segment path.) I would presume this was their author's way to avoid corruption of objects.
Possible workarounds: Would replacing swift in the environment with s3 multipart uploads provide the ability to expire as swift intends? (The segments get us to maximize network transfer speeds.) Is there an RBD equivalent for expiring objects?
Updated by Casey Bodley over 4 years ago
- Has duplicate Bug #42987: can't delete segment objects in Static large objects in swift API when set X-Delete-At in head added
Updated by Casey Bodley over 4 years ago
- Tracker changed from Bug to Feature
Changing this to a feature request, as our behavior appears to match openstack's.