Bug #42358
closedrgw: reduce per-shard entry count during ordered bucket listing
0%
Description
Currently, if someone requests the 1000 next entries from a bucket, each bucket index shard will receive a request for the 1000 next entries. When there are hundreds, thousands, or tens of thousands of bucket index shards, this results in a huge amplification of the request, even though only 1000 entries will be returned.
These changes reduce the per-bucket index shard requests. These also allow re-requests in edge cases where all of one shard's returned entries are consumed. Finally these changes improve the determination of whether the resulting list is truncated.
Updated by J. Eric Ivancich over 4 years ago
- Status changed from In Progress to Pending Backport
Updated by Beom-Seok Park over 4 years ago
This PR breaks the s3 client and radosgw-admin if there are a lot of uncomplete multiparts in the bucket.
Create a bucket.
$ s3cmd mb s3://test-bucket
Create over 150 uncomplete multiparts.
$ s3cmd -c testing.s3cfg put 1GB s3://test-bucket --multipart-chunk-size-mb=5
upload: '1GB' -> 's3://test-bucket/1GB' [part 1 of 205, 5MB] [1 of 1]
5242880 of 5242880 100% in 0s 27.60 MB/s done
...
upload: '1GB' -> 's3://test-bucket/1GB' [part 151 of 205, 5MB] [1 of 1]
5242880 of 5242880 100% in 0s 37.73 MB/s done
upload: '1GB' -> 's3://test-bucket/1GB' [part 152 of 205, 5MB] [1 of 1]
65536 of 5242880 1% in 0s 3.17 MB/s^CERROR:
Upload of '1GB' part 152 failed. Use
/usr/bin/s3cmd abortmp s3://test-bucket/1GB 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah
to abort the upload, or
/usr/bin/s3cmd --upload-id 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah put ...
to continue the upload.
See ya!
s3cmd shows the following error:
$ s3cmd ls s3://test-bucket --debug
...
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>test-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated></ListBucketResult>',
...
Invoked as: /usr/bin/s3cmd ls s3://test-bucket --debug
Problem: IndexError: list index out of range
...
Could not delete bucket
# radosgw-admin bucket rm --purge-objects --bypass-gc --bucket=test-bucket
The same message is repeated over and over.
2019-12-06 22:48:47.446570 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.448804 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.451057 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.453403 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.456087 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.458373 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
Updated by J. Eric Ivancich over 4 years ago
Thank you, Beom-Seok Park!
I've created a new tracker associated with the bug fix.
Updated by J. Eric Ivancich over 4 years ago
- Status changed from Pending Backport to Fix Under Review
This should not be backported until https://tracker.ceph.com/issues/43180 is resolved, and they should then be backported as one unit.
Updated by Nathan Cutler over 4 years ago
- Status changed from Fix Under Review to Pending Backport
Updated by Nathan Cutler over 4 years ago
- Precedes Bug #43180: rgw: ordered listing of bucket with many incomplete multipart uploads fails added
Updated by Nathan Cutler over 4 years ago
- Status changed from Pending Backport to Resolved
- Backport deleted (
nautilus,mimic)
Backport will be handled via #43180 (follow-on fix for this one)