Bug #42358
rgw: reduce per-shard entry count during ordered bucket listing
0%
Description
Currently, if someone requests the 1000 next entries from a bucket, each bucket index shard will receive a request for the 1000 next entries. When there are hundreds, thousands, or tens of thousands of bucket index shards, this results in a huge amplification of the request, even though only 1000 entries will be returned.
These changes reduce the per-bucket index shard requests. These also allow re-requests in edge cases where all of one shard's returned entries are consumed. Finally these changes improve the determination of whether the resulting list is truncated.
Related issues
History
#1 Updated by J. Eric Ivancich over 4 years ago
- Status changed from In Progress to Pending Backport
#4 Updated by Beom-Seok Park over 4 years ago
This PR breaks the s3 client and radosgw-admin if there are a lot of uncomplete multiparts in the bucket.
Create a bucket.
$ s3cmd mb s3://test-bucket
Create over 150 uncomplete multiparts.
$ s3cmd -c testing.s3cfg put 1GB s3://test-bucket --multipart-chunk-size-mb=5
upload: '1GB' -> 's3://test-bucket/1GB' [part 1 of 205, 5MB] [1 of 1]
5242880 of 5242880 100% in 0s 27.60 MB/s done
...
upload: '1GB' -> 's3://test-bucket/1GB' [part 151 of 205, 5MB] [1 of 1]
5242880 of 5242880 100% in 0s 37.73 MB/s done
upload: '1GB' -> 's3://test-bucket/1GB' [part 152 of 205, 5MB] [1 of 1]
65536 of 5242880 1% in 0s 3.17 MB/s^CERROR:
Upload of '1GB' part 152 failed. Use
/usr/bin/s3cmd abortmp s3://test-bucket/1GB 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah
to abort the upload, or
/usr/bin/s3cmd --upload-id 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah put ...
to continue the upload.
See ya!
s3cmd shows the following error:
$ s3cmd ls s3://test-bucket --debug
...
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>test-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated></ListBucketResult>',
...
Invoked as: /usr/bin/s3cmd ls s3://test-bucket --debug
Problem: IndexError: list index out of range
...
Could not delete bucket
# radosgw-admin bucket rm --purge-objects --bypass-gc --bucket=test-bucket
The same message is repeated over and over.
2019-12-06 22:48:47.446570 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.448804 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.451057 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.453403 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.456087 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.458373 7f0457018e00 1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
#5 Updated by J. Eric Ivancich over 4 years ago
Thank you, Beom-Seok Park!
I've created a new tracker associated with the bug fix.
#6 Updated by J. Eric Ivancich over 4 years ago
- Status changed from Pending Backport to Fix Under Review
This should not be backported until https://tracker.ceph.com/issues/43180 is resolved, and they should then be backported as one unit.
#7 Updated by Nathan Cutler over 4 years ago
- Status changed from Fix Under Review to Pending Backport
#8 Updated by Nathan Cutler over 4 years ago
- Precedes Bug #43180: rgw: ordered listing of bucket with many incomplete multipart uploads fails added
#9 Updated by Nathan Cutler over 4 years ago
- Status changed from Pending Backport to Resolved
- Backport deleted (
nautilus,mimic)
Backport will be handled via #43180 (follow-on fix for this one)