Project

General

Profile

Bug #42358

rgw: reduce per-shard entry count during ordered bucket listing

Added by Eric Ivancich 4 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

Currently, if someone requests the 1000 next entries from a bucket, each bucket index shard will receive a request for the 1000 next entries. When there are hundreds, thousands, or tens of thousands of bucket index shards, this results in a huge amplification of the request, even though only 1000 entries will be returned.

These changes reduce the per-bucket index shard requests. These also allow re-requests in edge cases where all of one shard's returned entries are consumed. Finally these changes improve the determination of whether the resulting list is truncated.


Related issues

Precedes rgw - Bug #43180: rgw: ordered listing of bucket with many incomplete multipart uploads fails Pending Backport

History

#1 Updated by Eric Ivancich 3 months ago

  • Status changed from In Progress to Pending Backport

#4 Updated by Beom-Seok Park 3 months ago

This PR breaks the s3 client and radosgw-admin if there are a lot of uncomplete multiparts in the bucket.

Create a bucket.

$ s3cmd mb s3://test-bucket

Create over 150 uncomplete multiparts.

$ s3cmd -c testing.s3cfg put 1GB s3://test-bucket --multipart-chunk-size-mb=5
upload: '1GB' -> 's3://test-bucket/1GB'  [part 1 of 205, 5MB] [1 of 1]
 5242880 of 5242880   100% in    0s    27.60 MB/s  done
...
upload: '1GB' -> 's3://test-bucket/1GB'  [part 151 of 205, 5MB] [1 of 1]
 5242880 of 5242880   100% in    0s    37.73 MB/s  done
upload: '1GB' -> 's3://test-bucket/1GB'  [part 152 of 205, 5MB] [1 of 1]
   65536 of 5242880     1% in    0s     3.17 MB/s^CERROR: 
Upload of '1GB' part 152 failed. Use
  /usr/bin/s3cmd abortmp s3://test-bucket/1GB 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah
to abort the upload, or
  /usr/bin/s3cmd --upload-id 2~NzYtsNgOq8hWplcDqnaZDHoPlP4Vkah put ...
to continue the upload.
See ya!

s3cmd shows the following error:

$ s3cmd ls s3://test-bucket --debug
...
DEBUG: Response:
{'data': '<?xml version="1.0" encoding="UTF-8"?><ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Name>test-bucket</Name><Prefix></Prefix><Marker></Marker><MaxKeys>1000</MaxKeys><IsTruncated>true</IsTruncated></ListBucketResult>',
...
Invoked as: /usr/bin/s3cmd ls s3://test-bucket --debug
Problem: IndexError: list index out of range
...

Could not delete bucket

# radosgw-admin bucket rm --purge-objects --bypass-gc --bucket=test-bucket

The same message is repeated over and over.

2019-12-06 22:48:47.446570 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.448804 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.451057 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.453403 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3
2019-12-06 22:48:47.456087 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #2
2019-12-06 22:48:47.458373 7f0457018e00  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #3

#5 Updated by Eric Ivancich 3 months ago

Thank you, Beom-Seok Park!

I've created a new tracker associated with the bug fix.

https://tracker.ceph.com/issues/43180

#6 Updated by Eric Ivancich 3 months ago

  • Status changed from Pending Backport to Fix Under Review

This should not be backported until https://tracker.ceph.com/issues/43180 is resolved, and they should then be backported as one unit.

#7 Updated by Nathan Cutler 2 months ago

  • Status changed from Fix Under Review to Pending Backport

#8 Updated by Nathan Cutler 2 months ago

  • Precedes Bug #43180: rgw: ordered listing of bucket with many incomplete multipart uploads fails added

#9 Updated by Nathan Cutler 2 months ago

  • Status changed from Pending Backport to Resolved
  • Backport deleted (nautilus,mimic)

Backport will be handled via #43180 (follow-on fix for this one)

Also available in: Atom PDF