Project

General

Profile

Actions

Bug #59400

open

rgw: When there are a large number of multiparts, the unorder list result may miss objects

Added by chaochao li about 1 year ago. Updated 8 months ago.

Status:
Pending Backport
Priority:
Normal
Target version:
-
% Done:

0%

Source:
Tags:
backport_processed
Backport:
pacific,quincy,reef
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rgw
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Steps to Reproduce:
1. Create a bucket and fill it with abort multipart whose quantity is greater than 1100.
2. uploaded less than 1000 objects (I uploaded 802 objects at the time).
3. Issue an unordered listing of the bucket by boto3 API with allow-unordered=true.

The partial logs are as follows.

2023-04-11T04:54:21.404+0200 7f4516a63700 10 req 14930413712671967438 0.082999997s s3:list_bucket cls_bucket_list_unordered: got 570
2023-04-11T04:54:21.404+0200 7f4516a63700 10 req 14930413712671967438 0.082999997s s3:list_bucket INFO: cls_bucket_list_unordered: reached max entries (1100) to return at "570" 

Upon checking the code, it was found that the 1101st object was set to last_added_entry and not placed in ent_list.
When there are more than 100 multiparts in a list, there may be cases of missing list results.
https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_rados.cc#L9122

      last_added_entry = dirent.key;
      *is_truncated = true;
      ldpp_dout(dpp, 10) << "INFO: " << __func__ <<
        ": reached max entries (" << num_entries << ") to return at \"" <<
        dirent.key << "\"" << dendl;
      goto check_updates;
    }

Please refer to the attachment for detailed logs


Files

radosgw.8000.log (316 KB) radosgw.8000.log unorder-list log chaochao li, 04/11/2023 04:34 AM

Related issues 3 (2 open1 closed)

Copied to rgw - Backport #62757: reef: rgw: When there are a large number of multiparts, the unorder list result may miss objectsNewJ. Eric IvancichActions
Copied to rgw - Backport #62758: pacific: rgw: When there are a large number of multiparts, the unorder list result may miss objectsRejectedJ. Eric IvancichActions
Copied to rgw - Backport #62759: quincy: rgw: When there are a large number of multiparts, the unorder list result may miss objectsNewJ. Eric IvancichActions
Actions #1

Updated by chaochao li about 1 year ago

https://github.com/ceph/ceph/pull/42151
This PR will cause the current problem.

Actions #2

Updated by Casey Bodley about 1 year ago

  • Assignee set to J. Eric Ivancich
Actions #3

Updated by J. Eric Ivancich 9 months ago

I was able to reproduce the situation and verify that the named PR was the cause. And when I looked at it more closely it was clearly problematic. So I will revert that commit. I will also try to reproduce the issue that PR was trying to address and fix that in a more correct manner as well.

Thank you for the report!

Actions #4

Updated by J. Eric Ivancich 9 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 52866
Actions #5

Updated by J. Eric Ivancich 8 months ago

  • Status changed from Fix Under Review to Pending Backport
  • Backport set to pacific,quincy,reef
Actions #6

Updated by Backport Bot 8 months ago

  • Copied to Backport #62757: reef: rgw: When there are a large number of multiparts, the unorder list result may miss objects added
Actions #7

Updated by Backport Bot 8 months ago

  • Copied to Backport #62758: pacific: rgw: When there are a large number of multiparts, the unorder list result may miss objects added
Actions #8

Updated by Backport Bot 8 months ago

  • Copied to Backport #62759: quincy: rgw: When there are a large number of multiparts, the unorder list result may miss objects added
Actions #9

Updated by Backport Bot 8 months ago

  • Tags set to backport_processed
Actions

Also available in: Atom PDF