Bug #59400
openrgw: When there are a large number of multiparts, the unorder list result may miss objects
0%
Description
Steps to Reproduce:
1. Create a bucket and fill it with abort multipart whose quantity is greater than 1100.
2. uploaded less than 1000 objects (I uploaded 802 objects at the time).
3. Issue an unordered listing of the bucket by boto3 API with allow-unordered=true.
The partial logs are as follows.
2023-04-11T04:54:21.404+0200 7f4516a63700 10 req 14930413712671967438 0.082999997s s3:list_bucket cls_bucket_list_unordered: got 570 2023-04-11T04:54:21.404+0200 7f4516a63700 10 req 14930413712671967438 0.082999997s s3:list_bucket INFO: cls_bucket_list_unordered: reached max entries (1100) to return at "570"
Upon checking the code, it was found that the 1101st object was set to last_added_entry and not placed in ent_list.
When there are more than 100 multiparts in a list, there may be cases of missing list results.
https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_rados.cc#L9122
last_added_entry = dirent.key;
*is_truncated = true;
ldpp_dout(dpp, 10) << "INFO: " << __func__ <<
": reached max entries (" << num_entries << ") to return at \"" <<
dirent.key << "\"" << dendl;
goto check_updates;
}
Please refer to the attachment for detailed logs
Files
Updated by chaochao li about 1 year ago
https://github.com/ceph/ceph/pull/42151
This PR will cause the current problem.
Updated by J. Eric Ivancich 10 months ago
I was able to reproduce the situation and verify that the named PR was the cause. And when I looked at it more closely it was clearly problematic. So I will revert that commit. I will also try to reproduce the issue that PR was trying to address and fix that in a more correct manner as well.
Thank you for the report!
Updated by J. Eric Ivancich 10 months ago
- Status changed from New to Fix Under Review
- Pull request ID set to 52866
Updated by J. Eric Ivancich 9 months ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to pacific,quincy,reef
Updated by Backport Bot 9 months ago
- Copied to Backport #62757: reef: rgw: When there are a large number of multiparts, the unorder list result may miss objects added
Updated by Backport Bot 9 months ago
- Copied to Backport #62758: pacific: rgw: When there are a large number of multiparts, the unorder list result may miss objects added
Updated by Backport Bot 9 months ago
- Copied to Backport #62759: quincy: rgw: When there are a large number of multiparts, the unorder list result may miss objects added