Project

General

Profile

Actions
Copy link

Bug #45955

closed

[rgw] listing bucket via s3 hangs on "ordered bucket listing requires read #1"

Added by Mariusz Gronczewski almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
J. Eric Ivancich
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
octopus, nautilus
Regression:
Yes
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v15.2.3
ceph-qa-suite:
Pull request ID:
34037
Crash signature (v1):
Crash signature (v2):

Description

I dont know whether it is related to #42358 but Ive seen something similar but on bucket that just have small files (< 1MB) and only around 21k objects

doing s3cmd ls s3://fcp/ generates

2020-06-09T12:49:16.511+0200 7f015b7a9700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.543+0200 7f015a7a7700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.579+0200 7f015afa8700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.611+0200 7f015b7a9700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.643+0200 7f015a7a7700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.675+0200 7f015afa8700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.707+0200 7f015b7a9700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.739+0200 7f015a7a7700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.771+0200 7f015afa8700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.803+0200 7f015b7a9700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1
2020-06-09T12:49:16.835+0200 7f015a7a7700  1 RGWRados::Bucket::List::list_objects_ordered INFO ordered bucket listing requires read #1

in logs (for a loong time,never completes)

it gets weirder tho, doing request to subdirectory does just fine


$ s3cmd ls s3://fcp/94/

                       DIR   s3://fcp/94/02/
                       DIR   s3://fcp/94/03/
                       DIR   s3://fcp/94/07/
                       DIR   s3://fcp/94/08/
              ...   
$ s3cmd ls s3://fcp/94/02/239402/
2020-06-09 09:34     31350   s3://fcp/94/02/239402/6480.0.jpg
2020-06-09 09:34     87325   s3://fcp/94/02/239402/6480.1.jpg

we've had pretty much same file layout on the old (v12.x) pre-production cluster

now doing same via radosgw-admin is nice and snappy:

# time radosgw-admin bi list --bucket=fcp  |grep idx |wc -l
21090

real    0m0.779s
user    0m0.305s
sys    0m0.023s

Files

Download all files
fcp-bi-list.00.gz (685 KB) fcp-bi-list.00.gz Mariusz Gronczewski, 06/11/2020 11:01 PM
fcp-bi-list.01.gz (685 KB) fcp-bi-list.01.gz Mariusz Gronczewski, 06/11/2020 11:01 PM

Related issues 2 (0 open — 2 closed)

Copied to rgw - Backport #46340: octopus: [rgw] listing bucket via s3 hangs on "ordered bucket listing requires read #1"ResolvedJ. Eric IvancichActions
Copied to rgw - Backport #46326: nautilus: [rgw] listing bucket via s3 hangs on "ordered bucket listing requires read #1"ResolvedJ. Eric IvancichActions
Actions
Copy link

Also available in: Atom PDF