Project

General

Profile

Actions
Copy link

Bug #36106

open

rgw_file: list directory can only get 1000 files on NFS-Ganesha-RGW when mounting a bucket

Added by min-sheng Lin over 5 years ago. Updated about 5 years ago.

Status:
Triaged
Priority:
Normal
Assignee:
Matt Benjamin
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

If using nfs-ganesha-rgw to export bucket as nfs, only first 1000 files be listed

[vagrant@admin ~]$ ls -l /mnt/test/ | grep test | wc -l
1000

Using s3cmd can correctly list all files

[vagrant@admin ~]$ s3cmd ls s3://test | wc -l
1500

In rgw log, marker(offset) is changed from test548 to 0 when execute cls_bucket_list_ordered function

2018-09-21 06:16:43.350 7fc1582e0700 15 READDIR offset: (nil) next marker: test548 is_truncated: 1
2018-09-21 06:16:43.351 7fc1582e0700 15 readdir final link count=1002
2018-09-21 06:16:43.374 7fc1266e6700 15 rgw_readdir2 offset=test548
2018-09-21 06:16:43.374 7fc1266e6700 1 ====== process_request starting new request req=0x7fc1266e1cd0 ======
2018-09-21 06:16:43.374 7fc1266e6700 20 HTTP_HOST=
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000005:: :list_bucket:initializing for trans_id = tx000000000000000000000-005ba48ccb-1b1e-default
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000016:: :list_bucket:authorizing
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000018:: :list_bucket:reading op permissions
2018-09-21 06:16:43.374 7fc1266e6700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/&quot;&gt;&lt;Owner&gt;&lt;ID&gt;admin&lt;/ID&gt;&lt;DisplayName&gt;admin&lt;/DisplayName&gt;&lt;/Owner&gt;&lt;AccessControlList&gt;&lt;Grant&gt;&lt;Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance&quot; xsi:type="CanonicalUser"><ID>admin</ID><DisplayName>admin</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000054:: :list_bucket:init op
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000055:: :list_bucket:verifying op mask
2018-09-21 06:16:43.374 7fc1266e6700 20 required_mask= 1 user.op_mask=7
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000056:: :list_bucket:verifying op permissions
2018-09-21 06:16:43.374 7fc1266e6700 20 -- Getting permissions begin with perm_mask=49
2018-09-21 06:16:43.374 7fc1266e6700 5 Searching permissions for identity=RGWDummyIdentityApplier(auth_id=admin, perm_mask=15, is_admin=0) mask=49
2018-09-21 06:16:43.374 7fc1266e6700 5 Searching permissions for uid=admin
2018-09-21 06:16:43.374 7fc1266e6700 5 Found permission: 15
2018-09-21 06:16:43.374 7fc1266e6700 5 Searching permissions for group=1 mask=49
2018-09-21 06:16:43.374 7fc1266e6700 5 Permissions for group not found
2018-09-21 06:16:43.374 7fc1266e6700 5 Searching permissions for group=2 mask=49
2018-09-21 06:16:43.374 7fc1266e6700 5 Permissions for group not found
2018-09-21 06:16:43.374 7fc1266e6700 5 -- Getting permissions done for identity=RGWDummyIdentityApplier(auth_id=admin, perm_mask=15, is_admin=0), owner=admin, perm=1
2018-09-21 06:16:43.374 7fc1266e6700 10 identity=RGWDummyIdentityApplier(auth_id=admin, perm_mask=15, is_admin=0) requested perm (type)=1, policy perm=1, user_perm_mask=1, acl perm=1
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000067:: :list_bucket:verifying op params
2018-09-21 06:16:43.374 7fc1266e6700 2 req 0:0.000068:: :list_bucket:executing
2018-09-21 06:16:43.374 7fc1266e6700 10 cls_bucket_list_ordered test[81509a51-c0fd-4af9-90f9-c09d6139738e.4138.1] start 0[] num_entries 1001

Actions
Copy link

Also available in: Atom PDF