Bug #23099
REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument
% Done:
0%
Source:
Community (dev)
Tags:
rest, rgw
Backport:
mimic luminous
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
Trying to use the /admin/metadata/ endpoints to review bucket & bucket.instance metadata for another bug, and found that the paging arguments do not work at all.@failed to parse cursor: 13:000fc580:::.bucket.meta.jenkins-jxjjrckl6yd3bwxzosmy7-31:default.61647851.14921:head
@ turns up in the RGW debug logs.
The error ends up being near identical for both bucket & bucket.instance, just the cursor value itself differs.
More detail below.
DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=5&format=json DEBUG:rgwadmin.rgw:Access Key: CENSORED DEBUG:rgwadmin.rgw:Verify: True CA Bundle: None INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=5&format=json HTTP/1.1" 200 435 {'keys': ['jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439', 'jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378', 'jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512', 'jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203', 'jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007'], 'count': 5, 'marker': '13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head', 'truncated': 'true'} 0 jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439 1 jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378 2 jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512 3 jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203 4 jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007 DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=5&marker=13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head&format=json DEBUG:rgwadmin.rgw:Access Key: CENSORED DEBUG:rgwadmin.rgw:Verify: True CA Bundle: None INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=5&marker=13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head&format=json HTTP/1.1" 400 128 ERROR:rgwadmin.rgw:{'Code': 'InvalidArgument', 'HostId': '1ee4fe7f-default-default', 'RequestId': 'tx0000000000000000df8d7-005a8fab0d-1ee4fe7f-default'} bucket.instance 13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=10&format=json DEBUG:rgwadmin.rgw:Access Key: CENSORED DEBUG:rgwadmin.rgw:Verify: True CA Bundle: None INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=10&format=json HTTP/1.1" 200 730 {'count': 10, 'keys': ['jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439', 'jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378', 'jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512', 'jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203', 'jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007', 'jenkins-o8baqgsl51kgugzka63va-5:default.64449186.47234', 'jenkins-0gpmeclhtllcyqbj8feqe-111:default.42572384.9453', 'jenkins-5irdeeiq719ut08stwhhp-104:default.42569158.243549', 'jenkins-6onbmsumnuza80o66br65-86:default.48899572.13142', 'jenkins-ael7q3zlukahy8q1vg5tu-153:default.49582276.43648'], 'marker': '13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head', 'truncated': 'true'} 0 jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439 1 jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378 2 jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512 3 jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203 4 jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007 5 jenkins-o8baqgsl51kgugzka63va-5:default.64449186.47234 6 jenkins-0gpmeclhtllcyqbj8feqe-111:default.42572384.9453 7 jenkins-5irdeeiq719ut08stwhhp-104:default.42569158.243549 8 jenkins-6onbmsumnuza80o66br65-86:default.48899572.13142 9 jenkins-ael7q3zlukahy8q1vg5tu-153:default.49582276.43648 DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=10&format=json&marker=13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head DEBUG:rgwadmin.rgw:Access Key: CENSORED DEBUG:rgwadmin.rgw:Verify: True CA Bundle: None INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io DEBUG:urllib3.connectionpool:Setting read timeout to None DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=10&format=json&marker=13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head HTTP/1.1" 400 128 ERROR:rgwadmin.rgw:{'HostId': '1ed5a276-default-default', 'Code': 'InvalidArgument', 'RequestId': 'tx000000000000000110529-005a8fab4d-1ed5a276-default'} bucket.instance 13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head
RGW debug logging:
2018-02-23 05:55:09.928328 7fcdf6766700 2 req 927072:0.003412::GET /admin/metadata/bucket.instance:list_metadata:executing 2018-02-23 05:55:09.928392 7fcdf6766700 10 failed to parse cursor: 13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78:default.51695731.283576:head 2018-02-23 05:55:09.928399 7fcdf6766700 10 failed to list objects pool_iterate_begin() returned r=-22 2018-02-23 05:55:09.928403 7fcdf6766700 5 ERROR: can't get key: (22) Invalid argument 2018-02-23 05:55:09.928409 7fcdf6766700 2 req 927072:0.003493::GET /admin/metadata/bucket.instance:list_metadata:completing 2018-02-23 05:55:09.928523 7fcdf6766700 2 req 927072:0.003606::GET /admin/metadata/bucket.instance:list_metadata:op status=0 2018-02-23 05:55:09.928528 7fcdf6766700 2 req 927072:0.003612::GET /admin/metadata/bucket.instance:list_metadata:http status=400
Related issues
History
#1 Updated by Casey Bodley about 6 years ago
- Related to Bug #23193: multsite use of metadata list pagination added
#2 Updated by Matt Benjamin about 6 years ago
- Assignee set to Matt Benjamin
#3 Updated by Matt Benjamin about 6 years ago
- Status changed from New to In Progress
#4 Updated by Matt Benjamin almost 6 years ago
Hi Robin, think I've reproduced this, in the "bucket" case. Will follow up.
Matt
#5 Updated by Yehuda Sadeh almost 6 years ago
Matt, any progress on this?
#6 Updated by Matt Benjamin almost 6 years ago
#7 Updated by Matt Benjamin almost 6 years ago
- Status changed from In Progress to Fix Under Review
#8 Updated by Matt Benjamin almost 6 years ago
- Backport set to mimic
#9 Updated by Abhishek Lekshmanan over 5 years ago
- Status changed from Fix Under Review to 17
#10 Updated by Yuri Weinstein over 5 years ago
#11 Updated by Casey Bodley over 5 years ago
- Status changed from 17 to Pending Backport
- Backport changed from mimic to mimic luminous
I believe this is needed on Luminous also - correct me if I'm wrong.
#12 Updated by Nathan Cutler over 5 years ago
- Copied to Backport #24813: mimic: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument added
#13 Updated by Nathan Cutler over 5 years ago
- Copied to Backport #24814: luminous: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument added
#14 Updated by Nathan Cutler over 5 years ago
- Status changed from Pending Backport to Resolved