Project

General

Profile

Bug #23099

REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument

Added by Robin Johnson about 6 years ago. Updated over 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
Target version:
-
% Done:

0%

Source:
Community (dev)
Tags:
rest, rgw
Backport:
mimic luminous
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Trying to use the /admin/metadata/ endpoints to review bucket & bucket.instance metadata for another bug, and found that the paging arguments do not work at all.
@failed to parse cursor: 13:000fc580:::.bucket.meta.jenkins-jxjjrckl6yd3bwxzosmy7-31:default.61647851.14921:head@ turns up in the RGW debug logs.

The error ends up being near identical for both bucket & bucket.instance, just the cursor value itself differs.

More detail below.

DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=5&format=json
DEBUG:rgwadmin.rgw:Access Key: CENSORED
DEBUG:rgwadmin.rgw:Verify: True  CA Bundle: None
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=5&format=json HTTP/1.1" 200 435
{'keys': ['jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439', 'jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378', 'jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512', 'jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203', 'jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007'], 'count': 5, 'marker': '13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head', 'truncated': 'true'}
0 jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439
1 jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378
2 jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512
3 jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203
4 jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007
DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=5&marker=13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head&format=json
DEBUG:rgwadmin.rgw:Access Key: CENSORED
DEBUG:rgwadmin.rgw:Verify: True  CA Bundle: None
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=5&marker=13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head&format=json HTTP/1.1" 400 128
ERROR:rgwadmin.rgw:{'Code': 'InvalidArgument', 'HostId': '1ee4fe7f-default-default', 'RequestId': 'tx0000000000000000df8d7-005a8fab0d-1ee4fe7f-default'}
bucket.instance 13:000005cf:::.bucket.meta.jenkins-o8baqgsl51kgugzka63va-5%3adefault.64449186.47234:head 
DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=10&format=json
DEBUG:rgwadmin.rgw:Access Key: CENSORED
DEBUG:rgwadmin.rgw:Verify: True  CA Bundle: None
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=10&format=json HTTP/1.1" 200 730
{'count': 10, 'keys': ['jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439', 'jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378', 'jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512', 'jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203', 'jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007', 'jenkins-o8baqgsl51kgugzka63va-5:default.64449186.47234', 'jenkins-0gpmeclhtllcyqbj8feqe-111:default.42572384.9453', 'jenkins-5irdeeiq719ut08stwhhp-104:default.42569158.243549', 'jenkins-6onbmsumnuza80o66br65-86:default.48899572.13142', 'jenkins-ael7q3zlukahy8q1vg5tu-153:default.49582276.43648'], 'marker': '13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head', 'truncated': 'true'}
0 jenkins-iqh9ki7zool7mgtzqjkh5-172:default.80666971.66439
1 jenkins-b4cp5uh3c67t6n1bumx91-112:default.72866483.24378
2 jenkins-q5xaj0cq5vhz868h2etm9-80:default.108161568.1512
3 jenkins-gm60yx3hwvfe8qszkwoyn-86:default.42569158.164203
4 jenkins-7h8fboa742tv7orxy0x2i-171:default.142849256.256007
5 jenkins-o8baqgsl51kgugzka63va-5:default.64449186.47234
6 jenkins-0gpmeclhtllcyqbj8feqe-111:default.42572384.9453
7 jenkins-5irdeeiq719ut08stwhhp-104:default.42569158.243549
8 jenkins-6onbmsumnuza80o66br65-86:default.48899572.13142
9 jenkins-ael7q3zlukahy8q1vg5tu-153:default.49582276.43648
DEBUG:rgwadmin.rgw:URL: https://objects-us-west-1.dream.io/admin/metadata/bucket.instance?max-entries=10&format=json&marker=13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head
DEBUG:rgwadmin.rgw:Access Key: CENSORED
DEBUG:rgwadmin.rgw:Verify: True  CA Bundle: None
INFO:urllib3.connectionpool:Starting new HTTPS connection (1): objects-us-west-1.dream.io
DEBUG:urllib3.connectionpool:Setting read timeout to None
DEBUG:urllib3.connectionpool:"GET /admin/metadata/bucket.instance?max-entries=10&format=json&marker=13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head HTTP/1.1" 400 128
ERROR:rgwadmin.rgw:{'HostId': '1ed5a276-default-default', 'Code': 'InvalidArgument', 'RequestId': 'tx000000000000000110529-005a8fab4d-1ed5a276-default'}
bucket.instance 13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78%3adefault.51695731.283576:head 

RGW debug logging:

2018-02-23 05:55:09.928328 7fcdf6766700  2 req 927072:0.003412::GET /admin/metadata/bucket.instance:list_metadata:executing
2018-02-23 05:55:09.928392 7fcdf6766700 10 failed to parse cursor: 13:000006fd:::.bucket.meta.jenkins-32qi0omvl8vj3teoqnkxu-78:default.51695731.283576:head
2018-02-23 05:55:09.928399 7fcdf6766700 10 failed to list objects pool_iterate_begin() returned r=-22
2018-02-23 05:55:09.928403 7fcdf6766700  5 ERROR: can't get key: (22) Invalid argument
2018-02-23 05:55:09.928409 7fcdf6766700  2 req 927072:0.003493::GET /admin/metadata/bucket.instance:list_metadata:completing
2018-02-23 05:55:09.928523 7fcdf6766700  2 req 927072:0.003606::GET /admin/metadata/bucket.instance:list_metadata:op status=0
2018-02-23 05:55:09.928528 7fcdf6766700  2 req 927072:0.003612::GET /admin/metadata/bucket.instance:list_metadata:http status=400


Related issues

Related to rgw - Bug #23193: multsite use of metadata list pagination New
Copied to rgw - Backport #24813: mimic: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument Resolved
Copied to rgw - Backport #24814: luminous: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument Resolved

History

#1 Updated by Casey Bodley about 6 years ago

  • Related to Bug #23193: multsite use of metadata list pagination added

#2 Updated by Matt Benjamin about 6 years ago

  • Assignee set to Matt Benjamin

#3 Updated by Matt Benjamin about 6 years ago

  • Status changed from New to In Progress

#4 Updated by Matt Benjamin almost 6 years ago

Hi Robin, think I've reproduced this, in the "bucket" case. Will follow up.

Matt

#5 Updated by Yehuda Sadeh almost 6 years ago

Matt, any progress on this?

#7 Updated by Matt Benjamin almost 6 years ago

  • Status changed from In Progress to Fix Under Review

#8 Updated by Matt Benjamin almost 6 years ago

  • Backport set to mimic

#9 Updated by Abhishek Lekshmanan over 5 years ago

  • Status changed from Fix Under Review to 17

#10 Updated by Yuri Weinstein over 5 years ago

Matt Benjamin wrote:

https://github.com/ceph/ceph/pull/22721

merged

#11 Updated by Casey Bodley over 5 years ago

  • Status changed from 17 to Pending Backport
  • Backport changed from mimic to mimic luminous

I believe this is needed on Luminous also - correct me if I'm wrong.

#12 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #24813: mimic: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument added

#13 Updated by Nathan Cutler over 5 years ago

  • Copied to Backport #24814: luminous: REST admin metadata API paging failure bucket & bucket.instance: InvalidArgument added

#14 Updated by Nathan Cutler over 5 years ago

  • Status changed from Pending Backport to Resolved

Also available in: Atom PDF