Actions
Bug #17372
closedRGW will not list Argonaut-era bucket via HTTP (but radosgw-admin works)
% Done:
0%
Source:
Community (dev)
Tags:
Backport:
jewel
Regression:
Yes
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
DreamObjects has some very old Ceph production buckets: Argonaut or older.
They do list if you use radosgw-admin bucket list, but NOT via HTTP.
This is definitely a regression since Hammer, possibly since sometime in Jewel (not easy to test each release in my environment).
# radosgw-admin bucket stats --bucket=bucket
{
"bucket": "bucket",
"pool": ".rgw.data.1",
"index_pool": ".rgw.data.1",
"id": "88283.1",
"marker": "88283.1",
"owner": "CENSORED",
"ver": "0#0",
"master_ver": "0#0",
"mtime": "0.000000",
"max_marker": "0#",
"usage": {
"rgw.main": {
"size_kb": 18,
"size_kb_actual": 20,
"num_objects": 1
}
},
"bucket_quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
}
}
# radosgw-admin metadata get bucket:bucket
{
"key": "bucket:bucket",
"ver": {
"tag": "",
"ver": 0
},
"mtime": "2012-09-12 23:43:46.861813Z",
"data": {
"bucket": {
"name": "",
"pool": "",
"data_extra_pool": "",
"index_pool": "",
"marker": "",
"bucket_id": "",
"tenant": ""
},
"owner": "",
"creation_time": "0.000000",
"linked": "false",
"has_bucket_info": "true",
"old_bucket_info": {
"bucket": {
"name": "bucket",
"pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1",
"marker": "88283.1",
"bucket_id": "88283.1",
"tenant": ""
},
"creation_time": "0.000000",
"owner": "CENSORED",
"flags": 0,
"zonegroup": "",
"placement_rule": "",
"has_instance_obj": "false",
"quota": {
"enabled": false,
"max_size_kb": -1,
"max_objects": -1
},
"num_shards": 0,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": ""
}
}
}
# radosgw-admin metadata get bucket.instance:bucket:88283.1
{
"key": "bucket.instance:bucket:88283.1",
"ver": {
"tag": "JqXTogWqCuSUkGsUgj+VMl1w",
"ver": 7
},
"mtime": "2015-07-01 20:12:08.000000Z",
"data": {
"bucket_info": {
"bucket": {
"name": "bucket",
"pool": ".rgw.data.1",
"data_extra_pool": "",
"index_pool": ".rgw.data.1",
"marker": "88283.1",
"bucket_id": "88283.1",
"tenant": ""
},
"creation_time": "0.000000",
"owner": "CENSORED",
"flags": 0,
"zonegroup": "",
"placement_rule": "",
"has_instance_obj": "true",
"quota": {
"enabled": false,
"max_size_kb": 10240000,
"max_objects": -1
},
"num_shards": 0,
"bi_shard_hash_type": 0,
"requester_pays": "false",
"has_website": "false",
"swift_versioning": "false",
"swift_ver_location": ""
},
"attrs": [
{
"key": "user.rgw.acl",
"val": "AgK5AAAAAgIUAAAABgAAAG1hdHdvZAYAAABtYXR3b2QDA5kAAAABAQAAAAYAAABtYXR3b2QPAAAAAgAAAAAAAAADAygAAAACAgQAAAACAAAAAAAAAAAAAAAAAAAAAgIEAAAAAQAAAAAAAAABAAAABgAAAG1hdHdvZAMDNAAAAAICBAAAAAAAAAAGAAAAbWF0d29kAAAAAAAAAAACAgQAAAAPAAAABgAAAG1hdHdvZAAAAAABAAAAAQAAAAEAAAA="
},
{
"key": "user.rgw.idtag",
"val": ""
},
{
"key": "user.rgw.manifest",
"val": ""
}
]
}
}
# radosgw-admin bucket list --bucket=bucket
[
{
"name": "object",
"instance": "",
"namespace": "",
"owner": "CENSORED",
"owner_display_name": "CENSORED",
"size": 18236,
"mtime": "2012-05-10 17:49:34.000000Z",
"etag": "971a661e70295d76f7801516a6767e59",
"content_type": "image\/png",
"tag": "",
"flags": 0
}
]
And now the RGW debug log:
# GET http://bucket.objects-us-west-1.dream.io/ 016-09-22 11:20:50.254766 7fd4362e8700 20 RGWEnv::set(): HTTP_USER_AGENT: curl/7.35.0 2016-09-22 11:20:50.254776 7fd4362e8700 20 RGWEnv::set(): HTTP_ACCEPT: */* 2016-09-22 11:20:50.254777 7fd4362e8700 20 RGWEnv::set(): HTTP_HOST: bucket.objects-us-west-1.dream.io 2016-09-22 11:20:50.254778 7fd4362e8700 20 RGWEnv::set(): REQUEST_METHOD: GET 2016-09-22 11:20:50.254779 7fd4362e8700 20 RGWEnv::set(): REQUEST_URI: / 2016-09-22 11:20:50.254781 7fd4362e8700 20 RGWEnv::set(): QUERY_STRING: 2016-09-22 11:20:50.254782 7fd4362e8700 20 RGWEnv::set(): REMOTE_USER: 2016-09-22 11:20:50.254783 7fd4362e8700 20 RGWEnv::set(): SCRIPT_URI: / 2016-09-22 11:20:50.254787 7fd4362e8700 20 RGWEnv::set(): SERVER_PORT: 0 2016-09-22 11:20:50.254788 7fd4362e8700 20 HTTP_ACCEPT=*/* 2016-09-22 11:20:50.254789 7fd4362e8700 20 HTTP_HOST=bucket.objects-us-west-1.dream.io 2016-09-22 11:20:50.254789 7fd4362e8700 20 HTTP_USER_AGENT=curl/7.35.0 2016-09-22 11:20:50.254790 7fd4362e8700 20 QUERY_STRING= 2016-09-22 11:20:50.254790 7fd4362e8700 20 REMOTE_USER= 2016-09-22 11:20:50.254791 7fd4362e8700 20 REQUEST_METHOD=GET 2016-09-22 11:20:50.254791 7fd4362e8700 20 REQUEST_URI=/ 2016-09-22 11:20:50.254792 7fd4362e8700 20 SCRIPT_URI=/ 2016-09-22 11:20:50.254792 7fd4362e8700 20 SERVER_PORT=0 2016-09-22 11:20:50.254793 7fd4362e8700 1 ====== starting new request req=0x7fd4362e27d0 ===== 2016-09-22 11:20:50.254810 7fd4362e8700 2 req 2:0.000017::GET /::initializing for trans_id = tx000000000000000000002-0057e3be92-1084d9eb-default 2016-09-22 11:20:50.254820 7fd4362e8700 10 rgw api priority: s3=4 s3website=-1 2016-09-22 11:20:50.254822 7fd4362e8700 10 host=bucket.objects-us-west-1.dream.io 2016-09-22 11:20:50.254824 7fd4362e8700 20 subdomain=bucket domain=objects-us-west-1.dream.io in_hosted_domain=1 in_hosted_domain_s3website=0 2016-09-22 11:20:50.254826 7fd4362e8700 20 final domain/bucket subdomain=bucket domain=objects-us-west-1.dream.io in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=objects-us-west-1.dream.io s->info.request_uri=/bucket/ 2016-09-22 11:20:50.254851 7fd4362e8700 20 get_handler handler=25RGWHandler_REST_Bucket_S3 2016-09-22 11:20:50.254854 7fd4362e8700 10 handler=25RGWHandler_REST_Bucket_S3 2016-09-22 11:20:50.254855 7fd4362e8700 2 req 2:0.000062:s3:GET /::getting op 0 2016-09-22 11:20:50.254862 7fd4362e8700 10 op=25RGWListBucket_ObjStore_S3 2016-09-22 11:20:50.254863 7fd4362e8700 2 req 2:0.000070:s3:GET /:list_bucket:authorizing 2016-09-22 11:20:50.254866 7fd4362e8700 2 req 2:0.000073:s3:GET /:list_bucket:normalizing buckets and tenants 2016-09-22 11:20:50.254868 7fd4362e8700 10 s->object=<NULL> s->bucket=bucket 2016-09-22 11:20:50.254870 7fd4362e8700 2 req 2:0.000077:s3:GET /:list_bucket:init permissions 2016-09-22 11:20:50.254884 7fd4362e8700 20 get_system_obj_state: rctx=0x7fd4362e26e0 obj=.rgw:bucket state=0x7fd5440086d8 s->prefetch_data=0 2016-09-22 11:20:50.254891 7fd4362e8700 10 cache get: name=.rgw+bucket : type miss (requested=22, cached=19) 2016-09-22 11:20:50.254935 7fd4362e8700 1 -- [2607:f298:4:d006::5795]:0/603440092 --> [2607:f298:4:d007::8852]:6860/111972 -- osd_op(client.277141995.0:383 13.3c5bc352 bucket [call version.read,getxattrs,stat] snapc 0=[] ack+read+known_if_redirected e1005544) v7 -- ?+0 0x7fd54400aac0 con 0x7fd576de0710 2016-09-22 11:20:50.259375 7fd227032700 1 -- [2607:f298:4:d006::5795]:0/603440092 <== osd.1010 [2607:f298:4:d007::8853]:6804/481920 2 ==== osd_op_reply(382 obj_delete_at_hint.0000000059 [call] v0'0 uv1283 ondisk = 0) v7 ==== 149+0+15 (2904161573 0 2149983739) 0x7fd4884220c0 con 0x7fd4b00a1ce0 2016-09-22 11:20:50.259537 7fd4bffff700 1 -- [2607:f298:4:d006::5795]:0/603440092 --> [2607:f298:4:d007::8853]:6804/481920 -- osd_op(client.277141995.0:384 18.1d0c86ea obj_delete_at_hint.0000000059 [call lock.unlock] snapc 0=[] ondisk+write+known_if_redirected e1005544) v7 -- ?+0 0x7fd4b00a3590 con 0x7fd4b00a1ce0 2016-09-22 11:20:50.273352 7fd22dea0700 1 -- [2607:f298:4:d006::5795]:0/603440092 <== osd.994 [2607:f298:4:d007::8852]:6860/111972 3 ==== osd_op_reply(383 bucket [call,getxattrs,stat] v0'0 uv5397 ondisk = 0) v7 ==== 210+0+255 (623213301 0 1810348938) 0x7fd43c5e6e40 con 0x7fd576de0710 2016-09-22 11:20:50.273425 7fd4362e8700 10 cache put: name=.rgw+bucket info.flags=22 2016-09-22 11:20:50.273436 7fd4362e8700 10 moving .rgw+bucket to cache LRU end 2016-09-22 11:20:50.273444 7fd4362e8700 10 updating xattr: name=user.rgw.acl bl.length()=191 2016-09-22 11:20:50.273451 7fd4362e8700 20 get_system_obj_state: s->obj_tag was set empty 2016-09-22 11:20:50.273454 7fd4362e8700 20 Read xattr: user.rgw.acl 2016-09-22 11:20:50.273455 7fd4362e8700 20 Read xattr: user.rgw.idtag 2016-09-22 11:20:50.273456 7fd4362e8700 20 Read xattr: user.rgw.manifest 2016-09-22 11:20:50.273460 7fd4362e8700 10 cache get: name=.rgw+bucket : hit (requested=17, cached=23) 2016-09-22 11:20:50.273479 7fd4362e8700 20 rgw_get_bucket_info: old bucket info, bucket=bucket(@.rgw.data.1[88283.1]) owner CENSOR 2016-09-22 11:20:50.273493 7fd4362e8700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>CENSOR</ID><DisplayName>CENSOR</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>CENSOR</ID><DisplayName>CENSOR</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2016-09-22 11:20:50.273506 7fd4362e8700 10 read_permissions on bucket(@.rgw.data.1[88283.1]) ret=-2 2016-09-22 11:20:50.273509 7fd4362e8700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2 2016-09-22 11:20:50.273592 7fd4362e8700 2 req 2:0.018799:s3:GET /:list_bucket:op status=0 2016-09-22 11:20:50.273599 7fd4362e8700 2 req 2:0.018805:s3:GET /:list_bucket:http status=404 2016-09-22 11:20:50.273602 7fd4362e8700 1 ====== req done req=0x7fd4362e27d0 op status=0 http_status=404 ====== 2016-09-22 11:20:50.273612 7fd4362e8700 20 process_request() returned -2 2016-09-22 11:20:50.273633 7fd4362e8700 1 civetweb: 0x7fd544001ea0: 2607:f298:4:d006::5796 - - [22/Sep/2016:11:20:50 +0000] "GET / HTTP/1.1" 404 0 - curl/7.35.0 # GET http://bucket.objects-us-west-1.dream.io/object 2016-09-22 11:24:36.697643 7fd435ae7700 20 RGWEnv::set(): HTTP_USER_AGENT: curl/7.35.0 2016-09-22 11:24:36.697656 7fd435ae7700 20 RGWEnv::set(): HTTP_ACCEPT: */* 2016-09-22 11:24:36.697659 7fd435ae7700 20 RGWEnv::set(): HTTP_HOST: bucket.objects-us-west-1.dream.io 2016-09-22 11:24:36.697660 7fd435ae7700 20 RGWEnv::set(): REQUEST_METHOD: GET 2016-09-22 11:24:36.697661 7fd435ae7700 20 RGWEnv::set(): REQUEST_URI: /object 2016-09-22 11:24:36.697662 7fd435ae7700 20 RGWEnv::set(): QUERY_STRING: 2016-09-22 11:24:36.697663 7fd435ae7700 20 RGWEnv::set(): REMOTE_USER: 2016-09-22 11:24:36.697664 7fd435ae7700 20 RGWEnv::set(): SCRIPT_URI: /object 2016-09-22 11:24:36.697666 7fd435ae7700 20 RGWEnv::set(): SERVER_PORT: 0 2016-09-22 11:24:36.697667 7fd435ae7700 20 HTTP_ACCEPT=*/* 2016-09-22 11:24:36.697668 7fd435ae7700 20 HTTP_HOST=bucket.objects-us-west-1.dream.io 2016-09-22 11:24:36.697668 7fd435ae7700 20 HTTP_USER_AGENT=curl/7.35.0 2016-09-22 11:24:36.697669 7fd435ae7700 20 QUERY_STRING= 2016-09-22 11:24:36.697669 7fd435ae7700 20 REMOTE_USER= 2016-09-22 11:24:36.697670 7fd435ae7700 20 REQUEST_METHOD=GET 2016-09-22 11:24:36.697671 7fd435ae7700 20 REQUEST_URI=/object 2016-09-22 11:24:36.697671 7fd435ae7700 20 SCRIPT_URI=/object 2016-09-22 11:24:36.697672 7fd435ae7700 20 SERVER_PORT=0 2016-09-22 11:24:36.697673 7fd435ae7700 1 ====== starting new request req=0x7fd435ae17d0 ===== 2016-09-22 11:24:36.697691 7fd435ae7700 2 req 3:0.000018::GET /object::initializing for trans_id = tx000000000000000000003-0057e3bf74-1084d9eb-default 2016-09-22 11:24:36.697700 7fd435ae7700 10 rgw api priority: s3=4 s3website=-1 2016-09-22 11:24:36.697701 7fd435ae7700 10 host=bucket.objects-us-west-1.dream.io 2016-09-22 11:24:36.697704 7fd435ae7700 20 subdomain=bucket domain=objects-us-west-1.dream.io in_hosted_domain=1 in_hosted_domain_s3website=0 2016-09-22 11:24:36.697706 7fd435ae7700 20 final domain/bucket subdomain=bucket domain=objects-us-west-1.dream.io in_hosted_domain=1 in_hosted_domain_s3website=0 s->info.domain=objects-us-west-1.dream.io s->info.request_uri=/bucket/object 2016-09-22 11:24:36.697729 7fd435ae7700 20 get_handler handler=22RGWHandler_REST_Obj_S3 2016-09-22 11:24:36.697732 7fd435ae7700 10 handler=22RGWHandler_REST_Obj_S3 2016-09-22 11:24:36.697734 7fd435ae7700 2 req 3:0.000060:s3:GET /object::getting op 0 2016-09-22 11:24:36.697741 7fd435ae7700 10 op=21RGWGetObj_ObjStore_S3 2016-09-22 11:24:36.697742 7fd435ae7700 2 req 3:0.000069:s3:GET /object:get_obj:authorizing 2016-09-22 11:24:36.697745 7fd435ae7700 2 req 3:0.000072:s3:GET /object:get_obj:normalizing buckets and tenants 2016-09-22 11:24:36.697747 7fd435ae7700 10 s->object=object s->bucket=bucket 2016-09-22 11:24:36.697749 7fd435ae7700 2 req 3:0.000076:s3:GET /object:get_obj:init permissions 2016-09-22 11:24:36.697765 7fd435ae7700 20 get_system_obj_state: rctx=0x7fd435ae16e0 obj=.rgw:bucket state=0x7fd4e400c6e8 s->prefetch_data=0 2016-09-22 11:24:36.697775 7fd435ae7700 10 cache get: name=.rgw+bucket : hit (requested=22, cached=23) 2016-09-22 11:24:36.697784 7fd435ae7700 20 get_system_obj_state: s->obj_tag was set empty 2016-09-22 11:24:36.697786 7fd435ae7700 20 Read xattr: user.rgw.acl 2016-09-22 11:24:36.697786 7fd435ae7700 20 Read xattr: user.rgw.idtag 2016-09-22 11:24:36.697787 7fd435ae7700 20 Read xattr: user.rgw.manifest 2016-09-22 11:24:36.697788 7fd435ae7700 10 cache get: name=.rgw+bucket : hit (requested=17, cached=23) 2016-09-22 11:24:36.697803 7fd435ae7700 20 rgw_get_bucket_info: old bucket info, bucket=bucket(@.rgw.data.1[88283.1]) owner CENSOR 2016-09-22 11:24:36.697819 7fd435ae7700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>CENSOR</ID><DisplayName>CENSOR</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="Group"><URI>http://acs.amazonaws.com/groups/global/AllUsers</URI></Grantee><Permission>READ</Permission></Grant><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>CENSOR</ID><DisplayName>CENSOR</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy> 2016-09-22 11:24:36.697827 7fd435ae7700 10 read_permissions on bucket(@.rgw.data.1[88283.1]) ret=-2 2016-09-22 11:24:36.697829 7fd435ae7700 20 op->ERRORHANDLER: err_no=-2 new_err_no=-2 2016-09-22 11:24:36.697913 7fd435ae7700 2 req 3:0.000240:s3:GET /object:get_obj:op status=0 2016-09-22 11:24:36.697923 7fd435ae7700 2 req 3:0.000250:s3:GET /object:get_obj:http status=404 2016-09-22 11:24:36.697931 7fd435ae7700 1 ====== req done req=0x7fd435ae17d0 op status=0 http_status=404 ====== 2016-09-22 11:24:36.697941 7fd435ae7700 20 process_request() returned -2 2016-09-22 11:24:36.697960 7fd435ae7700 1 civetweb: 0x7fd4e4001e20: 2607:f298:4:d006::5796 - - [22/Sep/2016:11:24:36 +0000] "GET /object HTTP/1.1" 404 0 - curl/7.35.0
Actions