Support #19308
open416 when Docker registry + Swift + rados gatway + ceph/daemon
0%
Description
We use registry:2.6 + Swift + ceph/daemon:tag-build-master-kraken-centos-7 to setup private docker registry.
we can push image success but pull failed and got the following message in the logs of ceph/daemon
1.
docker run -d -p 5000:5000 --name registry registry:2.6
2. docker run --name ac-ceph-rgw -d --net=host -v /var/lib/ceph/:/var/lib/ceph/ -v /etc/ceph:/etc/ceph ceph/daemon:tag-build-master-kraken-centos-7
2017-03-19 03:17:42.026844 7f6fbd684700 1 ====== starting new request req=0x7f6fbd67e340 =====
2017-03-19 03:17:42.026858 7f6fbd684700 2 req 119:0.000014::GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data::initializing for trans_id = tx000000000000000000077-0058cdf856-c8ea-default
2017-03-19 03:17:42.026863 7f6fbd684700 10 rgw api priority: s3=5 s3website=4
2017-03-19 03:17:42.026865 7f6fbd684700 10 host=10.94.64.103
2017-03-19 03:17:42.026866 7f6fbd684700 20 subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0
2017-03-19 03:17:42.026867 7f6fbd684700 20 final domain/bucket subdomain= domain= in_hosted_domain=0 in_hosted_domain_s3website=0 s->info.domain= s->info.request_uri=/swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data
2017-03-19 03:17:42.026879 7f6fbd684700 10 ver=v1 first=registry_swift111 req=files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data
2017-03-19 03:17:42.026882 7f6fbd684700 10 handler=25RGWHandler_REST_Obj_SWIFT
2017-03-19 03:17:42.026883 7f6fbd684700 2 req 119:0.000039:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data::getting op 0
2017-03-19 03:17:42.026886 7f6fbd684700 10 op=24RGWGetObj_ObjStore_SWIFT
2017-03-19 03:17:42.026887 7f6fbd684700 2 req 119:0.000042:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:authorizing
2017-03-19 03:17:42.026890 7f6fbd684700 5 trying auth engine: RGWSignedTokenAuthEngine
2017-03-19 03:17:42.026897 7f6fbd684700 10 swift_user=alphacloudrgw:swift
2017-03-19 03:17:42.026906 7f6fbd684700 20 build_token token=13000000616c706861636c6f75647267773a7377696674f8a58ca85bed2c94d2ffcd5808586116
2017-03-19 03:17:42.026938 7f6fbd684700 2 req 119:0.000093:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:normalizing buckets and tenants
2017-03-19 03:17:42.026942 7f6fbd684700 10 s->object=files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data s->bucket=registry_swift111
2017-03-19 03:17:42.026944 7f6fbd684700 2 req 119:0.000099:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:init permissions
2017-03-19 03:17:42.026956 7f6fbd684700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>alphacloudrgw</ID><DisplayName>alphacloudrgw</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>alphacloudrgw</ID><DisplayName>alphacloudrgw</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2017-03-19 03:17:42.026965 7f6fbd684700 20 get_system_obj_state: rctx=0x7f6fbd67d070 obj=default.rgw.users.uid:alphacloudrgw state=0x7f700b6c4498 s->prefetch_data=0
2017-03-19 03:17:42.026970 7f6fbd684700 10 cache get: name=default.rgw.users.uid+alphacloudrgw : hit (requested=0x6, cached=0x17)
2017-03-19 03:17:42.026973 7f6fbd684700 20 get_system_obj_state: s->obj_tag was set empty
2017-03-19 03:17:42.026974 7f6fbd684700 20 Read xattr: user.rgw.idtag
2017-03-19 03:17:42.026978 7f6fbd684700 2 req 119:0.000133:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:recalculating target
2017-03-19 03:17:42.026980 7f6fbd684700 10 Starting object retarget
2017-03-19 03:17:42.026995 7f6fbd684700 2 req 119:0.000138:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:reading permissions
2017-03-19 03:17:42.027004 7f6fbd684700 20 get_obj_state: rctx=0x7f6fbd67da60 obj=registry_swift111:files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data state=0x7f700b6c4498 s->prefetch_data=1
2017-03-19 03:17:42.027032 7f6fbd684700 1 -- 10.94.64.103:0/887126259 --> 10.94.64.97:6820/14967 -- osd_op(unknown.0.0:9511 17.4af68cd2 94d61149-d1d8-4c6a-af13-6a7731c8d89f.50882.4_files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data [getxattrs,stat,read 0~524288] snapc 0=[] ack+read+known_if_redirected e862) v7 -- 0x7f700c120940 con 0
2017-03-19 03:17:42.028129 7f6fee6e6700 1 -- 10.94.64.103:0/887126259 <== osd.23 10.94.64.97:6820/14967 345 ==== osd_op_reply(9511 94d61149-d1d8-4c6a-af13-6a7731c8d89f.50882.4_files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data [getxattrs,stat,read 0~0] v0'0 uv33 ondisk = 0) v7 ==== 359+0+1938 (3750747977 0 776209720) 0x7f700c120940 con 0x7f700b4ce000
2017-03-19 03:17:42.028197 7f6fbd684700 10 manifest: total_size = 0
2017-03-19 03:17:42.028201 7f6fbd684700 20 get_obj_state: setting s->obj_tag to 94d61149-d1d8-4c6a-af13-6a7731c8d89f.50924.900
2017-03-19 03:17:42.028207 7f6fbd684700 15 decode_policy Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>alphacloudrgw</ID><DisplayName>alphacloudrgw</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>alphacloudrgw</ID><DisplayName>alphacloudrgw</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2017-03-19 03:17:42.028212 7f6fbd684700 2 req 119:0.001368:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:init op
2017-03-19 03:17:42.028215 7f6fbd684700 2 req 119:0.001371:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:verifying op mask
2017-03-19 03:17:42.028217 7f6fbd684700 20 required_mask= 1 user.op_mask=7
2017-03-19 03:17:42.028218 7f6fbd684700 2 req 119:0.001374:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:verifying op permissions
2017-03-19 03:17:42.028222 7f6fbd684700 5 Searching permissions for identity=RGWThirdPartyAccountAuthApplier() > RGWLocalAuthApplier(acct_user=alphacloudrgw, acct_name=alphacloudrgw, subuser=swift, perm_mask=15, is_admin=>prefetch_data=1
2017-03-19 03:17:42.028225 7f6fbd684700 5 Searching permissions for uid=alphacloudrgw
2017-03-19 03:17:42.028226 7f6fbd684700 5 Found permission: 15
2017-03-19 03:17:42.028227 7f6fbd684700 5 Searching permissions for group=1 mask=49
2017-03-19 03:17:42.028228 7f6fbd684700 5 Permissions for group not found
2017-03-19 03:17:42.028229 7f6fbd684700 5 Searching permissions for group=2 mask=49
2017-03-19 03:17:42.028230 7f6fbd684700 5 Permissions for group not found
2017-03-19 03:17:42.028232 7f6fbd684700 5 Getting permissions identity=RGWThirdPartyAccountAuthApplier() -> RGWLocalAuthApplier(acct_user=alphacloudrgw, acct_name=alphacloudrgw, subuser=swift, perm_mask=15, is_admin=
2017-03-19 03:17:42.028234 7f6fbd684700 10 identity=RGWThirdPartyAccountAuthApplier() -> RGWLocalAuthApplier(acct_user=alphacloudrgw, acct_name=alphacloudrgw, subuser=swift, perm_mask=15, is_admin=
2017-03-19 03:17:42.028236 7f6fbd684700 2 req 119:0.001391:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:verifying op params
2017-03-19 03:17:42.028237 7f6fbd684700 2 req 119:0.001393:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:pre-executing
2017-03-19 03:17:42.028239 7f6fbd684700 2 req 119:0.001394:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:executing
2017-03-19 03:17:42.028244 7f6fbd684700 20 get_obj_state: rctx=0x7f6fbd67da60 obj=registry_swift111:files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data state=0x7f700b6c4498 s
2017-03-19 03:17:42.028252 7f6fbd684700 20 Read xattr: user.rgw.acl
2017-03-19 03:17:42.028253 7f6fbd684700 20 Read xattr: user.rgw.content_type
2017-03-19 03:17:42.028254 7f6fbd684700 20 Read xattr: user.rgw.etag
2017-03-19 03:17:42.028254 7f6fbd684700 20 Read xattr: user.rgw.idtag
2017-03-19 03:17:42.028255 7f6fbd684700 20 Read xattr: user.rgw.manifest
2017-03-19 03:17:42.028256 7f6fbd684700 20 Read xattr: user.rgw.pg_ver
2017-03-19 03:17:42.028256 7f6fbd684700 20 Read xattr: user.rgw.source_zone
2017-03-19 03:17:42.028257 7f6fbd684700 20 Read xattr: user.rgw.user_manifest
2017-03-19 03:17:42.028258 7f6fbd684700 20 Read xattr: user.rgw.x-amz-manifest
2017-03-19 03:17:42.028292 7f6fbd684700 2 req 119:0.001447:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:completing
2017-03-19 03:17:42.028296 7f6fbd684700 2 req 119:0.001452:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:op status=-34
2017-03-19 03:17:42.028299 7f6fbd684700 2 req 119:0.001454:swift:GET /swift/v1/registry_swift111/files/docker/registry/v2/blobs/sha256/51/511bd223022ae1fb3021b20bebfc820df0a377252ab64143cd4b02c994cb9dfa/data:get_obj:http status=416
2017-03-19 03:17:42.028302 7f6fbd684700 1 ====== req done req=0x7f6fbd67e340 op status=-34 http_status=416 ======
Updated by Nathan Cutler about 7 years ago
- Tracker changed from Tasks to Support
- Project changed from Stable releases to rgw
Updated by Alastair Dewhurst almost 7 years ago
Hi
I can't help, but I can at least report that we are seeing the exact same problem.
Since we upgraded to running Kraken 11.2.0 swift no longer works for docker registry. It was working when we were running Jewel 10.2.5.
Alastair