Actions
Bug #13817
openRGW does not send HTTP header
Added by Ian Babrou over 8 years ago. Updated over 7 years ago.
% Done:
0%
Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):
Description
ivan@myhost:~$ curl -v -H "x-auth-token: ABC" http://myhost:9064/swift/v1/docker-registry-v2/files/docker/registry/v2/repositories/myrepo/_uploads/08c55423-abde-4759-b674-98e8a14ca15e/data * Hostname was NOT found in DNS cache * Trying 1.2.3.4... * Connected to myhost (1.2.3.4) port 9064 (#0) > GET /swift/v1/docker-registry-v2/files/docker/registry/v2/repositories/myrepo/_uploads/08c55423-abde-4759-b674-98e8a14ca15e/data HTTP/1.1 > User-Agent: curl/7.38.0 > Host: myhost:9064 > Accept: */* > x-auth-token: ABC > Content-Length: 0 Date: Tue, 17 Nov 2015 10:49:58 GMT * Connection #0 to host myhost left intact
nginx in front of this rgw rightfully complains:
upstream sent no valid HTTP/1.0 header while reading response header from upstream, client: 5.6.7.8, server: swift.mydomain, request: "GET /swift/v1/docker-registry-v2/files/docker/registry/v2/repositories/myrepo/_uploads/08c55423-abde-4759-b674-98e8a14ca15e/data HTTP/1.1", upstream: "http://1.2.3.4:9064/swift/v1/docker-registry-v2/files/docker/registry/v2/repositories/myrepo/_uploads/08c55423-abde-4759-b674-98e8a14ca15e/data", host: "swift.mydomain"
This doesn't happen to all files, only to some of them.
ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43)
I moved files to the different directory for further investigation.
Updated by Yehuda Sadeh over 8 years ago
Are you sure that you actually reach rgw? Does the rgw log show anything? (add 'debug rgw = 20', 'debug civetweb = 10')
Updated by Ian Babrou over 8 years ago
Yes, I'm hitting rgw directly:
2015-11-17 19:59:50.424514 7fecfc8b8840 0 ceph version 0.94.5 (9764da52395923e0b32908d83a9f7304401fee43), process radosgw, pid 312
2015-11-17 19:59:52.065408 7fece27fc700 2 RGWDataChangesLog::ChangesRenewThread: start
2015-11-17 19:59:52.067859 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fd40 obj=.rgw.root:default.region state=0x3c7b0a0 s->prefetch_data=0
2015-11-17 19:59:54.035987 7fecfc8b8840 20 get_obj_state: s->obj_tag was set empty
2015-11-17 19:59:54.036027 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fd40 obj=.rgw.root:default.region state=0x3c7b0a0 s->prefetch_data=0
2015-11-17 19:59:54.036031 7fecfc8b8840 20 rados->read ofs=0 len=524288
2015-11-17 19:59:54.246046 7fecfc8b8840 20 rados->read r=0 bl.length=17
2015-11-17 19:59:54.246689 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fd40 obj=.rgw.root:region_info.default state=0x3c7b070 s->prefetch_data=0
2015-11-17 19:59:54.457171 7fecfc8b8840 20 get_obj_state: s->obj_tag was set empty
2015-11-17 19:59:54.457180 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fd40 obj=.rgw.root:region_info.default state=0x3c7b070 s->prefetch_data=0
2015-11-17 19:59:54.457182 7fecfc8b8840 20 rados->read ofs=0 len=524288
2015-11-17 19:59:54.670557 7fecfc8b8840 20 rados->read r=0 bl.length=153
2015-11-17 19:59:54.672201 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fee0 obj=.rgw.root:zone_info.default state=0x3c7b070 s->prefetch_data=0
2015-11-17 19:59:55.721003 7fecfc8b8840 20 get_obj_state: s->obj_tag was set empty
2015-11-17 19:59:55.721012 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fee0 obj=.rgw.root:zone_info.default state=0x3c7b070 s->prefetch_data=0
2015-11-17 19:59:55.721015 7fecfc8b8840 20 rados->read ofs=0 len=524288
2015-11-17 19:59:55.931339 7fecfc8b8840 20 rados->read r=0 bl.length=678
2015-11-17 19:59:55.931370 7fecfc8b8840 2 zone default is master
2015-11-17 19:59:55.931387 7fecfc8b8840 20 get_obj_state: rctx=0x7fff4c87fee0 obj=.rgw.root:region_map state=0x3c7b070 s->prefetch_data=0
2015-11-17 19:59:58.305978 7fecfc8b8840 20 add_watcher() i=0
2015-11-17 19:59:59.648486 7fecfc8b8840 20 add_watcher() i=1
2015-11-17 20:00:01.164596 7fecfc8b8840 20 add_watcher() i=2
2015-11-17 20:00:03.421971 7fecfc8b8840 20 add_watcher() i=3
2015-11-17 20:00:04.553138 7fecfc8b8840 20 add_watcher() i=4
2015-11-17 20:00:06.023179 7fecfc8b8840 20 add_watcher() i=5
2015-11-17 20:00:07.564240 7fecfc8b8840 20 add_watcher() i=6
2015-11-17 20:00:08.072190 7fecfc8b8840 20 add_watcher() i=7
2015-11-17 20:00:08.072197 7fecfc8b8840 2 all 8 watchers are set, enabling cache
2015-11-17 20:00:08.072372 7fece0ef1700 2 garbage collection: start
2015-11-17 20:00:08.072559 7fecfc8b8840 20 init_complete bucket index max shards: 0
2015-11-17 20:00:08.073470 7fecab7fe700 20 UserSyncThread: start
2015-11-17 20:00:08.074736 7fecabfff700 20 BucketsSyncThread: start
2015-11-17 20:00:08.076343 7fecfc8b8840 0 framework: civetweb
2015-11-17 20:00:08.076380 7fecfc8b8840 0 framework conf key: port, val: 12345
2015-11-17 20:00:08.076394 7fecfc8b8840 0 starting handler: civetweb
2015-11-17 20:00:08.076405 7fecfc8b8840 20 civetweb config: decode_url: no
2015-11-17 20:00:08.076406 7fecfc8b8840 20 civetweb config: enable_keep_alive: yes
2015-11-17 20:00:08.076423 7fecfc8b8840 20 civetweb config: listening_ports: 12345
2015-11-17 20:00:08.076425 7fecfc8b8840 20 civetweb config: num_threads: 100
2015-11-17 20:00:09.120345 7fecab7fe700 20 RGWRados::pool_iterate: got ivan
2015-11-17 20:00:09.120356 7fecab7fe700 20 RGWRados::pool_iterate: got ivan.buckets
2015-11-17 20:00:10.143901 7fecab7fe700 20 RGWRados::pool_iterate: got docker-registry-v2.buckets
2015-11-17 20:00:12.399131 7fecab7fe700 20 RGWRados::pool_iterate: got docker-registry-v2
2015-11-17 20:00:14.066865 7fece27fc700 2 RGWDataChangesLog::ChangesRenewThread: start
2015-11-17 20:00:14.949637 7fecab7fe700 20 RGWUserStatsCache: sync user=ivan
2015-11-17 20:00:15.158507 7fecab7fe700 20 user is idle, not doing a full sync (user=ivan)
2015-11-17 20:00:15.158515 7fecab7fe700 20 RGWUserStatsCache: sync user=docker-registry-v2
2015-11-17 20:00:15.581526 7fecab7fe700 20 reading from .rgw:.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:15.581545 7fecab7fe700 20 get_obj_state: rctx=0x7fecab7fcde0 obj=.rgw:.bucket.meta.docker-registry-v2:default.24443.4 state=0x7fecd40440e0 s->prefetch_data=0
2015-11-17 20:00:15.581553 7fecab7fe700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : miss
2015-11-17 20:00:15.710673 7feca8ff9700 20 RGWEnv::set(): HTTP_HOST: dev:12345
2015-11-17 20:00:15.710700 7feca8ff9700 20 RGWEnv::set(): HTTP_USER_AGENT: curl/7.43.0
2015-11-17 20:00:15.710704 7feca8ff9700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2015-11-17 20:00:15.710706 7feca8ff9700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: AUTH_rgwtk18000000646f636b65722d72656769737472792d76323a737769667426c8fae89367b01591574c56b2b4d004ec6f0326a5b68d44602d5734244ea25060d50beb
2015-11-17 20:00:15.710713 7feca8ff9700 20 RGWEnv::set(): REQUEST_METHOD: GET
2015-11-17 20:00:15.710715 7feca8ff9700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710717 7feca8ff9700 20 RGWEnv::set(): QUERY_STRING:
2015-11-17 20:00:15.710719 7feca8ff9700 20 RGWEnv::set(): REMOTE_USER:
2015-11-17 20:00:15.710721 7feca8ff9700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710735 7feca8ff9700 20 RGWEnv::set(): SERVER_PORT: 12345
2015-11-17 20:00:15.710737 7feca8ff9700 20 HTTP_ACCEPT=*/*
2015-11-17 20:00:15.710739 7feca8ff9700 20 HTTP_HOST=dev:12345
2015-11-17 20:00:15.710740 7feca8ff9700 20 HTTP_USER_AGENT=curl/7.43.0
2015-11-17 20:00:15.710741 7feca8ff9700 20 HTTP_X_AUTH_TOKEN=AUTH_rgwtk18000000646f636b65722d72656769737472792d76323a737769667426c8fae89367b01591574c56b2b4d004ec6f0326a5b68d44602d5734244ea25060d50beb
2015-11-17 20:00:15.710742 7feca8ff9700 20 QUERY_STRING=
2015-11-17 20:00:15.710743 7feca8ff9700 20 REMOTE_USER=
2015-11-17 20:00:15.710744 7feca8ff9700 20 REQUEST_METHOD=GET
2015-11-17 20:00:15.710745 7feca8ff9700 20 REQUEST_URI=/swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710747 7feca8ff9700 20 SCRIPT_URI=/swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710748 7feca8ff9700 20 SERVER_PORT=12345
2015-11-17 20:00:15.710751 7feca8ff9700 20 RGWEnv::set(): HTTP_HOST: dev:12345
2015-11-17 20:00:15.710753 7feca8ff9700 20 RGWEnv::set(): HTTP_USER_AGENT: curl/7.43.0
2015-11-17 20:00:15.710755 7feca8ff9700 20 RGWEnv::set(): HTTP_ACCEPT: */*
2015-11-17 20:00:15.710909 7feca8ff9700 20 RGWEnv::set(): HTTP_X_AUTH_TOKEN: AUTH_rgwtk18000000646f636b65722d72656769737472792d76323a737769667426c8fae89367b01591574c56b2b4d004ec6f0326a5b68d44602d5734244ea25060d50beb
2015-11-17 20:00:15.710921 7feca8ff9700 20 RGWEnv::set(): REQUEST_METHOD: GET
2015-11-17 20:00:15.710923 7feca8ff9700 20 RGWEnv::set(): REQUEST_URI: /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710925 7feca8ff9700 20 RGWEnv::set(): QUERY_STRING:
2015-11-17 20:00:15.710927 7feca8ff9700 20 RGWEnv::set(): REMOTE_USER:
2015-11-17 20:00:15.710928 7feca8ff9700 20 RGWEnv::set(): SCRIPT_URI: /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710933 7feca8ff9700 20 RGWEnv::set(): SERVER_PORT: 12345
2015-11-17 20:00:15.710935 7feca8ff9700 20 HTTP_ACCEPT=*/*
2015-11-17 20:00:15.710936 7feca8ff9700 20 HTTP_HOST=dev:12345
2015-11-17 20:00:15.710938 7feca8ff9700 20 HTTP_USER_AGENT=curl/7.43.0
2015-11-17 20:00:15.710939 7feca8ff9700 20 HTTP_X_AUTH_TOKEN=AUTH_rgwtk18000000646f636b65722d72656769737472792d76323a737769667426c8fae89367b01591574c56b2b4d004ec6f0326a5b68d44602d5734244ea25060d50beb
2015-11-17 20:00:15.710940 7feca8ff9700 20 QUERY_STRING=
2015-11-17 20:00:15.710941 7feca8ff9700 20 REMOTE_USER=
2015-11-17 20:00:15.710942 7feca8ff9700 20 REQUEST_METHOD=GET
2015-11-17 20:00:15.710943 7feca8ff9700 20 REQUEST_URI=/swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710945 7feca8ff9700 20 SCRIPT_URI=/swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.710946 7feca8ff9700 20 SERVER_PORT=12345
2015-11-17 20:00:15.710948 7feca8ff9700 1 ====== starting new request req=0x7fecb0004840 =====
2015-11-17 20:00:15.710979 7feca8ff9700 2 req 1:0.000031::GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e::initializing for trans_id = tx000000000000000000001-00564b874f-fa84-default
2015-11-17 20:00:15.710987 7feca8ff9700 10 host=dev
2015-11-17 20:00:15.710989 7feca8ff9700 20 subdomain= domain= in_hosted_domain=0
2015-11-17 20:00:15.711050 7feca8ff9700 10 ver=v1 first=docker-registry-v2 req=screwed/08c55423-abde-4759-b674-98e8a14ca15e
2015-11-17 20:00:15.711057 7feca8ff9700 10 s->object=screwed/08c55423-abde-4759-b674-98e8a14ca15e s->bucket=docker-registry-v2
2015-11-17 20:00:15.711069 7feca8ff9700 2 req 1:0.000121:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e::getting op
2015-11-17 20:00:15.711074 7feca8ff9700 2 req 1:0.000126:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:authorizing
2015-11-17 20:00:15.711433 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff3840 obj=.users.swift:docker-registry-v2:swift state=0x7fecb0023680 s->prefetch_data=0
2015-11-17 20:00:15.711447 7feca8ff9700 10 cache get: name=.users.swift+docker-registry-v2:swift : miss
2015-11-17 20:00:15.788035 7fecab7fe700 10 cache put: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:15.788040 7fecab7fe700 10 adding .rgw+.bucket.meta.docker-registry-v2:default.24443.4 to cache LRU end
2015-11-17 20:00:15.788042 7fecab7fe700 10 updating xattr: name=user.rgw.acl bl.length()=205
2015-11-17 20:00:15.788046 7fecab7fe700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:15.788051 7fecab7fe700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : type miss (requested=17, cached=22)
2015-11-17 20:00:15.788054 7fecab7fe700 20 get_obj_state: rctx=0x7fecab7fcde0 obj=.rgw:.bucket.meta.docker-registry-v2:default.24443.4 state=0x7fecd40440e0 s->prefetch_data=0
2015-11-17 20:00:15.788065 7fecab7fe700 20 rados->read ofs=0 len=524288
2015-11-17 20:00:15.931889 7feca8ff9700 10 cache put: name=.users.swift+docker-registry-v2:swift
2015-11-17 20:00:15.931896 7feca8ff9700 10 adding .users.swift+docker-registry-v2:swift to cache LRU end
2015-11-17 20:00:15.931901 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:15.931906 7feca8ff9700 10 cache get: name=.users.swift+docker-registry-v2:swift : type miss (requested=1, cached=6)
2015-11-17 20:00:15.931910 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff3840 obj=.users.swift:docker-registry-v2:swift state=0x7fecb0023680 s->prefetch_data=0
2015-11-17 20:00:15.931911 7feca8ff9700 20 rados->read ofs=0 len=524288
2015-11-17 20:00:16.014847 7fecab7fe700 20 rados->read r=0 bl.length=227
2015-11-17 20:00:16.014857 7fecab7fe700 10 cache put: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:16.014859 7fecab7fe700 10 moving .rgw+.bucket.meta.docker-registry-v2:default.24443.4 to cache LRU end
2015-11-17 20:00:16.151305 7feca8ff9700 20 rados->read r=0 bl.length=22
2015-11-17 20:00:16.151316 7feca8ff9700 10 cache put: name=.users.swift+docker-registry-v2:swift
2015-11-17 20:00:16.151320 7feca8ff9700 10 moving .users.swift+docker-registry-v2:swift to cache LRU end
2015-11-17 20:00:16.151334 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff35b0 obj=.users.uid:docker-registry-v2 state=0x7fecb0025780 s->prefetch_data=0
2015-11-17 20:00:16.151339 7feca8ff9700 10 cache get: name=.users.uid+docker-registry-v2 : miss
2015-11-17 20:00:16.354360 7feca8ff9700 10 cache put: name=.users.uid+docker-registry-v2
2015-11-17 20:00:16.354366 7feca8ff9700 10 adding .users.uid+docker-registry-v2 to cache LRU end
2015-11-17 20:00:16.354371 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:16.354375 7feca8ff9700 10 cache get: name=.users.uid+docker-registry-v2 : type miss (requested=17, cached=22)
2015-11-17 20:00:16.354379 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff35b0 obj=.users.uid:docker-registry-v2 state=0x7fecb0025780 s->prefetch_data=0
2015-11-17 20:00:16.354409 7feca8ff9700 20 rados->read ofs=0 len=524288
2015-11-17 20:00:16.572583 7feca8ff9700 20 rados->read r=0 bl.length=626
2015-11-17 20:00:16.572592 7feca8ff9700 10 cache put: name=.users.uid+docker-registry-v2
2015-11-17 20:00:16.572595 7feca8ff9700 10 moving .users.uid+docker-registry-v2 to cache LRU end
2015-11-17 20:00:16.572610 7feca8ff9700 10 chain_cache_entry: cache_locator=.users.uid+docker-registry-v2
2015-11-17 20:00:16.572618 7feca8ff9700 10 swift_user=docker-registry-v2:swift
2015-11-17 20:00:16.572628 7feca8ff9700 20 build_token token=18000000646f636b65722d72656769737472792d76323a737769667426c8fae89367b01591574c56b2b4d004
2015-11-17 20:00:16.572659 7feca8ff9700 2 req 1:0.861711:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:reading permissions
2015-11-17 20:00:16.572683 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff4160 obj=.rgw:docker-registry-v2 state=0x7fecb0025780 s->prefetch_data=0
2015-11-17 20:00:16.572687 7feca8ff9700 10 cache get: name=.rgw+docker-registry-v2 : miss
2015-11-17 20:00:16.780761 7feca8ff9700 10 cache put: name=.rgw+docker-registry-v2
2015-11-17 20:00:16.780767 7feca8ff9700 10 adding .rgw+docker-registry-v2 to cache LRU end
2015-11-17 20:00:16.780772 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:16.780774 7feca8ff9700 20 Read xattr: user.rgw.idtag
2015-11-17 20:00:16.780775 7feca8ff9700 20 Read xattr: user.rgw.manifest
2015-11-17 20:00:16.780778 7feca8ff9700 10 cache get: name=.rgw+docker-registry-v2 : type miss (requested=17, cached=22)
2015-11-17 20:00:16.780781 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff4160 obj=.rgw:docker-registry-v2 state=0x7fecb0025780 s->prefetch_data=0
2015-11-17 20:00:16.780799 7feca8ff9700 20 rados->read ofs=0 len=524288
2015-11-17 20:00:16.997336 7feca8ff9700 20 rados->read r=0 bl.length=163
2015-11-17 20:00:16.997350 7feca8ff9700 10 cache put: name=.rgw+docker-registry-v2
2015-11-17 20:00:16.997354 7feca8ff9700 10 moving .rgw+docker-registry-v2 to cache LRU end
2015-11-17 20:00:16.997365 7feca8ff9700 20 rgw_get_bucket_info: bucket instance: docker-registry-v2(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.24443.4])
2015-11-17 20:00:16.997372 7feca8ff9700 20 reading from .rgw:.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:16.997384 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff4160 obj=.rgw:.bucket.meta.docker-registry-v2:default.24443.4 state=0x7fecb00313d0 s->prefetch_data=0
2015-11-17 20:00:16.997390 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : type miss (requested=22, cached=19)
2015-11-17 20:00:17.202626 7feca8ff9700 10 cache put: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:17.202632 7feca8ff9700 10 moving .rgw+.bucket.meta.docker-registry-v2:default.24443.4 to cache LRU end
2015-11-17 20:00:17.202636 7feca8ff9700 10 updating xattr: name=user.rgw.acl bl.length()=205
2015-11-17 20:00:17.202642 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:17.202646 7feca8ff9700 20 Read xattr: user.rgw.acl
2015-11-17 20:00:17.202647 7feca8ff9700 20 Read xattr: user.rgw.idtag
2015-11-17 20:00:17.202648 7feca8ff9700 20 Read xattr: user.rgw.manifest
2015-11-17 20:00:17.202652 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : hit
2015-11-17 20:00:17.202667 7feca8ff9700 10 chain_cache_entry: cache_locator=.rgw+docker-registry-v2
2015-11-17 20:00:17.202669 7feca8ff9700 10 chain_cache_entry: cache_locator=.rgw+.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:17.202686 7feca8ff9700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>docker-registry-v2</ID><DisplayName>Docker Registry v2</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>docker-registry-v2</ID><DisplayName>Docker Registry v2</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-11-17 20:00:17.202699 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff4160 obj=docker-registry-v2:screwed/08c55423-abde-4759-b674-98e8a14ca15e state=0x7fecb002f0a0 s->prefetch_data=1
2015-11-17 20:00:18.254690 7feca8ff9700 10 manifest: total_size = 0
2015-11-17 20:00:18.254693 7feca8ff9700 20 get_obj_state: setting s->obj_tag to _qA34CpT8XQ3_pAQ3dzPMVB-xmOaf67Y
2015-11-17 20:00:18.254700 7feca8ff9700 15 Read AccessControlPolicy<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>docker-registry-v2</ID><DisplayName>Docker Registry v2</DisplayName></Owner><AccessControlList><Grant><Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"><ID>docker-registry-v2</ID><DisplayName>Docker Registry v2</DisplayName></Grantee><Permission>FULL_CONTROL</Permission></Grant></AccessControlList></AccessControlPolicy>
2015-11-17 20:00:18.254708 7feca8ff9700 2 req 1:2.543760:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:init op
2015-11-17 20:00:18.254712 7feca8ff9700 2 req 1:2.543764:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:verifying op mask
2015-11-17 20:00:18.254713 7feca8ff9700 20 required_mask= 1 user.op_mask=7
2015-11-17 20:00:18.254714 7feca8ff9700 2 req 1:2.543766:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:verifying op permissions
2015-11-17 20:00:18.254719 7feca8ff9700 5 Searching permissions for uid=docker-registry-v2 mask=49
2015-11-17 20:00:18.254720 7feca8ff9700 5 Found permission: 15
2015-11-17 20:00:18.254721 7feca8ff9700 5 Searching permissions for group=1 mask=49
2015-11-17 20:00:18.254721 7feca8ff9700 5 Permissions for group not found
2015-11-17 20:00:18.254722 7feca8ff9700 5 Searching permissions for group=2 mask=49
2015-11-17 20:00:18.254722 7feca8ff9700 5 Permissions for group not found
2015-11-17 20:00:18.254723 7feca8ff9700 5 Getting permissions id=docker-registry-v2 owner=docker-registry-v2 perm=1
2015-11-17 20:00:18.254723 7feca8ff9700 10 uid=docker-registry-v2 requested perm (type)=1, policy perm=1, user_perm_mask=15, acl perm=1
2015-11-17 20:00:18.254724 7feca8ff9700 2 req 1:2.543777:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:verifying op params
2015-11-17 20:00:18.254725 7feca8ff9700 2 req 1:2.543778:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:executing
2015-11-17 20:00:18.254732 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff4160 obj=docker-registry-v2:screwed/08c55423-abde-4759-b674-98e8a14ca15e state=0x7fecb002f0a0 s->prefetch_data=1
2015-11-17 20:00:18.254740 7feca8ff9700 20 Read xattr: user.rgw.acl
2015-11-17 20:00:18.254740 7feca8ff9700 20 Read xattr: user.rgw.content_type
2015-11-17 20:00:18.254741 7feca8ff9700 20 Read xattr: user.rgw.etag
2015-11-17 20:00:18.254741 7feca8ff9700 20 Read xattr: user.rgw.idtag
2015-11-17 20:00:18.254741 7feca8ff9700 20 Read xattr: user.rgw.manifest
2015-11-17 20:00:18.254742 7feca8ff9700 20 Read xattr: user.rgw.user_manifest
2015-11-17 20:00:18.254742 7feca8ff9700 20 Read xattr: user.rgw.x-amz-manifest
2015-11-17 20:00:18.254743 7feca8ff9700 2 RGWGetObj::handle_user_manifest() prefix=docker-registry-v2/segments/2f6/46f636b65722f72656769737472792f76322f7265706f7369746f726965732f656c737772697465722f5f75706c6f6164732f30386335353432332d616264652d343735392d623637342d3938653861313463613135652f646174616d8fd28d51d06cdddaa8eef47145a59f283e5310354243f68094f360f49ff3fbda39a3ee5e6b4b0d3255bfef95601890afd80709
2015-11-17 20:00:18.254748 7feca8ff9700 10 cls_bucket_list docker-registry-v2(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.24443.4]) start [] num_entries 101
2015-11-17 20:00:18.254753 7feca8ff9700 20 reading from .rgw:.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:18.254759 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff2bc0 obj=.rgw:.bucket.meta.docker-registry-v2:default.24443.4 state=0x7fecb0038c30 s->prefetch_data=0
2015-11-17 20:00:18.254763 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : hit
2015-11-17 20:00:18.254767 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:18.254768 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : hit
2015-11-17 20:00:18.459374 7feca8ff9700 10 cls_bucket_list docker-registry-v2(@{i=.rgw.buckets.index,e=.rgw.buckets.extra}.rgw.buckets[default.24443.4]) start [] num_entries 101
2015-11-17 20:00:18.459385 7feca8ff9700 20 reading from .rgw:.bucket.meta.docker-registry-v2:default.24443.4
2015-11-17 20:00:18.459395 7feca8ff9700 20 get_obj_state: rctx=0x7feca8ff2bc0 obj=.rgw:.bucket.meta.docker-registry-v2:default.24443.4 state=0x7fecb003b120 s->prefetch_data=0
2015-11-17 20:00:18.459400 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : hit
2015-11-17 20:00:18.459406 7feca8ff9700 20 get_obj_state: s->obj_tag was set empty
2015-11-17 20:00:18.459408 7feca8ff9700 10 cache get: name=.rgw+.bucket.meta.docker-registry-v2:default.24443.4 : hit
2015-11-17 20:00:18.669402 7feca8ff9700 2 req 1:2.958453:swift:GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e:get_obj:http status=200
2015-11-17 20:00:18.669430 7feca8ff9700 1 ====== req done req=0x7fecb0004840 http_status=200 ======
2015-11-17 20:00:18.669550 7feca8ff9700 1 civetweb: 0x7fecb0004b50: 192.168.99.1 - - [17/Nov/2015:20:00:15 +0000] "GET /swift/v1/docker-registry-v2/screwed/08c55423-abde-4759-b674-98e8a14ca15e HTTP/1.1" -1 0 - curl/7.43.0
Updated by Ka-Hing Cheung over 8 years ago
I noticed the same thing while diagnosing https://github.com/kahing/goofys/issues/46 . Seems like ceph doesn't send Content-Length when the file size is 0.
Updated by Ian Babrou about 8 years ago
Same issue with s3 rgw on ceph version 9.2.0 (bb2ecea240f3a1d525bcb35670cb07bd1f0ca299).
Updated by Ian Babrou about 8 years ago
I managed pin point the issue to this loop not happening for a sad object: https://github.com/ceph/ceph/blob/v9.2.0/src/rgw/rgw_op.cc#L755
Updated by Yehuda Sadeh over 7 years ago
Does this only happen when object size is 0?
Updated by Ian Babrou over 7 years ago
I'm not sure these objects were zero sized when uploaded, but I've only seen missing HTTP line with "Content-length: 0" from rgw.
Updated by Matt Benjamin over 7 years ago
- Status changed from New to In Progress
- Assignee set to Matt Benjamin
Actions