Bug #34540
openGet object: Failed to establish a new connection: [Errno 111] Connection refused
0%
Description
Hi all,
I'm facing with the problem that I can put and delete objects with swift client but got above error when I download one of them.
And after that, I also saw my RGW have died with following log.
- Ceph version: 2:13.2.1-0.el7
- python-swiftclient: 3.5.0
- Download command: swift -V1 -A http://10.60.17.237:7480/auth/1.0 -U daikk115:swift -K XMqeutpPji0XMEPQx8yA01vBKjbALZY7qmQNMK8v download --output test.jpg 1cb70a5481c8496abb665bdf5b61a41f_7 xyz.jpg --debug
DEBUG:requests.packages.urllib3.connectionpool:http://10.60.17.237:7480 "GET /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg HTTP/1.1" 206 222544 DEBUG:swiftclient:REQ: curl -i http://10.60.17.237:7480/swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg -X GET -H "Range: bytes=85432-" -H "If-Match: 869a340aa533a71eba00cbaacffd3a40" -H "X-Auth-Token: AUTH_rgwtk0c0000006461696476333a73776966740b926327ce89ff49044f8a5bc0a893198a479534b0224c4e9d929fb9ee1c4a1a5bbb178e" DEBUG:swiftclient:RESP STATUS: 206 Partial Content DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'222544', u'Content-Type': u'image/jpeg', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 31 Aug 2018 08:15:07 GMT', u'Content-Range': u'bytes 85432-307975/307976', u'etag': u'869a340aa533a71eba00cbaacffd3a40', u'X-Timestamp': u'1535703307.41060', u'X-Trans-Id': u'tx000000000000000000001-005b88fd85-221bac1-default', u'Date': u'Fri, 31 Aug 2018 08:34:13 GMT', u'X-Object-Meta-Mtime': u'1535542580.954851', u'X-Openstack-Request-Id': u'tx000000000000000000001-005b88fd85-221bac1-default'} DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: 10.60.17.237 DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.60.17.237 DEBUG:requests.packages.urllib3.connectionpool:http://10.60.17.237:7480 "GET /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg HTTP/1.1" 206 122632 DEBUG:swiftclient:REQ: curl -i http://10.60.17.237:7480/swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg -X GET -H "Range: bytes=185344-" -H "If-Match: 869a340aa533a71eba00cbaacffd3a40" -H "X-Auth-Token: AUTH_rgwtk0c0000006461696476333a73776966740b926327ce89ff49044f8a5bc0a893198a479534b0224c4e9d929fb9ee1c4a1a5bbb178e" DEBUG:swiftclient:RESP STATUS: 206 Partial Content DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'122632', u'Content-Type': u'image/jpeg', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 31 Aug 2018 08:15:07 GMT', u'Content-Range': u'bytes 185344-307975/307976', u'etag': u'869a340aa533a71eba00cbaacffd3a40', u'X-Timestamp': u'1535703307.41060', u'X-Trans-Id': u'tx000000000000000000001-005b88fd86-221bacd-default', u'Date': u'Fri, 31 Aug 2018 08:34:14 GMT', u'X-Object-Meta-Mtime': u'1535542580.954851', u'X-Openstack-Request-Id': u'tx000000000000000000001-005b88fd86-221bacd-default'} DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: 10.60.17.237 ERROR:swiftclient.service:HTTPConnectionPool(host='10.60.17.237', port=7480): Max retries exceeded with url: /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6b13a934d0>: Failed to establish a new connection: [Errno 111] Connection refused',)) Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 1235, in _download_object_job for chunk in obj_body: File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 410, in __iter__ for chunk in self._body: File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 266, in next buf = self.read(self.chunk_size) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 326, in read attempts=self.conn.attempts) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry service_token=self.service_token, **kwargs) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1167, in get_object conn.request(method, path, '', headers) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 439, in request files=files, **self.requests_args) File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 422, in _request return self.request_session.request(*arg, **kwarg) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request resp = self.send(prep, **send_kwargs) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send r = adapter.send(request, **kwargs) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 502, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='10.60.17.237', port=7480): Max retries exceeded with url: /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6b13a934d0>: Failed to establish a new connection: [Errno 111] Connection refused',)) 0/ 5 objclass 1/ 3 filestore 1/ 3 journal 0/ 5 ms 1/ 5 mon 0/10 monc 1/ 5 paxos 0/ 5 tp 1/ 5 auth 1/ 5 crypto 1/ 1 finisher 1/ 1 reserver 1/ 5 heartbeatmap 1/ 5 perfcounter 1/ 5 rgw 1/ 5 rgw_sync 1/10 civetweb 1/ 5 javaclient 1/ 5 asok 1/ 1 throttle 0/ 0 refs 1/ 5 xio 1/ 5 compressor 1/ 5 bluestore 1/ 5 bluefs 1/ 3 bdev 1/ 5 kstore 4/ 5 rocksdb 4/ 5 leveldb 4/ 5 memdb 1/ 5 kinetic 1/ 5 fuse 1/ 5 mgr 1/ 5 mgrc 1/ 5 dpdk 1/ 5 eventtrace -2/-2 (syslog threshold) -1/-1 (stderr threshold) max_recent 10000 max_new 1000 log_file /var/log/ceph/ceph-client.rgw.Compute207.log --- end dump of recent events --- 2018-08-31 15:34:15.088 7f0f80abf8c0 0 deferred set uid:gid to 167:167 (ceph:ceph) 2018-08-31 15:34:15.088 7f0f80abf8c0 0 ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable), process radosgw, pid 940416 2018-08-31 15:34:15.417 7f0f80abf8c0 0 starting handler: civetweb 2018-08-31 15:34:15.574 7f0f80abf8c0 1 mgrc service_daemon_register rgw.Compute207 metadata {arch=x86_64,ceph_release=mimic,ceph_version=ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable),ceph_version_short=13.2.1,cpu=Intel(R) Xeon(R) CPU E5-2658 0 @ 2.10GHz,distro=centos,distro_description=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=civetweb port=7480 num_threads=2000 request_timeout_ms=600000 error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log,frontend_type#0=civetweb,hostname=Compute207,kernel_description=#1 SMP Fri Apr 20 16:44:24 UTC 2018,kernel_version=3.10.0-862.el7.x86_64,mem_swap_kb=4194300,mem_total_kb=396030272,num_handles=1,os=Linux,pid=940416,zone_id=c4cdc84e-37e7-4f77-bcb2-3e02236b79e8,zone_name=default,zonegroup_id=6c04c8c9-1d41-48fe-b953-f30e176166fc,zonegroup_name=default}
Updated by Dai Dang Van over 5 years ago
Dai Dang Van wrote:
Hi all,
I'm facing with the problem that I can put and delete objects with swift client but got above error when I download one of them.
And after that, I also saw my RGW have died with following log.- Ceph version: 2:13.2.1-0.el7
- python-swiftclient: 3.5.0
- Download command: swift -V1 -A http://10.60.17.237:7480/auth/1.0 -U daikk115:swift -K XMqeutpPji0XMEPQx8yA01vBKjbALZY7qmQNMK8v download --output test.jpg 1cb70a5481c8496abb665bdf5b61a41f_7 xyz.jpg --debug[...]
By the way, my image size is 307976
And everything is OK with bellow test command:
swift-bench -c 64 -s 4096 -n 1000 -g 100 swift.conf
I wonder shoud I make some changes in ceph configuration file to fit with my real image size?
Updated by Abhishek Lekshmanan over 5 years ago
Can you paste a larger section of RGW log (including the part just before the coredump) most likely is https://github.com/ceph/ceph/pull/24027
Updated by Abhishek Lekshmanan over 5 years ago
- Status changed from New to Need More Info