Ceph » rgw

Hi all,

I'm facing with the problem that I can put and delete objects with swift client but got above error when I download one of them.
And after that, I also saw my RGW have died with following log.

- Ceph version: 2:13.2.1-0.el7
- python-swiftclient: 3.5.0
- Download command: swift -V1 -A http://10.60.17.237:7480/auth/1.0 -U daikk115:swift -K XMqeutpPji0XMEPQx8yA01vBKjbALZY7qmQNMK8v download --output test.jpg 1cb70a5481c8496abb665bdf5b61a41f_7 xyz.jpg --debug

DEBUG:requests.packages.urllib3.connectionpool:http://10.60.17.237:7480 "GET /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg HTTP/1.1" 206 222544
DEBUG:swiftclient:REQ: curl -i http://10.60.17.237:7480/swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg -X GET -H "Range: bytes=85432-" -H "If-Match: 869a340aa533a71eba00cbaacffd3a40" -H "X-Auth-Token: AUTH_rgwtk0c0000006461696476333a73776966740b926327ce89ff49044f8a5bc0a893198a479534b0224c4e9d929fb9ee1c4a1a5bbb178e" 
DEBUG:swiftclient:RESP STATUS: 206 Partial Content
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'222544', u'Content-Type': u'image/jpeg', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 31 Aug 2018 08:15:07 GMT', u'Content-Range': u'bytes 85432-307975/307976', u'etag': u'869a340aa533a71eba00cbaacffd3a40', u'X-Timestamp': u'1535703307.41060', u'X-Trans-Id': u'tx000000000000000000001-005b88fd85-221bac1-default', u'Date': u'Fri, 31 Aug 2018 08:34:13 GMT', u'X-Object-Meta-Mtime': u'1535542580.954851', u'X-Openstack-Request-Id': u'tx000000000000000000001-005b88fd85-221bac1-default'}
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: 10.60.17.237
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTP connection (1): 10.60.17.237
DEBUG:requests.packages.urllib3.connectionpool:http://10.60.17.237:7480 "GET /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg HTTP/1.1" 206 122632
DEBUG:swiftclient:REQ: curl -i http://10.60.17.237:7480/swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg -X GET -H "Range: bytes=185344-" -H "If-Match: 869a340aa533a71eba00cbaacffd3a40" -H "X-Auth-Token: AUTH_rgwtk0c0000006461696476333a73776966740b926327ce89ff49044f8a5bc0a893198a479534b0224c4e9d929fb9ee1c4a1a5bbb178e" 
DEBUG:swiftclient:RESP STATUS: 206 Partial Content
DEBUG:swiftclient:RESP HEADERS: {u'Content-Length': u'122632', u'Content-Type': u'image/jpeg', u'Accept-Ranges': u'bytes', u'Last-Modified': u'Fri, 31 Aug 2018 08:15:07 GMT', u'Content-Range': u'bytes 185344-307975/307976', u'etag': u'869a340aa533a71eba00cbaacffd3a40', u'X-Timestamp': u'1535703307.41060', u'X-Trans-Id': u'tx000000000000000000001-005b88fd86-221bacd-default', u'Date': u'Fri, 31 Aug 2018 08:34:14 GMT', u'X-Object-Meta-Mtime': u'1535542580.954851', u'X-Openstack-Request-Id': u'tx000000000000000000001-005b88fd86-221bacd-default'}
DEBUG:requests.packages.urllib3.connectionpool:Resetting dropped connection: 10.60.17.237
ERROR:swiftclient.service:HTTPConnectionPool(host='10.60.17.237', port=7480): Max retries exceeded with url: /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6b13a934d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 1235, in _download_object_job
    for chunk in obj_body:
  File "/usr/lib/python2.7/site-packages/swiftclient/service.py", line 410, in __iter__
    for chunk in self._body:
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 266, in next
    buf = self.read(self.chunk_size)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 326, in read
    attempts=self.conn.attempts)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1691, in _retry
    service_token=self.service_token, **kwargs)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 1167, in get_object
    conn.request(method, path, '', headers)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 439, in request
    files=files, **self.requests_args)
  File "/usr/lib/python2.7/site-packages/swiftclient/client.py", line 422, in _request
    return self.request_session.request(*arg, **kwarg)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 518, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 639, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 502, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='10.60.17.237', port=7480): Max retries exceeded with url: /swift/v1/1cb70a5481c8496abb665bdf5b61a41f_7/xyz.jpg (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x7f6b13a934d0>: Failed to establish a new connection: [Errno 111] Connection refused',))
   0/ 5 objclass
   1/ 3 filestore
   1/ 3 journal
   0/ 5 ms
   1/ 5 mon
   0/10 monc
   1/ 5 paxos
   0/ 5 tp
   1/ 5 auth
   1/ 5 crypto
   1/ 1 finisher
   1/ 1 reserver
   1/ 5 heartbeatmap
   1/ 5 perfcounter
   1/ 5 rgw
   1/ 5 rgw_sync
   1/10 civetweb
   1/ 5 javaclient
   1/ 5 asok
   1/ 1 throttle
   0/ 0 refs
   1/ 5 xio
   1/ 5 compressor
   1/ 5 bluestore
   1/ 5 bluefs
   1/ 3 bdev
   1/ 5 kstore
   4/ 5 rocksdb
   4/ 5 leveldb
   4/ 5 memdb
   1/ 5 kinetic
   1/ 5 fuse
   1/ 5 mgr
   1/ 5 mgrc
   1/ 5 dpdk
   1/ 5 eventtrace
  -2/-2 (syslog threshold)
  -1/-1 (stderr threshold)
  max_recent     10000
  max_new         1000
  log_file /var/log/ceph/ceph-client.rgw.Compute207.log
--- end dump of recent events ---
2018-08-31 15:34:15.088 7f0f80abf8c0  0 deferred set uid:gid to 167:167 (ceph:ceph)
2018-08-31 15:34:15.088 7f0f80abf8c0  0 ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable), process radosgw, pid 940416
2018-08-31 15:34:15.417 7f0f80abf8c0  0 starting handler: civetweb
2018-08-31 15:34:15.574 7f0f80abf8c0  1 mgrc service_daemon_register rgw.Compute207 metadata {arch=x86_64,ceph_release=mimic,ceph_version=ceph version 13.2.1 (5533ecdc0fda920179d7ad84e0aa65a127b20d77) mimic (stable),ceph_version_short=13.2.1,cpu=Intel(R) Xeon(R) CPU E5-2658 0 @ 2.10GHz,distro=centos,distro_description=CentOS Linux 7 (Core),distro_version=7,frontend_config#0=civetweb port=7480 num_threads=2000 request_timeout_ms=600000 error_log_file=/var/log/radosgw/civetweb.error.log access_log_file=/var/log/radosgw/civetweb.access.log,frontend_type#0=civetweb,hostname=Compute207,kernel_description=#1 SMP Fri Apr 20 16:44:24 UTC 2018,kernel_version=3.10.0-862.el7.x86_64,mem_swap_kb=4194300,mem_total_kb=396030272,num_handles=1,os=Linux,pid=940416,zone_id=c4cdc84e-37e7-4f77-bcb2-3e02236b79e8,zone_name=default,zonegroup_id=6c04c8c9-1d41-48fe-b953-f30e176166fc,zonegroup_name=default}