Project

General

Profile

Actions
Copy link

Bug #45013

closed

Nautilus: Intermittent dropped connections with iterate_obj() failed

Added by Alex Walender about 4 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Or Friedmann
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
Ceph - v14.2.6
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Hello,

Our cluster heavily uses S3 and we notice intermittent dropped connections in HAProxy error logs:

Mar 31 13:36:21 xxx haproxy[30442]: xxx.xx.xx.14:43836 [31/Mar/2020:13:36:08.606] rgw_8080_public~ rgw_8080_public/xxx 0/0/0/165/13022 200 2832351577 - - CD-- 1577/2/0/0/0 0/0 "GET /s3debug/file.txt HTTP/1.1

Corresponding to this, we find errors in rgw logs:

2020-04-01 14:25:54.043 7f7892a51700 0 iterate_obj() failed with -5

2020-04-01 14:25:54.043 7f7892a51700 1 ====== req done req=0x7f7892a4a8d0 op status=-5 http_status=200 latency=9.82881s ======

Sometimes the latency even reaches 100+ seconds. I'm not sure if this is really a rgw or a librados issue.
When this error occurs, the client (minio-client or s3cmd) reports that the peer has dropped the connection.
It seems that this randomly happens when downloading data. I made a script, which downloads the same file over and over again. A few times per hour this error occurs.

The ceph cluster itself is in a healthy state. Additionally, we found dropped packages in our infrastructure. Fixing this however did not resolve this described issue here.

RGW-Config:

[client.rgw.xxx]
rgw_frontends = "beast endpoint=xxx:8080" 

rgw keystone api version = 3
rgw keystone url = http://XXXX:5000
rgw keystone admin user = XXX
rgw keystone admin password = XXX
rgw keystone admin domain = default
rgw keystone admin project = service
rgw keystone accepted roles = admin, _member_, user
rgw keystone token cache size = 1000
rgw keystone revocation interval = 300
rgw s3 auth use keystone = true
rgw keystone verify ssl = false

Actions
Copy link

Also available in: Atom PDF