Project

General

Profile

Bug #18940

ERROR RESTFUL_IO with S3 GET/PUT operations

Added by yan zhou almost 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
% Done:

0%

Source:
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature:

Description

I have a Ceph version 12.0.0 (b7d9d6eb542e2b946ac778bd3a381ce466f60f6a) cluster consisting of 1 node = 1 MON + 4 OSD + 1 RGW (civetweb).

I also have the following radosgw configuration.

[client.radosgw.client6]
rgw_frontends = "civetweb port=9000"
log file = /var/log/ceph/client.radosgw.client6.log

While doing cosbench on object storage , i see lot of failures during S3 PUT requests, cosbench (version 0.4.2.c3) s3 configure as follow:

<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-sample" description="sample benchmark for s3">

  <storage type="s3" config="accesskey=xxxxxxxxxxx;secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxx;proxyport=9000;endpoint=http://172.17.6.191:9000" />

  <workflow>

    <workstage name="init">
      <work type="init" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" />
    </workstage>

    <workstage name="prepare">
      <work type="prepare" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,10);sizes=c(4)KB" />
    </workstage>

    <workstage name="main">
      <work name="main" workers="2048" runtime="20">
        <operation type="read" ratio="0" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(1,10)" />
        <operation type="write" ratio="100" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(11,2000000);sizes=c(4)KB" />
      </work>
    </workstage>

    <workstage name="cleanup">
      <work type="cleanup" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,20)" />
    </workstage>

    <workstage name="dispose">
      <work type="dispose" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" />
    </workstage>

  </workflow>

</workload>

I have found a similary issue upstream http://tracker.ceph.com/issues/15727 not sure what's the progress going on with this.

Error logs provided below:

2017-02-13 15:45:28.880965 7f1c533ff700 1 civetweb: 0x7f1c51c0da00: 192.168.1.102 - - [13/Feb/2017:15:43:32 +0800] "PUT /s3testqwer112/myobjects1976910 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.698208 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.698360 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects1290024 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.705891 7f1c60fff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.706067 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.706085 7f1c60fff700 1 civetweb: 0x7f1c5f00d7c0: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects907521 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.706190 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1231752 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.715360 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.715483 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1084660 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01

History

#1 Updated by Nathan Cutler almost 3 years ago

  • Project changed from Ceph to rgw
  • Category deleted (22)

#2 Updated by Yehuda Sadeh almost 3 years ago

does cosbench report any error?

#3 Updated by yan zhou almost 3 years ago

Yehuda Sadeh wrote:

does cosbench report any error?

no cosbench report no error

#4 Updated by Yehuda Sadeh almost 3 years ago

  • Description updated (diff)

#5 Updated by Yehuda Sadeh almost 3 years ago

This isn't necessarily a bug, it's just client closing down the connection. Could happen when cosbench is shutting down.

#6 Updated by Casey Bodley almost 3 years ago

  • Status changed from New to Closed

Also available in: Atom PDF