Ceph » rgw

I have a Ceph version 12.0.0 (b7d9d6eb542e2b946ac778bd3a381ce466f60f6a) cluster consisting of 1 node = 1 MON + 4 OSD + 1 RGW (civetweb).

I also have the following radosgw configuration.

[client.radosgw.client6]
rgw_frontends = "civetweb port=9000"
log file = /var/log/ceph/client.radosgw.client6.log

While doing cosbench on object storage , i see lot of failures during S3 PUT requests, cosbench (version 0.4.2.c3) s3 configure as follow:

<?xml version="1.0" encoding="UTF-8" ?>
<workload name="s3-sample" description="sample benchmark for s3">

  <storage type="s3" config="accesskey=xxxxxxxxxxx;secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxx;proxyport=9000;endpoint=http://172.17.6.191:9000" />

  <workflow>

    <workstage name="init">
      <work type="init" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" />
    </workstage>

    <workstage name="prepare">
      <work type="prepare" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,10);sizes=c(4)KB" />
    </workstage>

    <workstage name="main">
      <work name="main" workers="2048" runtime="20">
        <operation type="read" ratio="0" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(1,10)" />
        <operation type="write" ratio="100" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(11,2000000);sizes=c(4)KB" />
      </work>
    </workstage>

    <workstage name="cleanup">
      <work type="cleanup" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,20)" />
    </workstage>

    <workstage name="dispose">
      <work type="dispose" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" />
    </workstage>

  </workflow>

</workload>

I have found a similary issue upstream http://tracker.ceph.com/issues/15727 not sure what's the progress going on with this.

Error logs provided below:

2017-02-13 15:45:28.880965 7f1c533ff700 1 civetweb: 0x7f1c51c0da00: 192.168.1.102 - - [13/Feb/2017:15:43:32 +0800] "PUT /s3testqwer112/myobjects1976910 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.698208 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.698360 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects1290024 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.705891 7f1c60fff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.706067 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.706085 7f1c60fff700 1 civetweb: 0x7f1c5f00d7c0: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects907521 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.706190 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1231752 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01
2017-02-13 15:45:58.715360 7f1c4a3ff700 0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error
2017-02-13 15:45:58.715483 7f1c4a3ff700 1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1084660 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01