Ceph » rgw

Bug #18940

Updated by Yehuda Sadeh about 7 years ago

I have a Ceph version 12.0.0 (b7d9d6eb542e2b946ac778bd3a381ce466f60f6a) cluster consisting of 1 node = 1 MON + 4 OSD + 1 RGW (civetweb). 

 I also have the following radosgw configuration. 

 [client.radosgw.client6] 
 rgw_frontends = "civetweb port=9000" 
 log file = /var/log/ceph/client.radosgw.client6.log 


 While doing cosbench on object storage , i see lot of failures during S3 PUT requests, cosbench (version 0.4.2.c3) s3 configure as follow:  

 <pre> 
 <?xml version="1.0" encoding="UTF-8" ?> 
 <workload name="s3-sample" description="sample benchmark for s3"> 

   <storage type="s3" config="accesskey=xxxxxxxxxxx;secretkey=xxxxxxxxxxxxxxxxxxxxxxxxxx;proxyport=9000;endpoint=http://172.17.6.191:9000" /> 

   <workflow> 

     <workstage name="init"> 
       <work type="init" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" /> 
     </workstage> 

     <workstage name="prepare"> 
       <work type="prepare" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,10);sizes=c(4)KB" /> 
     </workstage> 

     <workstage name="main"> 
       <work name="main" workers="2048" runtime="20"> 
         <operation type="read" ratio="0" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(1,10)" /> 
         <operation type="write" ratio="100" config="cprefix=s3testqwer11;containers=u(1,2);objects=u(11,2000000);sizes=c(4)KB" /> 
       </work> 
     </workstage> 

     <workstage name="cleanup"> 
       <work type="cleanup" workers="1" config="cprefix=s3testqwer11;containers=r(1,2);objects=r(1,20)" /> 
     </workstage> 

     <workstage name="dispose"> 
       <work type="dispose" workers="1" config="cprefix=s3testqwer11;containers=r(1,2)" /> 
     </workstage> 

   </workflow> 

 </workload> 
 </pre> 


 



 I have found a similary issue upstream http://tracker.ceph.com/issues/15727 not sure what's the progress going on with this. 

 Error logs provided below: 

 2017-02-13 15:45:28.880965 7f1c533ff700    1 civetweb: 0x7f1c51c0da00: 192.168.1.102 - - [13/Feb/2017:15:43:32 +0800] "PUT /s3testqwer112/myobjects1976910 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01 
 2017-02-13 15:45:58.698208 7f1c4a3ff700    0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error 
 2017-02-13 15:45:58.698360 7f1c4a3ff700    1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects1290024 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01 
 2017-02-13 15:45:58.705891 7f1c60fff700    0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error 
 2017-02-13 15:45:58.706067 7f1c4a3ff700    0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error 
 2017-02-13 15:45:58.706085 7f1c60fff700    1 civetweb: 0x7f1c5f00d7c0: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer111/myobjects907521 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01 
 2017-02-13 15:45:58.706190 7f1c4a3ff700    1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1231752 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01 
 2017-02-13 15:45:58.715360 7f1c4a3ff700    0 ERROR: RESTFUL_IO(s)->complete_header() returned err=Input/output error 
 2017-02-13 15:45:58.715483 7f1c4a3ff700    1 civetweb: 0x7f1c48c0d400: 192.168.1.102 - - [13/Feb/2017:15:45:58 +0800] "PUT /s3testqwer112/myobjects1084660 HTTP/1.1" 1 0 - aws-sdk-java/1.4.0.1 Linux/4.2.8 OpenJDK_64-Bit_Server_VM/24.111-b01

Back