Ceph » rgw

2016-08-04 06:38:56.838677 7f53860ed700  1 -- 172.21.15.52:0/3091810318 <== osd.1 172.21.15.52:6804/1510 25 ==== osd_op_reply(134005 .meta:bucket.instance:rwtest:b41d1e6c-7eb2-4251-8bb1-8d8cccd80f84.4139.1:_ahE5Ku3iTICJi94HI_grw_w:2 [delete] v345'14 uv9 ondisk = -2 ((2) No such file or directory)) v7 ==== 219+0+0 (1404692979 0 0) 0xe8ac000 con 0x11f4d800
2016-08-04 06:38:56.838687 7f53618a4700  1 civetweb: 0xb248000: 172.21.15.52 - - [04/Aug/2016:06:37:34 +0000] "PUT /rwtest/ HTTP/1.1" 500 0 - Boto/2.42.0 Python/2.7.6 Linux/3.13.0-92-generic
2016-08-04 06:38:56.838701 7f53610a3700  0 ERROR: store->delete_system_obj()=.meta:bucket.instance:rwtest:b41d1e6c-7eb2-4251-8bb1-8d8cccd80f84.4139.1:_ahE5Ku3iTICJi94HI_grw_w:2) returned ret=-2
2016-08-04 06:38:56.838705 7f53610a3700  0 ERROR: put_entry: remove_from_heap() key=rwtest:b41d1e6c-7eb2-4251-8bb1-8d8cccd80f84.4139.1 returned ret=-2
2016-08-04 06:38:56.838732 7f53610a3700  2 req 31104:3.647428:s3:PUT /rwtest/:put_acls:completing
2016-08-04 06:38:56.838739 7f53610a3700  0 WARNING: set_req_state_err err_no=125 resorting to 500
2016-08-04 06:38:56.838803 7f53610a3700  1 -- 172.21.15.52:0/3091810318 >> 172.21.15.61:6808/1669 conn(0x12a39000 sd=19 :-1 s=STATE_OPEN pgs=338 cs=1 l=1). == tx == 0x13e45c80 osd_op(unknown.0.0:134015 7.b6fcb7f0 2016-08-04-06-b41d1e6c-7eb2-4251-8bb1-8d8cccd80f84.4139.1-rwtest [append 0~301] snapc 0=[] ack+ondisk+write+known_if_redirected e345) v7
2016-08-04 06:38:56.838855 7f53610a3700  2 req 31104:3.647552:s3:PUT /rwtest/:put_acls:op status=-125
2016-08-04 06:38:56.838857 7f53610a3700  2 req 31104:3.647554:s3:PUT /rwtest/:put_acls:http status=500
2016-08-04 06:38:56.838858 7f53610a3700  1 ====== req done req=0x7f536109d810 op status=-125 http_status=500 ======

looks like we lost some race here, and didn't recover gracefully.

https://github.com/ceph/ceph/pull/11286