Ceph

ubuntu@teuthology:/a/teuthology-2014-07-13_02:30:04-rados-next-testing-basic-plana/357668

see osd.2's log

2014-07-13 11:44:27.027817 7fd058b76700 10 osd.2 pg_epoch: 62 pg[3.8s2( v 58'134 (0'0,58'134] local-les=61 n=92 ec=8 les/c 61/61 60/60/60) [4,1,2] r=2 lpr=60 pi=8-59/6 luod=0'0 crt=0'0 active] handle_message: MOSDECSubOpWrite(3.8s2 61 ECSubWrite(tid=6, reqid=client.4118.0:319
7, at_version=61'135, trim_to=0'0)) v1
2014-07-13 11:44:27.027876 7fd058b76700 10 osd.2 pg_epoch: 62 pg[3.8s2( v 58'134 (0'0,58'134] local-les=61 n=93 ec=8 les/c 61/61 60/60/60) [4,1,2] r=2 lpr=60 pi=8-59/6 luod=0'0 crt=0'0 active] append_log log((0'0,58'134], crt=0'0) [61'135 (0'0) modify   976c3648/benchmark_dat
a_plana76_28610_object3196/head//3 by client.4118.0:3197 2014-07-13 11:44:26.825384]
2014-07-13 11:44:27.027926 7fd058b76700 10 osd.2 pg_epoch: 62 pg[3.8s2( v 61'135 (0'0,61'135] local-les=61 n=93 ec=8 les/c 61/61 60/60/60) [4,1,2] r=2 lpr=60 pi=8-59/6 luod=0'0 crt=0'0 lcod 58'134 active] add_log_entry 61'135 (0'0) modify   976c3648/benchmark_data_plana76_286
10_object3196/head//3 by client.4118.0:3197 2014-07-13 11:44:26.825384
2014-07-13 11:44:27.027969 7fd058b76700 10 osd.2 pg_epoch: 62 pg[3.8s2( v 61'135 (0'0,61'135] local-les=61 n=93 ec=8 les/c 61/61 60/60/60) [4,1,2] r=2 lpr=60 pi=8-59/6 luod=0'0 crt=0'0 lcod 58'134 active] append_log  adding 1 keys

the run also fails due to very slow recovery. not sure it is because this op got hung, but it did hang for

2014-07-13 11:52:27.185070 7fd06e4cf700  0 log [WRN] : slow request 480.358296 seconds old, received at 2014-07-13 11:44:26.826731: osd_op(client.4118.0:3197 benchmark_data_plana76_28610_object3196 [write 0~4194304] 3.976c3648 ack+ondisk+write+known_if_redirected e59) currently no flag points reached