Ceph

ubuntu@teuthology:/var/lib/teuthworker/archive/sage-2014-04-11_15:47:29-rados:thrash-testing-testing-basic-plana/185699

it appears that a rados op is hung on a missing object

HEALTH_WARN 4 pgs recovering; 4 pgs stuck unclean; 40 requests are blocked > 32 sec; 2 osds have slow requests; recovery 242/1356 objects degraded (17.847%); 'cache' at/near target max; pool metadata pg_num 74 > pgp_num 34
pg 3.1 is stuck unclean for 4231.401219, current state active+recovering, last acting [4,3]
pg 3.3 is stuck unclean for 4187.749857, current state active+recovering, last acting [4,5]
pg 4.1 is stuck unclean for 4238.101330, current state active+recovering, last acting [4,2]
pg 4.0 is stuck unclean for 4187.533232, current state active+recovering, last acting [4,5]
pg 4.1 is active+recovering, acting [4,2]
pg 4.0 is active+recovering, acting [4,5]
pg 3.1 is active+recovering, acting [4,3]
pg 3.3 is active+recovering, acting [4,5]
3 ops are blocked > 4194.3 sec
37 ops are blocked > 2097.15 sec
3 ops are blocked > 4194.3 sec on osd.4
33 ops are blocked > 2097.15 sec on osd.4
4 ops are blocked > 2097.15 sec on osd.5
2 osds have slow requests
recovery 242/1356 objects degraded (17.847%)
cache pool 'cache' with 246 objects at/near target max 250 objects
pool metadata pg_num 74 > pgp_num 34