Ceph

git-bisect shows that 8f104243f823c3b5c827a4c7e59d637d38846e3f broke test_unfound.sh, test_lost.sh, and probably some other unit tests.

The test output looks like this:

+ for i in '`seq -w 1 $num_objs`'
 + ./rados -p data put obj01 /tmp/tmp.dSNCR8Px0o/ver1
 + for i in '`seq -w 1 $num_objs`'
 + ./rados -p data put obj02 /tmp/tmp.dSNCR8Px0o/ver1
 + for i in '`seq -w 1 $num_objs`'
 + ./rados -p data put obj03 /tmp/tmp.dSNCR8Px0o/ver1
 + for i in '`seq -w 1 $num_objs`'
 + ./rados -p data put obj04 /tmp/tmp.dSNCR8Px0o/ver1
 + for i in '`seq -w 1 $num_objs`'
 + ./rados -p data put obj05 /tmp/tmp.dSNCR8Px0o/ver1
 error writing data/obj05: No such device or address
 + die 'radostool failed'
 + echo radostool failed
 radostool failed
 + exit 1
 + cleanup

After the test runs, I can replace obj01, obj02, obj03, or obj04 with new objects, or read old objects, but any attempt to create a new object fails.

Some interesting snippets from osd.0's log:

2011-01-05 15:48:38.554165 7fcd23fff710 osd0 6 request for pool=1 (metadata) owner=0 perm=7 may_read=0 may_write=1 may_exec=0 require_exec_caps=0
 2011-01-05 15:48:38.554175 7fcd23fff710 osd0 6 pg[1.3( empty n=0 ec=2 les=5 3/3/3) [1,0] r=1 active] misdirected op in 3
 2011-01-05 15:48:38.558943 7fcd23fff710 log [WRN] : mds0 10.3.14.10:6807/2947 misdirected mds0.1:19 1.3 to osd0 not [1,0]
 2011-01-05 15:48:38.558981 7fcd23fff710 -- 10.3.14.10:6800/2857 --> 10.3.14.10:6807/2947 -- osd_op_reply(19 mds0_sessionmap [writefull 0~17] ack = 6 (No such device or address)) v1 - ?+0 0x7fcd18
032050 con=0x29c8450
 2011-01-05 15:48:38.559053 7fcd237fe710 osd0 6 _dispatch 0x29d0f80 osd_sub_op(mds0.1:4 1.7 1.00000000/head [] v 6'3 snapset=0=[]:[] snapc=0=[]) v3
 2011-01-05 15:48:38.559088 7fcd23fff710 -- 10.3.14.10:6800/2857 <== mds0 10.3.14.10:6807/2947 9 ==== osd_op(mds0.1:20 mds_anchortable  [writefull 0~29] 1.f6a7) v1 ==== 123+0+29 (918054842 0 21879140
 5) 0x29cdfe0

There are a bunch of other places in the code where we see this from osd_op_reply.

Log file for osd.0 is attached.