Ceph

ubuntu@teuthology:/a/samuelj-2015-03-30_17:44:44-rados-wip-sam-testing-distro-basic-multi/829532/remote

2015-03-30 19:58:16.790747 7f0c8b336700 1 -- 10.214.132.30:6808/4566 <== client.4128 10.214.132.30:0/1024613 34 ==== osd_op(client.4128.0:6721 plana4824613-16@17f [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 2.64d2e308 ack+read+known_if_redirected e650) v5 ==== 364+0+32 (4268001709 0 2675358983) 0x654ba00 con 0x6f7c420
2015-03-30 19:58:16.791570 7f0ca7122700 1 -- 10.214.132.30:0/4566 --> 10.214.132.30:6808/4566 -- osd_op(osd.1.348:1214 plana4824613-16@17f [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 1.64d2e308 ack+read+ignore_cache+ignore_overlay+known_if_redirected e650) v5 -- ?+0 0x70c0a00 con 0x53a9dc0
2015-03-30 19:58:16.791828 7f0c8c439700 1 -- 10.214.132.30:6808/4566 <== osd.1 10.214.132.30:0/4566 233 ==== osd_op(osd.1.348:1214 plana4824613-16@17f [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 1.64d2e308 ack+read+ignore_cache+ignore_overlay+known_if_redirected e650) v5 ==== 364+0+32 (3671357830 0 2675358983) 0x741f500 con 0x6f7c6e0
2015-03-30 19:58:19.446417 7f0c96eeb700 1 -- 10.214.132.30:6808/4566 <== client.4128 10.214.132.30:0/1024613 9 ==== osd_op(client.4128.0:6721 plana4824613-16@17f [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] 2.64d2e308 RETRY=1 ack+retry+read+known_if_redirected e651) v5 ==== 364+0+32 (986055740 0 2675358983) 0xa820780 con 0x6f7c2c0
2015-03-30 19:58:19.447122 7f0ca491d700 1 -- 10.214.132.30:6808/4566 --> 10.214.132.30:0/1024613 -- osd_op_reply(6721 plana4824613-16 [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] v0'0 uv0 ack = 2 ((2) No such file or directory)) v6 - ?+0 0xb63ab00 con 0x6f7c2c0
2015-03-30 19:58:22.313310 7f0ca7122700 1 -- 10.214.132.30:6808/4566 --> 10.214.132.30:0/4566 -- osd_op_reply(1214 plana4824613-16 [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] v0'0 uv734 ondisk = 0) v6 -- ?+0 0x7154840 con 0x6f7c6e0
2015-03-30 19:58:22.316761 7f0c8bb37700 1 -- 10.214.132.30:0/4566 <== osd.1 10.214.132.30:6808/4566 214 ==== osd_op_reply(1214 plana4824613-16 [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] v0'0 uv734 ondisk = 0) v6 ==== 392+0+3173892 (904730137 0 1217463802) 0x77af840 con 0x53a9dc0
2015-03-30 19:58:22.330948 7f0c9f913700 1 -- 10.214.132.30:6808/4566 --> 10.214.132.30:0/1024613 -- osd_op_reply(6721 plana4824613-16 [read 0~3173806,omap-get-vals-by-keys 0~4,omap-get-keys 0~12,omap-get-vals 0~16,omap-get-header 0~0,getxattrs] v0'0 uv734 ondisk = 0) v6 -- ?+0 0xa37b580 con 0x6f7c420

Not sure what's going on yet, but it looks like the initial read op actually ends up succeeding after being proxied to the base tier. The retry, however, short circuits first with an ENOENT for some reason.