Ceph

During the start of CBT cache tiering tests of v10.1.0 on incerta05-08, we're hitting a bug when cbt runs the rbd command to create the RBD images. It also appears to happen during other rbd commands such as "ls". I've included log snippets from the client and OSD with ms, objecter, and osd debugging turned up to 20. It appears that message 43 is osd_op_reply as can be seen in the OSD log (also, thanks Greg for pointing out that this message type is in ceph_fs.h, not message.h!)

If I have time, I'll see if I can dump the hex into binary and run it through ceph-dencoder. The issue is quite repeatable and can be caused simply by doing a "sudo rbd ls" on perf@incerta[05-08].

version:

ceph version 10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)

command run:

sudo /usr/bin/rbd -c /home/perf/tmp/cbt/ceph/ceph.conf create cbt-librbdfio-incerta05-foo --size 131072 --pool cbt-librbdfio --order 22 --debug-rbd 20 --debug-ms 20 --debug-objecter 20

OSD message sent:

2016-03-29 19:16:35.181958 7f46a6dc6700 20 -- 10.0.10.107:6808/127105 >> 10.0.10.107:0/3904314947 pipe(0x7f46e10cc000 sd=174 :6808 s=2 pgs=314 cs=1 l=1 c=0x7f46e13c4c00).writer encoding 1 features 576460752032874495 0x7f46ddc8edc0 osd_op_reply(1 cbt-librbdfio-incerta05-foo.rbd [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7

Hex dump from the client:

2016-03-29 19:16:35.186561 7f756c3ca700  1 dump: 
00000000  1f 00 00 00 63 62 74 2d  6c 69 62 72 62 64 66 69  |....cbt-librbdfi|
00000010  6f 2d 69 6e 63 65 72 74  61 30 35 2d 66 6f 6f 2e  |o-incerta05-foo.|
00000020  72 62 64 01 02 00 00 00  00 00 00 00 c7 68 9a 98  |rbd..........h..|
00000030  ff ff ff ff 19 00 40 00  00 00 00 00 fe ff ff ff  |......@.........|
00000040  00 00 00 00 00 00 00 00  00 00 00 00 79 00 00 00  |............y...|
00000050  01 00 00 00 02 12 00 00  00 00 00 00 00 00 00 00  |................|
00000060  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000070  00 00 00 00 00 00 00 00  00 00 3b 01 00 00 00 00  |..........;.....|
00000080  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000090  00 00 00 00 00 00 00                              |.......|
00000097