Bug #15314
closedfailed to decode message of type 43 v7: buffer::end_of_buffer
0%
Description
During the start of CBT cache tiering tests of v10.1.0 on incerta05-08, we're hitting a bug when cbt runs the rbd command to create the RBD images. It also appears to happen during other rbd commands such as "ls". I've included log snippets from the client and OSD with ms, objecter, and osd debugging turned up to 20. It appears that message 43 is osd_op_reply as can be seen in the OSD log (also, thanks Greg for pointing out that this message type is in ceph_fs.h, not message.h!)
If I have time, I'll see if I can dump the hex into binary and run it through ceph-dencoder. The issue is quite repeatable and can be caused simply by doing a "sudo rbd ls" on perf@incerta[05-08].
version:
ceph version 10.1.0 (96ae8bd25f31862dbd5302f304ebf8bf1166aba6)
command run:
sudo /usr/bin/rbd -c /home/perf/tmp/cbt/ceph/ceph.conf create cbt-librbdfio-incerta05-foo --size 131072 --pool cbt-librbdfio --order 22 --debug-rbd 20 --debug-ms 20 --debug-objecter 20
OSD message sent:
2016-03-29 19:16:35.181958 7f46a6dc6700 20 -- 10.0.10.107:6808/127105 >> 10.0.10.107:0/3904314947 pipe(0x7f46e10cc000 sd=174 :6808 s=2 pgs=314 cs=1 l=1 c=0x7f46e13c4c00).writer encoding 1 features 576460752032874495 0x7f46ddc8edc0 osd_op_reply(1 cbt-librbdfio-incerta05-foo.rbd [stat] v0'0 uv0 ack = -2 ((2) No such file or directory)) v7
Hex dump from the client:
2016-03-29 19:16:35.186561 7f756c3ca700 1 dump: 00000000 1f 00 00 00 63 62 74 2d 6c 69 62 72 62 64 66 69 |....cbt-librbdfi| 00000010 6f 2d 69 6e 63 65 72 74 61 30 35 2d 66 6f 6f 2e |o-incerta05-foo.| 00000020 72 62 64 01 02 00 00 00 00 00 00 00 c7 68 9a 98 |rbd..........h..| 00000030 ff ff ff ff 19 00 40 00 00 00 00 00 fe ff ff ff |......@.........| 00000040 00 00 00 00 00 00 00 00 00 00 00 00 79 00 00 00 |............y...| 00000050 01 00 00 00 02 12 00 00 00 00 00 00 00 00 00 00 |................| 00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000070 00 00 00 00 00 00 00 00 00 00 3b 01 00 00 00 00 |..........;.....| 00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000090 00 00 00 00 00 00 00 |.......| 00000097
Files