Ceph » rbd

qemu-kvm(X86_64) to access ARM's Ceph cluster, IO is error;
2019-05-24 14:38:29.504104 7f2ccd17b700 20 librbd::io::AioCompletion: 0x55fb2052ab60 complete_request: cb=1, pending=0
2019-05-24 14:38:29.504107 7f2ccd17b700 20 librbd::io::AioCompletion: 0x55fb2052ab60 finalize: r=0
2019-05-24 14:38:29.504112 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb2052ac98 finish_op
2019-05-24 14:38:29.504233 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 aio_write: ictx=0x55fb1f0c3600, completion=0x55fb211181a0, off=3198484480, len=65536, flags=0
2019-05-24 14:38:29.504247 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 queue: ictx=0x55fb1f0c3600, req=0x55fb20a06630
2019-05-24 14:38:29.504257 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb211182d8 start_op
2019-05-24 14:38:29.504260 7f2ccd17b700 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 process: ictx=0x55fb1f0c3600, req=0x55fb20a06630
2019-05-24 14:38:29.504261 7f2ccd17b700 20 librbd::io::ImageRequest: 0x55fb20a06630 send: aio_write: ictx=0x55fb1f0c3600, completion=0x55fb211181a0
2019-05-24 14:38:29.504262 7f2ccd17b700 -1 librbd::io::AioCompletion: 0x55fb211181a0 fail: (22) Invalid argument
2019-05-24 14:38:29.504274 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb211182d8 finish_op
2019-05-24 14:38:29.504314 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 aio_discard: ictx=0x55fb1f0c3600, completion=0x55fb20a631e0, off=3198484480, len=65536
2019-05-24 14:38:29.504325 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 queue: ictx=0x55fb1f0c3600, req=0x55fb1f43a300
2019-05-24 14:38:29.504336 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb20a63318 start_op
2019-05-24 14:38:29.504339 7f2ccd17b700 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 process: ictx=0x55fb1f0c3600, req=0x55fb1f43a300
2019-05-24 14:38:29.504340 7f2ccd17b700 20 librbd::io::ImageRequest: 0x55fb1f43a300 send: aio_discard: ictx=0x55fb1f0c3600, completion=0x55fb20a631e0
2019-05-24 14:38:29.504341 7f2ccd17b700 -1 librbd::io::AioCompletion: 0x55fb20a631e0 fail: (22) Invalid argument
2019-05-24 14:38:29.504350 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb20a63318 finish_op
2019-05-24 14:38:29.504458 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 aio_flush: ictx=0x55fb1f0c3600, completion=0x55fb21119040
2019-05-24 14:38:29.504468 7f2cec67e1c0 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 queue: ictx=0x55fb1f0c3600, req=0x55fb1f43a340
2019-05-24 14:38:29.504482 7f2ccd17b700 20 librbd::io::ImageRequestWQ: 0x55fb1f098ea0 process: ictx=0x55fb1f0c3600, req=0x55fb1f43a340
2019-05-24 14:38:29.504484 7f2ccd17b700 20 librbd::io::ImageRequest: 0x55fb1f43a340 send: aio_flush: ictx=0x55fb1f0c3600, completion=0x55fb21119040
2019-05-24 14:38:29.504486 7f2ccd17b700 20 librbd::io::AioCompletion: 0x55fb21119040 set_request_count: pending=1
2019-05-24 14:38:29.504488 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb21119178 start_op
2019-05-24 14:38:29.504489 7f2ccd17b700 20 librbd::io::AioCompletion: 0x55fb21119040 complete_request: cb=1, pending=0
2019-05-24 14:38:29.504490 7f2ccd17b700 20 librbd::io::AioCompletion: 0x55fb21119040 finalize: r=0
2019-05-24 14:38:29.504502 7f2ccd17b700 20 librbd::io::AsyncOperation: 0x55fb21119178 finish_op

qemu-kvm version: qemu-kvm-ev-2.9.0-16.el7_4.8.1.x86_64
Ceph Version: 12.2.11 for ARM

Mykola Golub wrote:

It is failing because the librbd thinks that the request offset 3198484480 (note, it is larger 2G, max 32 bit signed) is larger than the image size. What is the image size (it might be interesting to see `rbd info` from both arm and x86 nodes)? Dose the VM (qemu) reports the same size? Could you please attach the full librbd log?

Hello Mykola Golub
The image's format is qcow2, i forgot to convert the format to raw, The problem has been solved；
thank you very much.