Bug #19854
closedceph-fuse write a big file,The file is only written in part
0%
Description
application write a big file( 26GB) to cephfs, the file is only written in part(23GB);
ceph version: 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)
ceph-fuse verson: 10.2.6 (656b5b63ed7c43bd014bcafd81b001959d5f089f)
the client log file contain many error info:
2017-05-03 17:42:59.370411 7fd1e73cd700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.146:6805/716577 pipe(0x7fd205a7a800 sd=2 :11528 s=1 pgs=0 cs=0 l=1 c=0x7fd203f2e280).fault
2017-05-03 17:42:59.403699 7fd1e70ca700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.15:6800/1045522 pipe(0x7fd203aa2800 sd=2 :61026 s=1 pgs=0 cs=0 l=1 c=0x7fd203f2c900).fault
2017-05-03 17:42:59.437929 7fd1e6ec8700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.147:6800/708432 pipe(0x7fd203aa2800 sd=2 :54645 s=1 pgs=0 cs=0 l=1 c=0x7fd203f2e700).fault
2017-05-03 17:42:59.466523 7fd1e6cc6700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.147:6834/711698 pipe(0x7fd203aa2800 sd=2 :32720 s=1 pgs=0 cs=0 l=1 c=0x7fd203f2d380).fault
2017-05-03 17:42:59.517838 7fd1e4fc5700 0 cephx: verify_reply couldn't decrypt with error: error decoding block for decryption
2017-05-03 17:42:59.517844 7fd1e4fc5700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.17:6801/771266 pipe(0x7fd205c40000 sd=2 :64426 s=1 pgs=0 cs=0 l=1 c=0x7fd203e20f00).failed verifying authorize reply
2017-05-03 17:42:59.517885 7fd1e4fc5700 0 -- 10.150.0.41:0/2362030005 >> 10.150.0.17:6801/771266 pipe(0x7fd205c40000 sd=2 :64426 s=1 pgs=0 cs=0 l=1 c=0x7fd203e20f00).fault
Files
Updated by junming rao almost 7 years ago
- File client_cephfs.log client_cephfs.log added
upload client log file
Updated by Zheng Yan almost 7 years ago
what does 'written in part' mean? application wrote ~23G, failed to write the rest, or application wrote 26G but the file size is 23G.
Updated by junming rao almost 7 years ago
Zheng Yan wrote:
what does 'written in part' mean? application wrote ~23G, failed to write the rest, or application wrote 26G but the file size is 23G.
hi zheng yang:
application wrote 26G but the file size is 23G;
Updated by Zheng Yan almost 7 years ago
can you reproduce this issue? (errors in the client.log are normal, they shouldn't cause this issue)
Updated by junming rao almost 7 years ago
hi zheng yan:
reproduce this issue is easy, but I don't know what specific logs need to be opened at the client and server?
Updated by Zheng Yan almost 7 years ago
Zheng Yan wrote:
can you reproduce this issue? (errors in the client.log are normal, they shouldn't cause this issue)
try using 'ceph-fuse --debug_client=10 /mnt/xxx' to mount cephfs
Updated by Zheng Yan almost 7 years ago
Zheng Yan wrote:
can you reproduce this issue? (errors in the client.log are normal, they shouldn't cause this issue)
It will be very helpful if you can write a script to reproduce this issue
Updated by junming rao almost 7 years ago
hi zheng yan:
disable page cache at the client(fuse_disable_pagecache = true), problem disappear;
Updated by junming rao almost 7 years ago
hi zheng yan:
the problem occurs when multiple clients read/write a file at the same time in our environment；
Updated by Zheng Yan almost 7 years ago
- Status changed from New to Duplicate
It's a known issue. If you use ceph-fuse, and multiple client read/modify a file at the same time, you should disable page cache