Bug #1226
closed
kclient can't read newly-created file with a custom layout
Added by Greg Farnum almost 13 years ago.
Updated almost 13 years ago.
Description
After using cephfs to set a custom layout on a file, the kernel client can't read that file until the file has been written to. It can write to the file, and you can usually (always?) cancel the read, but any attempted reads will hang.
It doesn't matter if the custom layout is the same as the dir layout. If it inherits a dir layout that is different from the system default layout, it can read it.
Setting a custom layout via the kernel client does not prevent cfuse from reading the empty file. Doing so is insufficient to let the kclient read it, though. Similarly cfuse can write to the file (but I believe leads to the kclient getting substantially more stuck if it tries to access that file).
From a brief glance at logs this doesn't appear to be an MDS problem, since the kclient attempt to open the file succeeds, and cfuse can open it.
- Assignee set to Greg Farnum
From a brief look I think there's a problem with the way it's dropping and refreshing caps to effect the layout change.
- Status changed from New to In Progress
Aha, or perhaps not the caps...
[ 41.660000] ceph: readpage inode 00000000a036e328 file 000000009fe62d00 page 0000000062faf568 index 0
[ 41.660000] libceph: readpages on ino 10000000004.fffffffffffffffe on 0~4096
[ 41.660000] libceph: mapping 0~4096 osize 2097152 fl_su 1048576
[ 41.660000] libceph: osize 2097152 / su 1048576 = su_per_object 2
[ 41.660000] libceph: off 0 / su 1048576 = bl 0
[ 41.660000] libceph: objset 0 * sc 2 = ono 0
[ 41.660000] libceph: obj extent 0~4096
[ 41.660000] libceph: calc_layout bno=0 0~4096 (1 pages)
[ 41.660000] libceph: readpages final extent is 0~4096 (1 pages align 0)
[ 41.660000] libceph: __register_request 00000000a053d400 tid 1
[ 41.660000] libceph: first request, scheduling timeout
[ 41.660000] libceph: map_request 00000000a053d400 tid 1
[ 41.660000] libceph: calc_object_layout '10000000004.00000000' pgid 0.17c5e6c5p0
[ 41.660000] libceph: send_request 00000000a053d400 no up osds in pg
[ 41.660000] libceph: request_next_osdmap have 2
[ 41.660000] libceph: __send_subscribe sub_sent=0 exp=0 want_osd=1
[ 41.660000] libceph: __send_subscribe to 'osdmap' 2
[ 41.660000] libceph: __send_subscribe to 'mdsmap' 5+
[ 41.660000] libceph: ----- 000000009ff1b9c0 to mon0 15=mon_subscribe len 85+0+0 -----
[ 41.660000] libceph: ===== 000000009f9760c0 6 from mon0 4=mon_map len 191+0 (1542097682 0 0) =====
[ 41.660000] libceph: handle_monmap
[ 41.660000] libceph: monmap_decode 000000009fa19a84 000000009fa19b3f len 187
[ 41.660000] libceph: monmap_decode epoch 1, num_mon 1
[ 41.660000] libceph: monmap_decode mon0 is 10.0.1.205:6789
[ 41.660000] libceph: ===== 000000009ff1b300 7 from mon0 16=mon_subscribe_ack len 20+0 (3806796851 0 0) =====
[ 41.660000] libceph: handle_subscribe_ack after 300 seconds
[ 43.860000] libceph: __unregister_request 00000000a053d400 tid 1
[ 43.860000] libceph: no requests, canceling timeout
[ 43.860000] libceph: wait_request tid 1 canceled/timed out
[ 43.860000] libceph: readpages result -512
[ 43.860000] ceph: aio_read 00000000a036e328 10000000004.fffffffffffffffe dropping cap refs on Fcr = -512
...and in fact it can't write it either, for the same reason. It's just that writes are buffered so you don't see the problem if you don't try to sync or umount. sigh
- Status changed from In Progress to Resolved
Fixed by commit:9735451e4b1ec197545de5daf222d518cbf7369c
This did turn out to be an MDS/cephfs bug, actually -- it was filling in the pg_preferred field improperly.
Also available in: Atom
PDF