Ceph » CephFS

Its most likely all the same bug, but fsx fails in different ways each time (always because of a truncate down). The one I'm looking at does the following:

truncate down from 104576 -> 75711

Then it tries to verify the size with a getattr, but getattr returns 104576. The kernel log shows that:

• The setattr successfully sets the size:

Dec 31 10:09:58 plana23 kernel: [1218326.551021] ceph: setattr ffff88020ea01c60 size 104576 -> 75711

• Then on the trace reply:

Dec 31 10:10:07 plana23 kernel: [1218401.360063] ceph: size 104576 -> 75711

• But then during the getattr, the CAP_TRUNC message arrives:

Dec 31 10:10:12 plana23 kernel: [1218406.195224] ceph: handle_cap_trunc inode ffff88020ea01c60 mds0 seq 154 to 75711 seq 36
Dec 31 10:10:12 plana23 kernel: [1218406.254293] ceph: size 75711 -> 104576

Note that second line is setting the i_size back to 104576, because the CAP_TRUNC message has a truncate_size=75711, but size=104576.

So the problem seems like its coming from the mds. But on inspecting the mds, the Locker::issue_truncate() is called after pop_and_dirty_projected_inode (which should correctly set the inode size to the value sent in the setattr). So its still unclear how the mds is sending back size=104576...

Attached is the client log.

The race here is between a truncate down, and completion of osd write ops triggering a cap flush. The exact order that triggers it is:

a) write increases size, updates inode->i_size to 104576
b) setattr to mds for truncate down size 75711
c) write to osds complete, cap flushed, cap update sends size 104576 to mds
d) setattr response, inode->i_size set to 75711
e) cap truncate message received with size 104576, sets inode->i_size to 104576

Pushed a proposed fix to wip-3681, which is currently being tested. This resolves the issue that the setattr is sending a request to the mds if it has the exclusive cap, and ensures that i_size has a correct value, but the same race still exists if the client doesn't have the exclusive cap. Opening a separate bug/issue for the multi-client case.

Should review entire kernel locking around truncate.

added fsx back into the kcephfs test suite. reportedly fsx now passes, but we should verify before closing this bug.