NFS reexport file creation lags 1-3 seconds
I'm attaching the kernel logs and mds logs for the creation of a file called scale-product/testfoo3 via a touch running on a remote NFS client.
I've looked through the logs, but they're a little difficult to track.
There are a few other file accesses in these logs for other services we're using.
However, I think it unlikely that those are causing problems, as I can create files locally on the ceph kernel mount without experiencing this lag.
#3 Updated by Sage Weil over 9 years ago
It looks like after the mknod nfsd is calling write_inode via commit_metadata() in fs/nfsd/vfs.c. This triggers cap writeback, which does not flush teh mds cache immediately because normally nobody (except maybe sync(2)) waits on it.
If you export with option async and it should skip this step and be fast.
We can also defined a ->commit_metadata hook in the export_operations that does something different/smarter. What this is, I'm not sure. In this specific case, mknod is setting the mtime/atime/ctime, which is pretty useless (the mds did that too). In other cases safety might matter more.
Wanna try the async option just to verify this is in fact what's going on?
#12 Updated by Brian Chrisman about 9 years ago
I've reproduced this half-second-NFS-create bug (f3de1a506d6c2debb399a1ae71f8f50714a31c8a), and it only appears through NFS (and occurs with async export and mount options set).
With local ceph mount, I get < 0.02s file creates, so there's something mucked up with the nfs side of this.
I'm going to attach mds and messages logs... (and move on to testing for stale fhs).
Here I'm creating a sequence of files, foo42, foo43 ... there's are being created manually (not being run as soon as last 'touch' returns) via nfs.
(Note, I previously exported non-ceph filesystems from the same server/node without any significant file create lag)
[root@buildrhel6 ~]# time touch /root/testmnt/foo42
[root@buildrhel6 ~]# time touch /root/testmnt/foo43
[root@buildrhel6 ~]# time touch /root/testmnt/foo44