Ceph » CephFS

I'd be happy enough if the mds wouldn't mess with timestamps of files of clients that are still up, and sync would make sure any files that are not modified afterwards keep their synced timestamps. I suppose this is already the case of umount, but I haven't checked. Meaning it's sort of ok if mds has to go through file probing if the client modifying the file is gone before the metadata can make the new mds.

Now, it would be nice if data writes and metadata updates could somehow be made transactional/atomic, in both-or-none and serializable-versions senses, so that an mds could tell whether metadata updates related with or subsequent to data writes, so that it doesn't need to probe or, if it is probing, it can tell the mtime update for the latest write it finds is already reflected in metadata, and so that a write whose corresponding metadata update didn't reach stable storage before the mds dies does not get unnoticed by the subsequent mds.

Since we don't have osd&mds transactions AFAIK, one thought that occurred to me was to use osd transactions to write the data and an xattr in a single transaction, and have the recovering mds look for the xattr to know about pending metadata updates. The client would remove the xattr once the mds confirmed the metadata update hit the mds journal; if the client is gone, the recovering mds would do that.

Transactions like that would be much more expensive than e.g. adding backtraces to file data objects, which I seem to remember you complaining bitterly about. ;) We're not going to do that.

What if writes carried the timestamp that ctime and mtime in the object are about to be set to, and the osd set the object's mtime to that timestamp, and mds recovery compared the timestamps of the osd objects with the ctime of the inode to decide whether there have been changes after the latest mds metadata commit for the file? I think this would preserve all intended properties: writes after commit will update mtime and ctime, whereas mtime warps after writes won't get clobbered because their ctime would necessarily be newer than any earlier writes, even if the writes only make to osds at a later time.

Now, maybe it would be more appropriate for mtime and size updates to be sent to the mds only after the osds ack a write. I had a ceph.ko client failure part-way through an rsync of a large tree, and later found out that many small files had the right size, but none of the data.

Now, I realize this wouldn't cover the case of the client crashing between the write and the metadata update, so this wouldn't be a complete solution. Now, if logging on the object might bring a solution to this, say by writing an mtime update to such a log before the write, then we might as well issue the mtime update to the mds, when we request write caps on the file.