Ceph » CephFS

I removed the assertion: assert(in->is_head());

That allowed the mds servers to restart and complete recovery, and from initial tests the filesystem seems fine. If I add the assertion back into the code and restart the mds, I don't hit that assertion any more, so I can't give you the debug output.

I did try to debug this a little bit. The is_head() function verifies that the inode isn't a snapshot, but in this case the value of last.val is 16 (not -1,-2, or -3).

Based on the warning about "bad client_range", it looks like the errors here are all related to client leases being invalid. In that case, it doesn't seem like that assertion is necessary (maybe just a warning). If the mds just needs to be able to recover, the client leases could all just be revoked, couldn't they?

As for the original error, it does seem reproducible by creating a snapshot of a directory using the mkdir system call (from a C program) with 0777 as the mask, and then moving another directory into the directory that had just been snapshotted.

Hmmm. If last.val isn't -1 then it's not a head inode.

With the reproduction steps (we love reproduceable bugs!) and the backtrace, I bet it's because the moved directory is getting pulled into the middle of a snapshot-in-progress and not handling it properly.

replay crash looks like the one fixed in commit:8c5e7dcf8cf7f3daa65eb9905, yay!