Ceph » CephFS

besides, I'm curious when was the fs created (which version)

FS was created on 0.72.2 then upgraded to 0.78, 0.79 following by 0.80~rc1.
Somehow journal was corrupted during cluster recovery; MDS was crashing on journal replay;
Unfortunately I lost crash dump because of the mentioned log flood.

I had to resort to "--reset-journal" to get access to files.
Some files are corrupted (that's not a problem) but now I'm getting errors like

2014-05-01 03:37:22.059908 mds.0 [ERR] dir 10000421bec object missing on disk; some files may be lost
2014-05-01 03:53:58.440638 mds.0 [ERR] dir 10000421646 object missing on disk; some files may be lost

on MDS start.

I moved "epiphany" directory out of the way and rebooted client(s) that were accessing it.
Now I still can't remove it but there is nothing in MDS log whatsoever. Could it be that one directory entry stuck but MDS do not show it due to above error?

I wonder how can I wipe out affected directories (together with directory fragments)?

get inode number of 'epiphany' directory, then modify Server::_dir_is_nonempty_unlocked() and Server::_dir_is_nonempty() in src/mds/Server.cc, add line
"if (in->ino() == <inode number>) return false;" to the beginning of these two functions.

Thanks, I might try that or make a new file system from scratch.

There are more than one issue mentioned in this ticket but as for TODO I think we should rate-limit logging to prevent flood of similar messages. Perhaps logger can remember last message and just increment the counter if the same message is repeated. Then it can print something like "previous message repeated NNN times." every 10 or 30 seconds.