creating a new fs on pools from an old fs can lead to lost MDS Tables
See the thread at http://firstname.lastname@example.org/msg03918.html
If you delete your FS (but not the data pools), then create a new FS on the same pools, the MDS client IDs are repeated. This means that if you haven't started recycling ops in the PG log, the OSDs will incorrectly return success on writes without doing anything. This can lead to eg not having a snaptable on disk, because the MDS tries to write out the initial one and the OSD says "I already have that tid, and the op succeeded" and short-circuits.
The best solution I've come up with so far is to somehow add an FS generation number to the MDS entity/invocation stuff, but I don't like having to maintain or steal that number in the Monitors...