Ceph » CephFS

Can you dump the ops in flight on both the MDS and the client issuing the snap rmdir when this happens? And the perfcounters on the MDS?

My blind guess about what's blocking this is actually not snapshot trimming, but if the queue for deleting inodes (or one of the directory fragments, as you're on Jewel) is at its max size.

Here is data collected from a recent attempt to delete a very old and very large snapshot:

The snapshot extended attributes looks like:

1. file: cephfs/.snap/snapshot.2017-02-24_22_17_01-1487992621
   ceph.dir.entries="3"
   ceph.dir.files="0"
   ceph.dir.rbytes="30500769204664"
   ceph.dir.rctime="1504695439.09966088000"
   ceph.dir.rentries="7802785"
   ceph.dir.rfiles="7758691"
   ceph.dir.rsubdirs="44094"
   ceph.dir.subdirs="3"

ops in flight during the deletion looks like: {
"ops": [],
"num_ops": 0
}

The problem is that it takes almost 24 hours to delete a single snapshot and it puts the cluster into a warning state whenever it is happening.

Is there a quicker "backdoor" way to purge our snapshots without blowing up the cluster? We really want to clean it up and get it back to a more usable state. At the current rate, it will literally take almost 13 YEARS to clean up the snapshots. Our only other alternative at this point is to destroy the entire filesystem and re-create it and then restore all of the data that was on it (we already backed it up, which took over a week).

Here is a dump of the cephfs 'dentry_lru' table, in case it is interesting.

Zheng, is this issue resolved with the snapshot changes for Mimic?