Ceph » CephFS

In Server::handle_client_reconnect, a inode numbers that had client caps but were not in cache are passed into MDCache::rejoin_recovered_caps. This puts them into MDCache::cap_imports. Later, during rejoin, in MDCache::process_imported_caps, cap_imports is iterated over and every item generates a call to MDCache::open_ino (i.e. a getxattr to the data pool to read the backtrace).

This is massively inefficient because (in almost any real workload) many of the files being resolved are in fact in the same directory as one another, and whichever one is fetched first will fetch the whole dirfrag, rendering the backtrace lookups for all the other files in that fragment redundant.

In practice, this is causing a user to experience 15-minute long rejoin phases on a system with ~5m files with capabilities:
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015959.html ("[ceph-users] MDS flapping: how to increase MDS timeouts?")

One simple solution would be to throttle the number of calls to open_ino from process_imported_caps to some configurable threshold (e.g. 1000 by default), so that by the time they came to try resolving the next batch of inodes, the dirfrags from the first batch would already be in cache and the open_inos would be no-ops (need to check that open_ino really handles the no-op case efficiently).

It might even make sense to do that throttling for open_ino in general, as if a workload was hitting lots of hard links at the same time, it's possible that they would hit a similar case where they generated far more getxattrs than needed.