Ceph » CephFS

See also #4485.

From #10875:

A very sparse file of length slightly larger than 1GB had got a few scattered writes when the mds restarted.

Recovery decided to scan all 512 objects from 0 to 2GB.

This takes a very long time on my cluster. Each object stat is taking a few seconds, presumably because of the ongoing migration of data to an EC pool.

The only information the mds logs is the delayed attempts to obtain rdlocks when I access the file.

If we probed multiple objects in parallel, I think it would go much faster, but it's statting only one object at a time, going backwards. Starting the search so far away from the actual size surely doesn't help either.

Regardless, recovery might still take a long time in particularly pathological cases, so it would be nice if the mds would log long-running probe aggregate operations, just as it logs delayed client requests. This would at least give users a clue on what is going on when accessing a file takes a very, very long time.

So: parallel object checks. More visibility about ongoing recovery operations. Unfortunately going backwards from max_size is necessary, since we need to demonstrate that we know the last object. :(