Project

General

Profile

Bug #10875

recovery may leave files inaccessible for a very long time

Added by Alexandre Oliva about 9 years ago. Updated about 9 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

A very sparse file of length slightly larger than 1GB had got a few scattered writes when the mds restarted.

Recovery decided to scan all 512 objects from 0 to 2GB.

This takes a very long time on my cluster. Each object stat is taking a few seconds, presumably because of the ongoing migration of data to an EC pool.

The only information the mds logs is the delayed attempts to obtain rdlocks when I access the file.

If we probed multiple objects in parallel, I think it would go much faster, but it's statting only one object at a time, going backwards. Starting the search so far away from the actual size surely doesn't help either.

Regardless, recovery might still take a long time in particularly pathological cases, so it would be nice if the mds would log long-running probe aggregate operations, just as it logs delayed client requests. This would at least give users a clue on what is going on when accessing a file takes a very, very long time.

Of course, if ceph already did that, I probably wouldn't have bothered looking into what was getting the stat stuck, never found out about file probing for size and mtime recovery, and never realized that this might have been the cause of bug 10874 ;-)

History

#1 Updated by Greg Farnum about 9 years ago

  • Status changed from New to Duplicate

Also available in: Atom PDF