Project

General

Profile

Actions
Copy link

Bug #1108

closed

Large number of files in a directory makes things grind to a halt

Added by Damien Churchill almost 13 years ago. Updated over 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Tags:
Backport:
Regression:
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Whilst extracting a copy of our mail directories onto a 10 node cluster(3xmds, 3xmon, 10xosd) I found that there was one person who had 2.5 million files in their Trash folder. I left this extracting over the weekend and when I returned to it on Monday it had ground to a halt, extracting perhaps 1 message per second, it had only extracted 330,000 files at this point. I managed to find the count using:

python -c "import os; print len(os.listdir('/path/to/folder'))"

Using ls or find just took too long (presumably it was fetching file metadata as well), but even that python statement took a long time (approx 1hr). I have no idea if this is just a symptom of being a distributed file system or if there is anyway to speed this up however.

Actions
Copy link

Also available in: Atom PDF