Project

General

Profile

Actions
Copy link

Bug #2494

closed

mds: Cannot remove directory despite it being empty.

Added by Shawn Bohrer almost 12 years ago. Updated almost 7 years ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Community (user)
Tags:
Backport:
Regression:
No
Severity:
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Running ceph 0.47.1 on top of XFS I've got at least two directories which used to contain files but are now empty and cannot be removed. For example where /c is my ceph mount.

-bash-3.2$ rmdir /c/users/sbohrer/fio/o5007
rmdir: /c/users/sbohrer/fio/o5007: Directory not empty
-bash-3.2$ ls -al /c/users/sbohrer/fio/o5007/
total 1
drwxr-xr-x 1 sbohrer hbi 18446744073276227584 May 30 16:12 .
drwxr-xr-x 1 sbohrer hbi          30942429184 May 30 16:09 ..

Looking at mon.a log I see numerous messages similar to:

2012-05-30 12:18:15.069905 mds.0 192.168.50.194:6800/26686 1 : [ERR] loaded dup inode 1000001aa48 [2,head] v44262 at /users/sbohrer/fio/o5007/writer.1.106, but inode 1000001aa48.head v1630929 already exists at ~mds0/stray8/1000001aa48
2012-05-30 12:18:15.069947 mds.0 192.168.50.194:6800/26686 2 : [ERR] loaded dup inode 1000001aadf [2,head] v44408 at /users/sbohrer/fio/o5007/writer.1.112, but inode 1000001aadf.head v1631159 already exists at ~mds0/stray8/1000001aadf
...
2012-05-30 12:22:26.206209 mds.4109 192.168.50.195:6800/27459 1 : [WRN]  replayed op client.4104:70041,70040 used ino 10000035751 but session next is 1000000000d
2012-05-30 12:22:26.206277 mds.4109 192.168.50.195:6800/27459 2 : [WRN]  replayed op client.4104:70042,70040 used ino 1000003576f but session next is 1000000000d

See the attached mon.a.log for all of the ERR/WRN messages. Additionally /var/log/ceph/ceph-mds.a.log is enormous (54205911 lines!) but I've attached a massively trimmed down version that I think has relevent messages that start at the time of the problem. See the attached ceph-mds.a.log.gz It appears that the majority of the log is similar lines repeated over and over.

Files

Download all files
mon.a.log (126 KB) mon.a.log Shawn Bohrer, 05/31/2012 02:40 PM
ceph-mds.a.log.gz (6.52 MB) ceph-mds.a.log.gz Shawn Bohrer, 05/31/2012 02:40 PM
Actions
Copy link

Also available in: Atom PDF