Project

General

Profile

Bug #9178

samba: ENOTEMPTY on "rm -rf"

Added by Greg Farnum about 5 years ago. Updated almost 5 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
Start date:
08/20/2014
Due date:
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:

Description

2014-08-18T01:23:14.854 INFO:teuthology.orchestra.run.plana83:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp'
2014-08-18T01:23:19.626 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p0/d1/d12/d57/d65': Directory not empty
2014-08-18T01:23:22.085 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p0/d1/d8c/db0/de0/d11b': Directory not empty
2014-08-18T01:23:25.481 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p2/dc/d45': Directory not empty
2014-08-18T01:23:30.356 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p3/da/d2e/d3a': Directory not empty
2014-08-18T01:23:37.616 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p4/d0/dc/d11/d94/d101': Directory not empty
2014-08-18T01:23:38.884 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p5/d7/d37/d28': Directory not empty
2014-08-18T01:23:43.076 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p5/d7/da0/d11c': Directory not empty
2014-08-18T01:23:51.992 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p7/d1/dc': Directory not empty
2014-08-18T01:23:53.705 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p8/d0/d44': Directory not empty
2014-08-18T01:23:54.804 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p8/d0/dd/d15/d35/d61/d8b/d102/d4c/d50/df2': Directory not empty
2014-08-18T01:24:04.190 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1...
2014-08-18T01:24:04.190 INFO:teuthology.orchestra.run.plana83:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-08-18T01:24:04.207 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/workunit.py", line 363, in _run_tests
    args=['sudo', 'rm', '-rf', '--', scratch_tmp],
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana83 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp' 

http://qa-proxy.ceph.com/teuthology/teuthology-2014-08-17_23:14:01-samba-next-testing-basic-multi/431541/

So far this is new and just on Samba; we'll see if it pops up elsewhere.

Associated revisions

Revision 600af254 (diff)
Added by Zheng Yan almost 5 years ago

client: introduce a new flag indicating if dentries in directory are sorted

When creating a file, Client::insert_dentry_inode() set the dentry's offset
based on directory's max offset. The offset does not reflect the real
postion of the dentry in directory. Later readdir reply from real postion
of the dentry in directory. Later readdir reply from MDS may change the
dentry's position/offset. This inconsistency can cause missing/duplicate
entries in readdir result if readdir is partly satisfied by dcache_readdir().

The fix is introduce a new flag indicating if dentries in directory are
sorted. We use _readdir_cache_cb() to handle readdir only when the flag is
set, clear the flag after creating/deleting/renaming file.

Fixes: #9178
Signed-off-by: Yan, Zheng <>

Revision 0671c112 (diff)
Added by Zheng Yan almost 5 years ago

client: introduce a new flag indicating if dentries in directory are sorted

When creating a file, Client::insert_dentry_inode() set the dentry's offset
based on directory's max offset. The offset does not reflect the real
postion of the dentry in directory. Later readdir reply from real postion
of the dentry in directory. Later readdir reply from MDS may change the
dentry's position/offset. This inconsistency can cause missing/duplicate
entries in readdir result if readdir is partly satisfied by dcache_readdir().

The fix is introduce a new flag indicating if dentries in directory are
sorted. We use _readdir_cache_cb() to handle readdir only when the flag is
set, clear the flag after creating/deleting/renaming file.

Fixes: #9178
Signed-off-by: Yan, Zheng <>
(cherry picked from commit 600af25493947871c38214aa370e2544a7fea399)

History

#2 Updated by Zheng Yan almost 5 years ago

The strange thing is that the MDS never reply ENOTEMPTY.

#3 Updated by Greg Farnum almost 5 years ago

It might not need to if there's a client bug somewhere. (Or some other issue?)

#5 Updated by Zheng Yan almost 5 years ago

  • Status changed from New to Testing
2014-09-08 02:25:36.393142 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 0 frag * fragpos 0 at_end=0
2014-09-08 02:25:36.393156 7fe28a9d2740 15 client.4119  including .
2014-09-08 02:25:36.393158 7fe28a9d2740 10 client.4119 fill_dirent '.' -> 1000000036a type 4 w/ next_off 1
2014-09-08 02:25:36.393160 7fe28a9d2740 10 client.4119 fill_stat on 1000000036a snap/devhead mode 040755 mtime 2014-09-08 02:24:24.131352 ctime 2014-09-08 02:24:24.131352
2014-09-08 02:25:36.393167 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 1 frag * fragpos 1 at_end=0
2014-09-08 02:25:36.393178 7fe28a9d2740 15 client.4119  including ..
2014-09-08 02:25:36.393180 7fe28a9d2740 10 client.4119 fill_dirent '..' -> 1000000036a type 4 w/ next_off 2
2014-09-08 02:25:36.393182 7fe28a9d2740 10 client.4119 fill_stat on 1000000036a snap/devhead mode 040755 mtime 2014-09-08 02:24:24.131352 ctime 2014-09-08 02:24:24.131352
2014-09-08 02:25:36.393187 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 2 frag * fragpos 2 at_end=0
2014-09-08 02:25:36.393198 7fe28a9d2740 10 client.4119 offset 2 at_cache_name  snapid head complete 1 issued pAsLsXsFsx
2014-09-08 02:25:36.393204 7fe28a9d2740 10 client.4119 _readdir_cache_cb 0x7fe28cc55cc0 on 1000000036a at_cache_name  offset 2
2014-09-08 02:25:36.393206 7fe28a9d2740 15 client.4119  skipping null 'd87'
2014-09-08 02:25:36.393208 7fe28a9d2740 10 client.4119 fill_stat on 1000000044d snap/devhead mode 040755 mtime 2014-09-08 02:25:29.206007 ctime 2014-09-08 02:25:29.206007
2014-09-08 02:25:36.393214 7fe28a9d2740 10 client.4119 fill_dirent 'da9' -> 1000000044d type 4 w/ next_off 3
2014-09-08 02:25:36.393216 7fe28a9d2740 15 client.4119  de da9 off 5 = 1
2014-09-08 02:25:36.393219 7fe28a9d2740  3 client.4119 telldir(0x7fe28cc55cc0) = 6

"de da9 off 5 = 1" is wrong, off should be 2

https://github.com/ceph/ceph/commit/2249179a8d6a6feec5992a1b9bb71f49fe910a5c

#6 Updated by Sage Weil almost 5 years ago

  • Priority changed from Normal to Urgent

#7 Updated by Sage Weil almost 5 years ago

  • Priority changed from Urgent to High

#8 Updated by Greg Farnum almost 5 years ago

/a/teuthology-2014-09-08_23:14:02-samba-master-testing-basic-multi/474551/

#9 Updated by Zheng Yan almost 5 years ago

the fix https://github.com/ceph/ceph/pull/2431 hasn't been merged yet

#10 Updated by Zheng Yan almost 5 years ago

  • Status changed from Testing to Resolved

Also available in: Atom PDF