Project

General

Profile

Actions
Copy link

Bug #9178

closed

samba: ENOTEMPTY on "rm -rf"

Added by Greg Farnum over 9 years ago. Updated over 9 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2014-08-18T01:23:14.854 INFO:teuthology.orchestra.run.plana83:Running: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp'
2014-08-18T01:23:19.626 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p0/d1/d12/d57/d65': Directory not empty
2014-08-18T01:23:22.085 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p0/d1/d8c/db0/de0/d11b': Directory not empty
2014-08-18T01:23:25.481 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p2/dc/d45': Directory not empty
2014-08-18T01:23:30.356 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p3/da/d2e/d3a': Directory not empty
2014-08-18T01:23:37.616 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p4/d0/dc/d11/d94/d101': Directory not empty
2014-08-18T01:23:38.884 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p5/d7/d37/d28': Directory not empty
2014-08-18T01:23:43.076 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p5/d7/da0/d11c': Directory not empty
2014-08-18T01:23:51.992 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p7/d1/dc': Directory not empty
2014-08-18T01:23:53.705 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p8/d0/d44': Directory not empty
2014-08-18T01:23:54.804 INFO:tasks.workunit.client.1.plana83.stderr:rm: cannot remove `/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress-plana835050/p8/d0/dd/d15/d35/d61/d8b/d102/d4c/d50/df2': Directory not empty
2014-08-18T01:24:04.190 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1...
2014-08-18T01:24:04.190 INFO:teuthology.orchestra.run.plana83:Running: 'rm -rf -- /home/ubuntu/cephtest/workunits.list /home/ubuntu/cephtest/workunit.client.1'
2014-08-18T01:24:04.207 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/var/lib/teuthworker/src/ceph-qa-suite_next/tasks/workunit.py", line 363, in _run_tests
    args=['sudo', 'rm', '-rf', '--', scratch_tmp],
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 114, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 401, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 102, in wait
    exitstatus=status, node=self.hostname)
CommandFailedError: Command failed on plana83 with status 1: 'sudo rm -rf -- /home/ubuntu/cephtest/mnt.1/client.1/tmp'

http://qa-proxy.ceph.com/teuthology/teuthology-2014-08-17_23:14:01-samba-next-testing-basic-multi/431541/

So far this is new and just on Samba; we'll see if it pops up elsewhere.

Actions
Copy link
 #2

Updated by Zheng Yan over 9 years ago

The strange thing is that the MDS never reply ENOTEMPTY.

Actions
Copy link
 #3

Updated by Greg Farnum over 9 years ago

It might not need to if there's a client bug somewhere. (Or some other issue?)

Actions
Copy link
 #5

Updated by Zheng Yan over 9 years ago

  • Status changed from New to 7
2014-09-08 02:25:36.393142 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 0 frag * fragpos 0 at_end=0
2014-09-08 02:25:36.393156 7fe28a9d2740 15 client.4119  including .
2014-09-08 02:25:36.393158 7fe28a9d2740 10 client.4119 fill_dirent '.' -> 1000000036a type 4 w/ next_off 1
2014-09-08 02:25:36.393160 7fe28a9d2740 10 client.4119 fill_stat on 1000000036a snap/devhead mode 040755 mtime 2014-09-08 02:24:24.131352 ctime 2014-09-08 02:24:24.131352
2014-09-08 02:25:36.393167 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 1 frag * fragpos 1 at_end=0
2014-09-08 02:25:36.393178 7fe28a9d2740 15 client.4119  including ..
2014-09-08 02:25:36.393180 7fe28a9d2740 10 client.4119 fill_dirent '..' -> 1000000036a type 4 w/ next_off 2
2014-09-08 02:25:36.393182 7fe28a9d2740 10 client.4119 fill_stat on 1000000036a snap/devhead mode 040755 mtime 2014-09-08 02:24:24.131352 ctime 2014-09-08 02:24:24.131352
2014-09-08 02:25:36.393187 7fe28a9d2740 10 client.4119 readdir_r_cb 1000000036a.head(ref=3 cap_refs={} open={} mode=40755 size=0/0 mtime=2014-09-08 02:24:24.131352 caps=pAsLsXsFsx(0=pAsLsXsFsx) COMPLETE parents=0x7fe28cc00220 0x7fe25818c510) offset 2 frag * fragpos 2 at_end=0
2014-09-08 02:25:36.393198 7fe28a9d2740 10 client.4119 offset 2 at_cache_name  snapid head complete 1 issued pAsLsXsFsx
2014-09-08 02:25:36.393204 7fe28a9d2740 10 client.4119 _readdir_cache_cb 0x7fe28cc55cc0 on 1000000036a at_cache_name  offset 2
2014-09-08 02:25:36.393206 7fe28a9d2740 15 client.4119  skipping null 'd87'
2014-09-08 02:25:36.393208 7fe28a9d2740 10 client.4119 fill_stat on 1000000044d snap/devhead mode 040755 mtime 2014-09-08 02:25:29.206007 ctime 2014-09-08 02:25:29.206007
2014-09-08 02:25:36.393214 7fe28a9d2740 10 client.4119 fill_dirent 'da9' -> 1000000044d type 4 w/ next_off 3
2014-09-08 02:25:36.393216 7fe28a9d2740 15 client.4119  de da9 off 5 = 1
2014-09-08 02:25:36.393219 7fe28a9d2740  3 client.4119 telldir(0x7fe28cc55cc0) = 6

"de da9 off 5 = 1" is wrong, off should be 2

https://github.com/ceph/ceph/commit/2249179a8d6a6feec5992a1b9bb71f49fe910a5c

Actions
Copy link
 #6

Updated by Sage Weil over 9 years ago

  • Priority changed from Normal to Urgent
Actions
Copy link
 #7

Updated by Sage Weil over 9 years ago

  • Priority changed from Urgent to High
Actions
Copy link
 #8

Updated by Greg Farnum over 9 years ago

/a/teuthology-2014-09-08_23:14:02-samba-master-testing-basic-multi/474551/

Actions
Copy link
 #9

Updated by Zheng Yan over 9 years ago

the fix https://github.com/ceph/ceph/pull/2431 hasn't been merged yet

Actions
Copy link
 #10

Updated by Zheng Yan over 9 years ago

  • Status changed from 7 to Resolved
Actions
Copy link

Also available in: Atom PDF