Project

General

Profile

Actions
Copy link

Bug #5927

closed

kcephfs: ENOTEMPTY on rm -r

Added by Sage Weil over 10 years ago. Updated almost 8 years ago.

Status:
Resolved
Priority:
High
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
kceph
Labels (FS):
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

2013-08-09T05:57:47.722 INFO:teuthology.task.workunit:Running workunit misc/multiple_rsync.sh...
...
2013-08-09T06:51:02.059 DEBUG:teuthology.orchestra.run:Running [10.214.132.13]: 'sudo rm -rf -- /home/ubuntu/cephtest/101297/mnt.0/client.0/tmp'
2013-08-09T06:51:42.701 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/include/c++/4.6/ext': Directory not empty
2013-08-09T06:52:47.454 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/lib/debug/usr': Directory not empty
2013-08-09T06:53:08.238 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/lib/jvm/java-6-openjdk-common/jre': Directory not empty
2013-08-09T06:55:01.048 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/lib/perl5/auto': Directory not empty
2013-08-09T07:11:26.212 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/man/sv': Directory not empty
2013-08-09T07:11:38.037 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/junit/junit': Directory not empty
2013-08-09T07:11:38.081 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/org/apache/ant/ant': Directory not empty
2013-08-09T07:12:00.509 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/org/hamcrest/hamcrest-core': Directory not empty
2013-08-09T07:12:02.852 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/xerces/xercesImpl': Directory not empty
2013-08-09T07:12:03.331 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/xml-apis/xml-apis-1.4': Directory not empty
2013-08-09T07:12:04.543 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/maven-repo/xml-resolver/xml-resolver': Directory not empty
2013-08-09T07:15:57.775 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/zoneinfo/posix': Directory not empty
2013-08-09T07:16:15.539 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/share/zoneinfo/right': Directory not empty
2013-08-09T07:20:28.507 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/arch/powerpc/platforms': Directory not empty
2013-08-09T07:20:49.804 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/arch/sh/include': Directory not empty
2013-08-09T07:21:00.099 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/arch/tile/include': Directory not empty
2013-08-09T07:21:59.428 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/drivers/media/dvb': Directory not empty
2013-08-09T07:22:03.056 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/drivers/media/video': Directory not empty
2013-08-09T07:22:19.328 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/drivers/net/ethernet': Directory not empty
2013-08-09T07:22:38.286 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/drivers/scsi/aic7xxx': Directory not empty
2013-08-09T07:23:27.672 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/fs': Directory not empty
2013-08-09T07:24:21.663 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51/include': Directory not empty
2013-08-09T07:25:40.468 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51-generic/arch': Directory not empty
2013-08-09T07:32:45.593 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51-generic/include': Directory not empty
2013-08-09T07:32:55.596 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.1/src/linux-headers-3.2.0-51-generic/scripts': Directory not empty
2013-08-09T07:33:46.072 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/include/c++/4.6': Directory not empty
2013-08-09T07:34:45.164 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/gcc/x86_64-linux-gnu/4.6': Directory not empty
2013-08-09T07:34:56.304 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/jvm/java-6-openjdk-amd64/include': Directory not empty
2013-08-09T07:35:02.723 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/jvm/java-6-openjdk-amd64/jre/lib/amd64': Directory not empty
2013-08-09T07:35:11.297 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/jvm/java-6-openjdk-amd64/man': Directory not empty
2013-08-09T07:35:15.303 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/jvm/java-7-openjdk-amd64': Directory not empty
2013-08-09T07:37:11.423 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/pymodules/python2.7/matplotlib': Directory not empty
2013-08-09T07:37:14.355 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/pymodules/python2.7/mpl_toolkits': Directory not empty
2013-08-09T07:37:24.816 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/config': Directory not empty
2013-08-09T07:37:30.047 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/OpenSSL/test': Directory not empty
2013-08-09T07:37:30.467 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/UpdateManager/Core': Directory not empty
2013-08-09T07:37:31.092 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/apt/progress': Directory not empty
2013-08-09T07:37:36.361 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/dateutil/zoneinfo': Directory not empty
2013-08-09T07:37:37.024 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/dbus/mainloop': Directory not empty
2013-08-09T07:37:38.916 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/flask/ext': Directory not empty
2013-08-09T07:37:48.670 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/flask/testsuite/test_apps/flaskext/oldext_package': Directory not empty
2013-08-09T07:37:51.367 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/flask/testsuite/test_apps/moduleapp/apps': Directory not empty
2013-08-09T07:37:53.925 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/gi/_glib': Directory not empty
2013-08-09T07:37:53.963 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/gi/_gobject': Directory not empty
2013-08-09T07:37:54.069 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/gi/overrides': Directory not empty
2013-08-09T07:37:54.097 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/gi/repository': Directory not empty
2013-08-09T07:37:58.203 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/jinja2/testsuite/res': Directory not empty
2013-08-09T07:37:59.604 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/nose/ext': Directory not empty
2013-08-09T07:37:59.818 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/nose/plugins': Directory not empty
2013-08-09T07:37:59.886 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/nose/sphinx': Directory not empty
2013-08-09T07:38:01.174 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/compat': Directory not empty
2013-08-09T07:38:04.181 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/core': Directory not empty
2013-08-09T07:38:07.455 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/distutils/tests/f2py_ext': Directory not empty
2013-08-09T07:38:08.426 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/distutils/tests/f2py_f90_ext': Directory not empty
2013-08-09T07:38:09.835 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/distutils/tests/gen_ext': Directory not empty
2013-08-09T07:38:10.431 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/distutils/tests/pyrex_ext': Directory not empty
2013-08-09T07:38:11.210 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/distutils/tests/swig_ext': Directory not empty
2013-08-09T07:38:11.483 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/f2py': Directory not empty
2013-08-09T07:38:12.275 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/fft': Directory not empty
2013-08-09T07:38:14.041 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/lib': Directory not empty
2013-08-09T07:38:14.609 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/linalg': Directory not empty
2013-08-09T07:38:15.633 INFO:teuthology.task.workunit.client.0.err:[10.214.132.13]: rm: cannot remove `/home/ubuntu/cephtest/101297/mnt.0/client.0/tmp/usr.2/lib/python2.7/dist-packages/numpy/ma': Directory not empty

job was 
ubuntu@teuthology:/a/teuthology-2013-08-09_01:01:44-kcephfs-next-testing-basic-plana/101297$ cat orig.config.yaml 
kernel:
  kdb: true
  sha1: 68d04c9a036bea12d76fdd51e926aef771e8e041
machine_type: plana
nuke-on-error: true
os_type: ubuntu
overrides:
  admin_socket:
    branch: next
  ceph:
    conf:
      mon:
        debug mon: 20
        debug ms: 1
        debug paxos: 20
      osd:
        osd op thread timeout: 60
    fs: btrfs
    log-whitelist:
    - slow request
    sha1: 0aacd10e2557c55021b5be72ddf39b9cea916be4
  ceph-deploy:
    branch:
      dev: next
    conf:
      client:
        log file: /var/log/ceph/ceph-$name.$pid.log
      mon:
        debug mon: 1
        debug ms: 20
        debug paxos: 20
  install:
    ceph:
      sha1: 0aacd10e2557c55021b5be72ddf39b9cea916be4
  s3tests:
    branch: next
  workunit:
    sha1: 0aacd10e2557c55021b5be72ddf39b9cea916be4
roles:
- - mon.a
  - mon.c
  - osd.0
  - osd.1
  - osd.2
- - mon.b
  - mds.a
  - osd.3
  - osd.4
  - osd.5
- - client.0
tasks:
- chef: null
- clock.check: null
- install: null
- ceph:
    conf:
      mds:
        debug mds: 20
        debug ms: 1
- kclient: null
- workunit:
    clients:
      all:
      - misc
teuthology_branch: next

haven't seen this in a long time!

Actions
Copy link
 #1

Updated by Zheng Yan over 10 years ago

my patch "ceph: rework trim caps code" can explain this issue

Actions
Copy link
 #2

Updated by Sage Weil over 10 years ago

  • Status changed from New to 7
Actions
Copy link
 #3

Updated by Sage Weil over 10 years ago

  • Status changed from 7 to 15
Actions
Copy link
 #4

Updated by Zheng Yan over 10 years ago

  • Status changed from 15 to Resolved
Actions
Copy link
 #5

Updated by Greg Farnum almost 8 years ago

  • Component(FS) kceph added
Actions
Copy link

Also available in: Atom PDF