Project

General

Profile

Actions
Copy link

Bug #4003

closed

rbd: EBUSY errors from rbd unmap

Added by Sam Lang about 11 years ago. Updated about 11 years ago.

Status:
Resolved
Priority:
Urgent
Assignee:
Sage Weil
Target version:
-
% Done:

0%

Source:
Development
Tags:
Backport:
Regression:
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

From the teuthology kernel untar task on rbd, we get EBUSY trying to unmap. I'm investigating that this isn't somehow caused by teuthology changes...

2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_cm.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_marshall.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_sa.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_umem.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_pack.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/Kbuild'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_sa.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_user_cm.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_cm.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/iw_cm.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_verbs.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_cm_ib.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9/include/rdma'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9/include'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/vmlinux.o'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `t'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9.tar.bz2'
2013-02-04T10:40:39.326 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'rm rf - /tmp/cephtest/mnt.0/client.0/tmp'
2013-02-04T10:40:39.336 INFO:teuthology.task.workunit:Stopping kernel_untar_build.sh on client.0...
2013-02-04T10:40:39.337 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'rm rf - /tmp/cephtest/workunits.list /tmp/cephtest/workunit.client.0'
2013-02-04T10:40:39.383 DEBUG:teuthology.parallel:result is None
2013-02-04T10:40:39.383 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x194e110>
2013-02-04T10:40:39.383 INFO:teuthology.task.rbd:Unmounting rbd images...
2013-02-04T10:40:39.384 INFO:teuthology.task.rbd:Unmapping rbd devices...
2013-02-04T10:40:39.384 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'sudo LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/rbd c /tmp/cephtest/ceph.conf -p rbd unmap /dev/rbd/rbd/testimage.client.0 && while test -e /dev/rbd/rbd/testimage.client.0 ; do sleep 1 ; done'
2013-02-04T10:40:39.489 INFO:teuthology.orchestra.run.err:rbd: failed to remove rbd device: (16) Device or resource busy
2013-02-04T10:40:39.489 INFO:teuthology.orchestra.run.err:rbd: remove failed: (16) Device or resource busy
2013-02-04T10:40:39.491 INFO:teuthology.task.rbd:Unloading rbd kernel module...
2013-02-04T10:40:39.491 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'sudo modprobe -r rbd || true'
2013-02-04T10:40:39.506 INFO:teuthology.orchestra.run.err:FATAL: Module rbd is in use.
2013-02-04T10:40:39.508 INFO:teuthology.task.rbd:Deleting rbd images...
2013-02-04T10:40:39.508 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/rbd -c /tmp/cephtest/ceph.conf -p rbd rm testimage.client.0'
2013-02-04T10:40:39.914 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:18.959420 7fcfda6f4700 0 - 10.214.132.32:0/1029143 >> 10.214.132.29:6800/8592 pipe(0x2085340 sd=12 :36056 s=2 pgs=10 cs=1 l=1).injecting socket failure
2013-02-04T10:40:39.966 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:19.011264 7fcfdacfa700 0 -- 10.214.132.32:0/1029143 >> 10.214.132.31:6803/22416 pipe(0x207ddd0 sd=9 :52957 s=2 pgs=18 cs=1 l=1).injecting socket failure
2013-02-04T10:40:48.084 INFO:teuthology.orchestra.run.out:^MRemoving image: 1% complete...^MRemoving image: 2% complete...^MRemoving image: 3% complete...^MRemoving image: 4% complete...^MRemoving image: 5% complete...^MRemoving image: 6% complete...^MRemoving image: 7% complete...^MRemoving image: 8% complete...^MRemoving image: 9% complete...^MRemoving image: 10% complete...^MRemoving image: 11% complete...^MRemoving image: 12% complete...^MRemoving image: 13% complete...^MRemoving image: 14% complete...^MRemoving image: 15% complete...^MRemoving image: 16% complete...^MRemoving image: 17% complete...^MRemoving image: 18% complete...^MRemoving image: 19% complete...^MRemoving image: 20% complete...^MRemoving image: 21% complete...^MRemoving image: 22% complete...^MRemoving image: 23% complete...^MRemoving image: 24% complete...^MRemoving image: 25% complete...^MRemoving image: 26% complete...^MRemoving image: 27% complete...^MRemoving image: 28% complete...^MRemoving image: 29% complete...^MRemoving image: 30% complete...^MRemoving image: 31% complete...^MRemoving image: 32% complete...^MRemoving image: 33% complete...^MRemoving image: 34% complete...^MRemoving image: 35% complete...^MRemoving image: 36% complete...^MRemoving image: 37% complete...^MRemoving image: 38% complete...^MRemoving image: 39% complete...^MRemoving image: 40% complete...^MRemoving image: 41% complete...^MRemoving image: 42% complete...^MRemoving image: 43% complete...^MRemoving image: 44% complete...^MRemoving image: 45% complete...^MRemoving image: 46% complete...^MRemoving image: 47% complete...^MRemoving image: 48% complete...^MRemoving image: 49% complete...^MRemoving image: 50% complete...^MRemoving image: 51% complete...^MRemoving image: 52% complete...^MRemoving image: 53% complete...^MRemoving image: 54% complete...^MRemoving image: 55% complete...^MRemoving image: 56% complete...^MRemoving image: 57% complete...^MRemoving image: 58% complete...^MRemoving image: 59% complete...^MRemoving image: 60% complete...^MRemoving image: 61% complete...^MRemoving image: 62% complete...^MRemoving image: 63% complete...^MRemoving image: 64% complete...^MRemoving image: 65% complete...^MRemoving image: 66% complete...^MRemoving image: 67% complete...^MRemoving image: 68% complete...^MRemoving image: 69% complete...^MRemoving image: 70% complete...^MRemoving image: 71% complete...^MRemoving image: 72% complete...^MRemoving image: 73% complete...^MRemoving image: 74% complete...^MRemoving image: 75% complete...^MRemoving image: 76% complete...^MRemoving image: 77% complete...^MRemoving image: 78% complete...^MRemoving image: 79% complete...^MRemoving image: 80% complete...^MRemoving image: 81% complete...^MRemoving image: 82% complete...^MRemoving image: 83% complete...^MRemoving image: 84% complete...^MRemoving image: 85% complete...^MRemoving image: 86% complete...^MRemoving image: 87% complete...^MRemoving image: 88% complete...^MRemoving image: 89% complete...^MRemoving image: 90% complete...^MRemoving image: 91% complete...^MRemoving image: 92% complete...^MRemoving image: 93% complete...^MRemoving image: 94% complete...^MRemoving image: 95% complete...^MRemoving image: 96% complete...^MRemoving image: 97% complete...^MRemoving image: 98% complete...^MRemoving image: 99% complete...^MRemoving image: 99% complete...failed.
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:rbd: error: image still has watchers
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:27.129557 7fcfef4a9780 -1 librbd: error removing header: (16) Device or resource busy

Actions
Copy link

Also available in: Atom PDF