Bug #4003
Updated by Alex Elder about 11 years ago
From the teuthology kernel untar task on rbd, we get EBUSY trying to unmap. I'm investigating that this isn't somehow caused by teuthology changes...
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_cm.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_marshall.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_sa.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_umem.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_pack.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/Kbuild'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_sa.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_user_cm.h'
2013-02-04T10:40:39.324 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_cm.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/iw_cm.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/ib_user_verbs.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/include/rdma/rdma_cm_ib.h'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9/include/rdma'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9/include'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9/vmlinux.o'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `linux-3.2.9'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed directory: `t'
2013-02-04T10:40:39.325 INFO:teuthology.task.workunit.client.0.out:removed `linux-3.2.9.tar.bz2'
2013-02-04T10:40:39.326 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'rm -rf -- /tmp/cephtest/mnt.0/client.0/tmp'
2013-02-04T10:40:39.336 INFO:teuthology.task.workunit:Stopping kernel_untar_build.sh on client.0...
2013-02-04T10:40:39.337 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'rm -rf -- /tmp/cephtest/workunits.list /tmp/cephtest/workunit.client.0'
2013-02-04T10:40:39.383 DEBUG:teuthology.parallel:result is None
2013-02-04T10:40:39.383 DEBUG:teuthology.run_tasks:Unwinding manager <contextlib.GeneratorContextManager object at 0x194e110>
2013-02-04T10:40:39.383 INFO:teuthology.task.rbd:Unmounting rbd images...
2013-02-04T10:40:39.384 INFO:teuthology.task.rbd:Unmapping rbd devices...
2013-02-04T10:40:39.384 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'sudo LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/rbd -c /tmp/cephtest/ceph.conf -p rbd unmap /dev/rbd/rbd/testimage.client.0 && while test -e /dev/rbd/rbd/testimage.client.0 ; do sleep 1 ; done'
2013-02-04T10:40:39.489 INFO:teuthology.orchestra.run.err:rbd: failed to remove rbd device: (16) Device or resource busy
2013-02-04T10:40:39.489 INFO:teuthology.orchestra.run.err:rbd: remove failed: (16) Device or resource busy
2013-02-04T10:40:39.491 INFO:teuthology.task.rbd:Unloading rbd kernel module...
2013-02-04T10:40:39.491 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'sudo modprobe -r rbd || true'
2013-02-04T10:40:39.506 INFO:teuthology.orchestra.run.err:FATAL: Module rbd is in use.
2013-02-04T10:40:39.508 INFO:teuthology.task.rbd:Deleting rbd images...
2013-02-04T10:40:39.508 DEBUG:teuthology.orchestra.run:Running [10.214.132.32]: 'LD_LIBRARY_PATH=/tmp/cephtest/binary/usr/local/lib /tmp/cephtest/enable-coredump /tmp/cephtest/binary/usr/local/bin/ceph-coverage /tmp/cephtest/archive/coverage /tmp/cephtest/binary/usr/local/bin/rbd -c /tmp/cephtest/ceph.conf -p rbd rm testimage.client.0'
2013-02-04T10:40:39.914 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:18.959420 7fcfda6f4700 0 -- 10.214.132.32:0/1029143 >> 10.214.132.29:6800/8592 pipe(0x2085340 sd=12 :36056 s=2 pgs=10 cs=1 l=1).injecting socket failure
2013-02-04T10:40:39.966 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:19.011264 7fcfdacfa700 0 -- 10.214.132.32:0/1029143 >> 10.214.132.31:6803/22416 pipe(0x207ddd0 sd=9 :52957 s=2 pgs=18 cs=1 l=1).injecting socket failure
2013-02-04T10:40:48.084 INFO:teuthology.orchestra.run.out:^MRemoving image: 1% complete...^MRemoving image: 2% complete...^MRemoving image: 3% complete...^MRemoving image: 4% complete...^MRemoving image: 5% complete...^MRemoving image: 6% complete...^MRemoving image: 7% complete...^MRemoving image: 8% complete...^MRemoving image: 9% complete...^MRemoving image: 10% complete...^MRemoving image: 11% complete...^MRemoving image: 12% complete...^MRemoving image: 13% complete...^MRemoving image: 14% complete...^MRemoving image: 15% complete...^MRemoving image: 16% complete...^MRemoving image: 17% complete...^MRemoving image: 18% complete...^MRemoving image: 19% complete...^MRemoving image: 20% complete...^MRemoving image: 21% complete...^MRemoving image: 22% complete...^MRemoving image: 23% complete...^MRemoving image: 24% complete...^MRemoving image: 25% complete...^MRemoving image: 26% complete...^MRemoving image: 27% complete...^MRemoving image: 28% complete...^MRemoving image: 29% complete...^MRemoving image: 30% complete...^MRemoving image: 31% complete...^MRemoving image: 32% complete...^MRemoving image: 33% complete...^MRemoving image: 34% complete...^MRemoving image: 35% complete...^MRemoving image: 36% complete...^MRemoving image: 37% complete...^MRemoving image: 38% complete...^MRemoving image: 39% complete...^MRemoving image: 40% complete...^MRemoving image: 41% complete...^MRemoving image: 42% complete...^MRemoving image: 43% complete...^MRemoving image: 44% complete...^MRemoving image: 45% complete...^MRemoving image: 46% complete...^MRemoving image: 47% complete...^MRemoving image: 48% complete...^MRemoving image: 49% complete...^MRemoving image: 50% complete...^MRemoving image: 51% complete...^MRemoving image: 52% complete...^MRemoving image: 53% complete...^MRemoving image: 54% complete...^MRemoving image: 55% complete...^MRemoving image: 56% complete...^MRemoving image: 57% complete...^MRemoving image: 58% complete...^MRemoving image: 59% complete...^MRemoving image: 60% complete...^MRemoving image: 61% complete...^MRemoving image: 62% complete...^MRemoving image: 63% complete...^MRemoving image: 64% complete...^MRemoving image: 65% complete...^MRemoving image: 66% complete...^MRemoving image: 67% complete...^MRemoving image: 68% complete...^MRemoving image: 69% complete...^MRemoving image: 70% complete...^MRemoving image: 71% complete...^MRemoving image: 72% complete...^MRemoving image: 73% complete...^MRemoving image: 74% complete...^MRemoving image: 75% complete...^MRemoving image: 76% complete...^MRemoving image: 77% complete...^MRemoving image: 78% complete...^MRemoving image: 79% complete...^MRemoving image: 80% complete...^MRemoving image: 81% complete...^MRemoving image: 82% complete...^MRemoving image: 83% complete...^MRemoving image: 84% complete...^MRemoving image: 85% complete...^MRemoving image: 86% complete...^MRemoving image: 87% complete...^MRemoving image: 88% complete...^MRemoving image: 89% complete...^MRemoving image: 90% complete...^MRemoving image: 91% complete...^MRemoving image: 92% complete...^MRemoving image: 93% complete...^MRemoving image: 94% complete...^MRemoving image: 95% complete...^MRemoving image: 96% complete...^MRemoving image: 97% complete...^MRemoving image: 98% complete...^MRemoving image: 99% complete...^MRemoving image: 99% complete...failed.
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:rbd: error: image still has watchers
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
2013-02-04T10:40:48.085 INFO:teuthology.orchestra.run.err:2013-02-04 10:40:27.129557 7fcfef4a9780 -1 librbd: error removing header: (16) Device or resource busy