Bug #17049
closed
"Cannot allocate memory" aka "rbd: error: image still has watchers" in rbd-master-distro-basic-vps
Added by Yuri Weinstein over 7 years ago.
Updated over 7 years ago.
Description
This is QE validation oh hammer 0.94.8
sha1 08277b7bc7c0e533c3fd56a0040dc0ddc74637d6
Run: http://pulpito.ceph.com/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/
Job: many
Logs: http://qa-proxy.ceph.com/teuthology/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368522/teuthology.log
2016-08-16T17:54:26.466 INFO:tasks.qemu.client.0.vpm137.stderr:daemon-helper: command failed with exit status 1
2016-08-16T17:54:26.466 INFO:tasks.rbd:Deleting rbd clones...
2016-08-16T17:54:26.466 INFO:teuthology.orchestra.run.vpm137:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0-clone'
2016-08-16T17:54:26.561 INFO:teuthology.orchestra.run.vpm137.stderr:2016-08-16 17:54:26.733275 7f2ad6bc27c0 -1 librbd: image has watchers - not removing
2016-08-16T17:54:26.561 INFO:teuthology.orchestra.run.vpm137.stderr:
Removing image: 0% complete...failed.
2016-08-16T17:54:26.561 INFO:teuthology.orchestra.run.vpm137.stderr:rbd: error: image still has watchers
2016-08-16T17:54:26.561 INFO:teuthology.orchestra.run.vpm137.stderr:This means the image is still open or the client using it crashed. Try again after closing/unmapping it or waiting 30s for the crashed client to timeout.
2016-08-16T17:54:26.561 DEBUG:tasks.qemu:cleaning up base image files
2016-08-16T17:54:26.562 INFO:teuthology.orchestra.run.vpm137:Running: 'rm -f /home/ubuntu/cephtest/qemu/base.client.0.qcow2'
2016-08-16T17:54:26.669 INFO:teuthology.orchestra.run.vpm137:Running: 'rm -f /home/ubuntu/cephtest/qemu/client.0.iso /home/ubuntu/cephtest/qemu/userdata.client.0 /home/ubuntu/cephtest/qemu/metadata.client.0 /home/ubuntu/cephtest/qemu/client.0.test.sh'
2016-08-16T17:54:26.675 INFO:teuthology.orchestra.run.vpm137:Running: 'rmdir /home/ubuntu/cephtest/qemu || true'
2016-08-16T17:54:26.748 INFO:tasks.rbd:Deleting rbd images...
2016-08-16T17:54:26.749 INFO:teuthology.orchestra.run.vpm137:Running: 'adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rbd -p rbd rm client.0.0'
2016-08-16T17:54:26.894 INFO:teuthology.orchestra.run.vpm137.stderr:2016-08-16 17:54:27.062797 7fb14ccb57c0 -1 librbd: image has snapshots - not removing
2016-08-16T17:54:26.895 INFO:teuthology.orchestra.run.vpm137.stderr:
Removing image: 0% complete...failed.
2016-08-16T17:54:26.895 INFO:teuthology.orchestra.run.vpm137.stderr:rbd: image has snapshots - these must be deleted with 'rbd snap purge' before the image can be removed.
2016-08-16T17:54:26.895 ERROR:teuthology.run_tasks:Manager failed: qemu
- Project changed from Ceph to sepia
So this is low memory on vps nodes after all :(
FYI
yuriw@teuthology ~ [20:33:13]> grep "Cannot allocate memory" /a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/*/teuthology.log
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368407/teuthology.log:2016-08-16T16:54:02.222 INFO:tasks.qemu.client.0.vpm181.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368416/teuthology.log:2016-08-16T16:54:15.319 INFO:tasks.qemu.client.0.vpm117.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368424/teuthology.log:2016-08-16T16:53:51.800 INFO:tasks.qemu.client.0.vpm061.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368434/teuthology.log:2016-08-16T16:53:54.719 INFO:tasks.qemu.client.0.vpm047.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368442/teuthology.log:2016-08-16T17:22:09.438 INFO:tasks.qemu.client.0.vpm085.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368450/teuthology.log:2016-08-16T17:40:22.819 INFO:tasks.qemu.client.0.vpm085.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368460/teuthology.log:2016-08-16T19:59:38.798 INFO:tasks.qemu.client.0.vpm027.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368468/teuthology.log:2016-08-16T17:38:12.336 INFO:tasks.qemu.client.0.vpm061.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368476/teuthology.log:2016-08-16T18:25:58.526 INFO:tasks.qemu.client.0.vpm023.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368487/teuthology.log:2016-08-16T17:47:58.805 INFO:tasks.qemu.client.0.vpm181.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368495/teuthology.log:2016-08-16T19:02:34.670 INFO:tasks.qemu.client.0.vpm117.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368504/teuthology.log:2016-08-16T18:13:48.454 INFO:tasks.qemu.client.0.vpm005.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368513/teuthology.log:2016-08-16T17:49:20.303 INFO:tasks.qemu.client.0.vpm197.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368522/teuthology.log:2016-08-16T17:54:26.466 INFO:tasks.qemu.client.0.vpm137.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368529/teuthology.log:2016-08-16T19:33:24.117 INFO:tasks.qemu.client.0.vpm085.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368540/teuthology.log:2016-08-16T18:06:36.154 INFO:tasks.qemu.client.0.vpm175.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368547/teuthology.log:2016-08-16T19:36:12.308 INFO:tasks.qemu.client.0.vpm069.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
/a/yuriw-2016-08-16_16:29:58-rbd-master-distro-basic-vps/368556/teuthology.log:2016-08-16T19:31:57.237 INFO:tasks.qemu.client.0.vpm053.stderr:Cannot set up guest memory 'pc.ram': Cannot allocate memory
- Subject changed from "rbd: error: image still has watchers" in rbd-master-distro-basic-vps to "Cannot allocate memory" aka "rbd: error: image still has watchers" in rbd-master-distro-basic-vps
Are these tests attempting to host VMs on VMs?
- Related to Bug #15052: Rebalance vps instances to allocate more RAM added
- Status changed from New to Won't Fix
Also available in: Atom
PDF