Bug #21166: unable to reserve vps nodes - sepia - Ceph

Actions

Copy link

Bug #21166

closed

unable to reserve vps nodes

Added by Vasu Kulkarni over 6 years ago. Updated about 6 years ago.

Status:

Closed

Priority:

High

Assignee:

David Galloway

Category:

Infrastructure Service

Target version:

% Done:

Source:

Tags:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Crash signature (v1):

Crash signature (v2):

Description

I am unable to lock vps nodes and hit the following issue when reserving, previously this used to work


2017-08-29 03:41:55,972.972 INFO:teuthology.provision.downburst:Provisioning a centos 7.3 vps
2017-08-29 03:41:57,599.599 INFO:teuthology.provision.downburst:Fetching default SSH key from https://git.ceph.com/?p=keys.git;a=blob_plain;f=ssh/teuthology-ubuntu.pub;hb=HEAD

2017-08-29 03:41:57,599.599 INFO:teuthology.provision.downburst:DEBUG:downburst.create:Connecting to libvirt...
DEBUG:downburst.create:Opening libvirt pool...
DEBUG:downburst.image:Listing cloud image in libvirt...
DEBUG:downburst.image:Considering image: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Saw image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.image:Already have cloud image: centos-7.3-20170118-cloudimg-amd64.img
libvirt: QEMU Driver error : internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory
Traceback (most recent call last):
  File "/home/vasu/src/downburst/virtualenv/bin/downburst", line 9, in <module>
    load_entry_point('downburst', 'console_scripts', 'downburst')()
  File "/home/vasu/src/downburst/downburst/cli.py", line 64, in main
    return args.func(args)
  File "/home/vasu/src/downburst/downburst/create.py", line 209, in create
    dom.create()
  File "/home/vasu/src/downburst/virtualenv/local/lib/python2.7/site-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory

2017-08-29 03:41:57,599.599 INFO:teuthology.provision.downburst:Downburst failed on ubuntu@vpm135.front.sepia.ceph.com: DEBUG:downburst.create:Connecting to libvirt...
DEBUG:downburst.create:Opening libvirt pool...
DEBUG:downburst.image:Listing cloud image in libvirt...
DEBUG:downburst.image:Considering image: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Saw image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.image:Already have cloud image: centos-7.3-20170118-cloudimg-amd64.img
libvirt: QEMU Driver error : internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory
Traceback (most recent call last):
  File "/home/vasu/src/downburst/virtualenv/bin/downburst", line 9, in <module>
    load_entry_point('downburst', 'console_scripts', 'downburst')()
  File "/home/vasu/src/downburst/downburst/cli.py", line 64, in main
    return args.func(args)
  File "/home/vasu/src/downburst/downburst/create.py", line 209, in create
    dom.create()
  File "/home/vasu/src/downburst/virtualenv/local/lib/python2.7/site-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: process exited while connecting to monitor: Cannot set up guest memory 'pc.ram': Cannot allocate memory
2017-08-29 03:41:57,600.600 ERROR:teuthology.lock.ops:Unable to create virtual machine: ubuntu@vpm135.front.sepia.ceph.com
2017-08-29 03:41:58,133.133 INFO:teuthology.provision.downburst:
2017-08-29 03:41:58,133.133 INFO:teuthology.provision.downburst:DEBUG:downburst.destroy:Connecting to libvirt...
DEBUG:downburst.destroy:Terminating the virtual machine
libvirt: QEMU Driver error : Requested operation is not valid: domain is not running
DEBUG:downburst.destroy:Getting livirt pool list...
DEBUG:downburst.destroy:Checking Pool: default
DEBUG:downburst.destroy:Checking Pool: vpm135
DEBUG:downburst.destroy:Checking Volumel: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.destroy:Checking Volumel: vpm135.img
DEBUG:downburst.destroy:Deleting volume: 'vpm135.img'
DEBUG:downburst.destroy:Checking Volumel: cloud-init.vpm135.iso
DEBUG:downburst.destroy:Deleting volume: 'cloud-init.vpm135.iso'
DEBUG:downburst.destroy:Checking Volumel: vpm135-2.img
DEBUG:downburst.destroy:Deleting volume: 'vpm135-2.img'
DEBUG:downburst.destroy:Checking Volumel: vpm135-3.img
DEBUG:downburst.destroy:Deleting volume: 'vpm135-3.img'
DEBUG:downburst.destroy:Checking Volumel: vpm135-4.img
DEBUG:downburst.destroy:Deleting volume: 'vpm135-4.img'
DEBUG:downburst.destroy:Checking Volumel: vpm135-5.img
DEBUG:downburst.destroy:Deleting volume: 'vpm135-5.img'
DEBUG:downburst.destroy:Checking Pool: vpm131
DEBUG:downburst.destroy:Checking Volumel: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.destroy:Checking Volumel: vpm131.img
DEBUG:downburst.destroy:Checking Volumel: cloud-init.vpm131.iso
DEBUG:downburst.destroy:Checking Volumel: vpm131-2.img
DEBUG:downburst.destroy:Checking Volumel: vpm131-3.img
DEBUG:downburst.destroy:Checking Volumel: vpm131-4.img
DEBUG:downburst.destroy:Checking Volumel: vpm131-5.img
DEBUG:downburst.destroy:Checking Pool: vpm133
DEBUG:downburst.destroy:Checking Volumel: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.destroy:Checking Volumel: vpm133.img
DEBUG:downburst.destroy:Checking Volumel: cloud-init.vpm133.iso
DEBUG:downburst.destroy:Checking Volumel: vpm133-2.img
DEBUG:downburst.destroy:Checking Volumel: vpm133-3.img
DEBUG:downburst.destroy:Checking Volumel: vpm133-4.img
DEBUG:downburst.destroy:Checking Volumel: vpm133-5.img
DEBUG:downburst.destroy:Checking Pool: vpm129
DEBUG:downburst.destroy:Checking Volumel: ubuntu-14.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: ubuntu-16.04-20170307-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.destroy:Checking Volumel: vpm129.img
DEBUG:downburst.destroy:Checking Volumel: cloud-init.vpm129.iso
DEBUG:downburst.destroy:Checking Volumel: vpm129-2.img
DEBUG:downburst.destroy:Checking Volumel: vpm129-3.img
DEBUG:downburst.destroy:Checking Volumel: vpm129-4.img
DEBUG:downburst.destroy:Checking Volumel: vpm129-5.img

2017-08-29 03:41:58,133.133 INFO:teuthology.provision.downburst:Destroyed vpm135.front.sepia.ceph.com
2017-08-29 03:41:58,156.156 INFO:teuthology.lock.ops:unlocked vpm135.front.sepia.ceph.com
2017-08-29 03:41:58,282.282 ERROR:teuthology.misc:usage: ssh-keyscan [-46cHv] [-f file] [-p port] [-T timeout] [-t type]
2017-08-29 03:41:58,282.282 ERROR:teuthology.misc:[host | addrlist namelist] ...
2017-08-29 03:41:58,317.317 INFO:teuthology.provision.downburst:Provisioning a centos 7.3 vps
2017-08-29 03:42:01,963.963 INFO:teuthology.provision.downburst:Fetching default SSH key from https://git.ceph.com/?p=keys.git;a=blob_plain;f=ssh/teuthology-ubuntu.pub;hb=HEAD

2017-08-29 03:42:01,963.963 INFO:teuthology.provision.downburst:DEBUG:downburst.create:Connecting to libvirt...
DEBUG:downburst.create:Opening libvirt pool...
DEBUG:downburst.image:Listing cloud image in libvirt...
DEBUG:downburst.image:Considering image: centos-6.4-20140108-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: debian-7.0-20130801-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: ubuntu-12.04-20140130-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-6.5-20131214-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: ubuntu-12.10-20140409-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: opensuse-12.2-20121108-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: centos-6.4-20140908-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: centos-6.5-20140908-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: centos-7.2-20151214-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.0-20140723-cloudimg-amd64.raw
DEBUG:downburst.image:Considering image: ubuntu-16.04-20160610-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Saw image: centos-7.3-20170118-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: fedora-25-20161115-cloudimg-amd64.img
DEBUG:downburst.image:Considering image: ubuntu-14.04-20170302-cloudimg-amd64.img
DEBUG:downburst.image:Already have cloud image: centos-7.3-20170118-cloudimg-amd64.img
libvirt: QEMU Driver error : internal error: early end of file from monitor: possible problem:
Cannot set up guest memory 'pc.ram': Cannot allocate memory
Traceback (most recent call last):
  File "/home/vasu/src/downburst/virtualenv/bin/downburst", line 9, in <module>
    load_entry_point('downburst', 'console_scripts', 'downburst')()
  File "/home/vasu/src/downburst/downburst/cli.py", line 64, in main
    return args.func(args)
  File "/home/vasu/src/downburst/downburst/create.py", line 209, in create
    dom.create()
  File "/home/vasu/src/downburst/virtualenv/local/lib/python2.7/site-packages/libvirt.py", line 1035, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: early end of file from monitor: possible problem:
Cannot set up guest memory 'pc.ram': Cannot allocate memory

2017-08-29 03:42:01,963.963 INFO:teuthology.provision.downburst:Downburst failed on ubuntu@vpm093.front.sepia.ceph.com: DEBUG:downburst.create:Connecting to libvirt...