Tools » teuthology

Openstack is a strategic partnership and delivery vehicle for us, but we do not do automated testing of either the install or the correct operation of the resulting system. For now we probably have adequate functional validation tools, but we need to run them against full openstack systems installed by standard tools.

Can we deploy OpenStack on CentOS or, even better, RHEL please so we can improve our operational experience with this OS.

It appears the vps host I'm using to run OS doesn't have enough memory to run a guest that requires 64MB of memory. And as I type this, I discover I don't even have enough memory for /usr/bin/top ...

ubuntu@vpm112:~/devstack$ sudo reboot
bash: fork: Cannot allocate memory

So I'm in the process of building a new OS machine using a burnupi. It will have to be re-imaged when I am done. I'm attempting to get it to use the one node ceph cluster I created last week, and I'm having problems, not the least of which is the ceph machine being periodically unresponsive with a load average of 4... while otherwise appearing to sit idle.

Attempting to get the burnupi OS machine to talk to the ceph cluster...

ubuntu@vpm072:~$ ceph --verbose auth get-key client.cinder
Error initializing cluster client: Error

From dmesg:
https://gist.github.com/zmc/8696844

Josh was a big help today. He discovered that the OS node had very old ceph packages installed; upgrading them helped a bit. But then we found that the machine had a non-stock kernel on it that was causing problems.

Once the ceph and kernel packages were fixed on the OS node, I tried to access Horizon again, to no avail. I found that apache2 was only listening on 127.0.0.1 because it couldn't figure out what ServerName to use, so I did this:

ubuntu@burnupi23:~$ echo "ServerName burnupi23.front.sepia.ceph.com" > /etc/apache2/conf.d/servername
ubuntu@burnupi23:~$ /etc/init.d/apache2 restart

Next up, create an instance. It failed unhelpfully:

ubuntu@burnupi23:~$ nova list
+--------------------------------------+-------+--------+------------+-------------+----------+
| ID                                   | Name  | Status | Task State | Power State | Networks |
+--------------------------------------+-------+--------+------------+-------------+----------+
| 46e40b8f-b8c0-4432-97c1-4eb5fb3295a0 | tiny1 | ERROR  | -          | NOSTATE     |          |
+--------------------------------------+-------+--------+------------+-------------+----------+

2014-01-29 22:31:21.250+0000: 1624: error : virExecWithHook:327 : Cannot find 'pm-is-supported' in path: No such file or directory

Next, I noticed that devstack was trying to execute:

sg libvirtd /usr/local/bin/nova-compute --config-file /etc/nova/nova.conf

I'm working on the beginnings of a task that will install devstack on one node and ceph on another, and configure them to play nice. I hit a problem very early in stack.sh (which installs devstack):

[...]
The following packages will be upgraded:
  bc curl iptables libcurl3 libgssapi-krb5-2 libk5crypto3 libkrb5-3 libkrb5support0 libldap-2.4-2 libsqlite3-0 libxml2 openssh-client openssh-server openssl parted python-boto
  sudo
17 upgraded, 116 newly installed, 0 to remove and 124 not upgraded.
Need to get 31.0 MB of archives.
After this operation, 111 MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
  libcurl3 curl
E: There are problems and -y was used without --force-yes
+++ failed
+++ local r=100
++++ jobs -p
+++ kill
+++ set +o xtrace
ubuntu@vpm001:~/devstack$ apt-cache policy curl
curl:
  Installed: 7.22.0-3ubuntu4
  Candidate: 7.29.0-1precise.ceph
  Version table:
     7.29.0-1precise.ceph 0
        500 http://ceph.com/packages/ceph-extras/debian/ precise/main amd64 Packages
     7.22.0-3ubuntu4.6 0
        500 http://apt-mirror.front.sepia.ceph.com/archive.ubuntu.com/ubuntu/ precise-updates/main amd64 Packages
        500 http://apt-mirror.front.sepia.ceph.com/security.ubuntu.com/ubuntu/ precise-security/main amd64 Packages
 *** 7.22.0-3ubuntu4 0
        500 http://apt-mirror.front.sepia.ceph.com/archive.ubuntu.com/ubuntu/ precise/main amd64 Packages
        100 /var/lib/dpkg/status

I'm unsure why we are building copies of curl...

All previous issues have been resolved (I think). I've got the task mostly finished, but right now I'm hitting an issue with configparser where it isn't parsing the OS config files at all. No error or anything, just empty objects...

The task is just about finished. It successfully brings up Ceph and OpenStack and configures Glance and Cinder, making sure services are up.

But it runs out of memory. I need to somehow provision vps nodes with more.

Solved that last issue yesterday. The task should be about done as far as establishing the cluster is concerned. I do need to figure out what sort of smoke testing to do, though.

Waited 2+ hours for my cluster to get built; VMs were totally borked. I guess I'll try later when we have more resources available.

The task now runs exercise.sh, which is part of the devstack repo - unfortunately there are lots of failures. I'm asking around for advice about what might make sense to smoke-test.