Ceph

we may need to check this for centos as well. I am not able to check this right now as v0.67.1 is missing on gitbuilder.

ubuntu@teuthology:/a/teuthology-2013-08-23_01:35:04-upgrade-parallel-next-testing-basic-vps/1749

2013-08-23T02:34:41.340 INFO:teuthology.orchestra.run.out:[10.214.138.70]: No package ceph-debug-0.67.1-17.geb4380d.el6 available.
2013-08-23T02:34:41.828 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/teuthology-next/teuthology/contextutil.py", line 25, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/teuthology-next/teuthology/task/install.py", line 602, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/teuthology-next/teuthology/task/install.py", line 384, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/teuthology-next/teuthology/parallel.py", line 88, in __exit__
    raise
CommandFailedError: Command failed on 10.214.138.70 with status 1: 'sudo yum install ceph-debug-0.67.1-17.geb4380d.el6 -y'

adding a note here for centos, so we keep track of it.
will file a separate bug for this, if it happens again.

The debug packages are built following the conventions of the platforms they are built on.

For debian the packages are named <package>-dbg... and there is a debug package built for each subpackage with binaries.

For rpm based platforms the debug packages are named <package>-debuginfo and there is just one debug package built for the main package which includes the debug info for all the sub-packages.

Within this scheme there are two anomolies, one of the fedora18 builds uploaded an incomplete debuginfo file that had a suffix on the package name. Subsequent builds produced correctly named packages. The centos build invoked the debug_package macro directly instead of via redhat-rpm-config. The centos gitbuilder has been updated to follow the redhat conventions.

The problem with suffix on the debug package is happening intermittently on fedora17, fedora18, and sles11sp2 gitbuilders. Fedora19 seems to be ok. It's not frequent, and hasn't occurred while I've been watching.

The logs show that the rpm is being built with the correct name, but when it is rsynced to gitbuilder.ceph.com, it has the suffix appended. I'm guessing that problem might be load or disk space releated, since the debug package is one of the larger ones at > 300MB. The package is probably copied to a temp file, then renamed to the original as part of the some process and the rename is failing. Probably the package signing, yum repo creation, of the gitbuilder copyout steps.

The incorrectly named debug package is valid and signed with the autobuild key. So we know that it's good through at least that phase of the build process. The permissions on the incorrectly named file are 600 instead of 644.

[ubuntu@gitbuilder-centos6-amd64 tmp]$ rpm -qi -p ceph-debuginfo-0.67.2-18.g3f86634.fc18.x86_64.rpm.anQj8O
Name : ceph-debuginfo Relocations: (not relocatable)
Version : 0.67.2 Vendor: (none)
Release : 18.g3f86634.fc18 Build Date: Tue 27 Aug 2013 09:44:50 PM EDT
Install Date: (not installed) Build Host: gitbuilder-fedora18-amd64.front.sepia.ceph.com
Group : Development/Debug Source RPM: ceph-0.67.2-18.g3f86634.fc18.src.rpm
Size : 1689455008 License: GPL-2.0
Signature : DSA/SHA1, Tue 27 Aug 2013 09:50:19 PM EDT, Key ID 6eaeae2203c3951a
URL : http://ceph.com/
Summary : Debug information for package ceph
Description :
This package provides debug information for package ceph.
Debug information is useful when developing applications that use this
package or when debugging this package.

Looks the issue is with the expect script that does the rpm signing timing out. It has a default time out of 10 seconds. When the system is heavily loaded, it's taking more that 10 seconds for the rpm sign program to start up and prompt for the pass phrase. The start up time seems to be proportionally to the size of the rpm. Most rpms are less than 1MB. The debuginfo rpms are over 300MB.

I've disabled the timeout in the expect script and that appears to have solved the problem.