Ceph » devops

Run: http://pulpito.ceph.com/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/
Jobs: almost all
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/890375/

2015-05-13T16:20:24.061 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b
2015-05-13T16:20:24.061 INFO:teuthology.orchestra.run.plana83:Running: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version'
2015-05-13T16:20:24.062 INFO:teuthology.orchestra.run.burnupi56:Running: 'lsb_release -sc'
2015-05-13T16:20:24.147 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install
    raise VersionNotFoundError("%s/version" % base_url)
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
2015-05-13T16:20:24.148 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 833, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 578, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 89, in __exit__
    raise
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version

And in this run as well - http://pulpito.ceph.com/teuthology-2015-05-12_17:42:50-rgw-hammer-distro-basic-multi/

Every time I load one of those gitbuilder "version" URLs, I get something back.

$ curl -I http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
HTTP/1.1 200 OK
Date: Thu, 14 May 2015 16:34:47 GMT
Server: Apache/2.2.22 (Ubuntu)
Last-Modified: Thu, 14 May 2015 06:52:20 GMT
ETag: "e00dffb24-1d-516052b42d100" 
Accept-Ranges: bytes
Content-Length: 29

I don't know what this implies for these teuthology failures.

It would be really great if we could have Teuthology log the full HTTP error that wget sees, instead of just checking wget's error code and logging "it failed".

Again in run: http://pulpito.ceph.com/teuthology-2015-05-13_15:03:35-fs-hammer-testing-basic-multi/

Jobs: 889968, 889987

Logs: http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-13_15:03:35-fs-hammer-testing-basic-multi/889987/

Every time I check the URLs in the logs here, the HTTP response is 200 OK, and there are valid version strings available in the HTTP response body.

Filed #11630 to get more information here.

In the smoke suite, no os_type was provided so we don't have a way to check for packages up front.

In the upgrade suite, we aren't currently scanning for every instance of the install task to see if we're doing an upgrade in order to check for those additional packages.

I do see some strangeness though: It looks like two out of three remotes is fetching the URL successfully; at this point I think it makes sense to use the requests module locally, and call raise_for_status() on the response object so that we can see what HTTP error is occurring.

https://github.com/ceph/teuthology/pull/505