Bug #11591
"VersionNotFoundError: Failed to fetch package version" in rados-hammer-distro-basic-typica run
0%
Description
Run: http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-11_21:02:36-rados-hammer-distro-basic-typica/
Jobs: many
Logs for one: http://typica002.front.sepia.ceph.com/teuthology-2015-05-11_21:02:36-rados-hammer-distro-basic-typica/14184/teuthology.log
basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main librados2-dbg amd64 0.94.1-111-gb69fb89-1trusty [18.4 MB] 2015-05-12T02:11:11.824 INFO:teuthology.orchestra.run.typica019.stdout:Get:4 http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main librbd1-dbg amd64 0.94.1-111-gb69fb89-1trusty [22.1 MB] 2015-05-12T02:11:12.653 INFO:teuthology.orchestra.run.typica019.stdout:Get:5 http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main radosgw-dbg amd64 0.94.1-111-gb69fb89-1trusty [22.4 MB] 2015-05-12T02:11:13.919 INFO:teuthology.orchestra.run.typica019.stderr: Extracting templates from packages: 76% Extracting templates from packages: 100% 2015-05-12T02:11:13.966 INFO:teuthology.orchestra.run.typica019.stdout:Fetched 464 MB in 19s (23.9 MB/s) 2015-05-12T02:11:17.427 ERROR:teuthology.parallel:Exception in parallel execution Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__ for result in self: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next resurrect_traceback(result) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install raise VersionNotFoundError("%s/version" % base_url) VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
History
#1 Updated by Yuri Weinstein almost 9 years ago
Run: http://pulpito.ceph.com/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/
Jobs: almost all
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/890375/
2015-05-13T16:20:24.061 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b 2015-05-13T16:20:24.061 INFO:teuthology.orchestra.run.plana83:Running: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version' 2015-05-13T16:20:24.062 INFO:teuthology.orchestra.run.burnupi56:Running: 'lsb_release -sc' 2015-05-13T16:20:24.147 ERROR:teuthology.parallel:Exception in parallel execution Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__ for result in self: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next resurrect_traceback(result) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install raise VersionNotFoundError("%s/version" % base_url) VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version 2015-05-13T16:20:24.148 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 833, in install install_packages(ctx, install_info, config) File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 578, in install_packages ctx, remote, pkgs[system_type], config) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 89, in __exit__ raise VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
#2 Updated by Yuri Weinstein almost 9 years ago
- ceph-qa-suite rgw added
#3 Updated by Yuri Weinstein almost 9 years ago
And in this run as well - http://pulpito.ceph.com/teuthology-2015-05-12_17:42:50-rgw-hammer-distro-basic-multi/
#4 Updated by Yuri Weinstein almost 9 years ago
- Severity changed from 3 - minor to 1 - critical
#5 Updated by Ken Dreyer almost 9 years ago
Every time I load one of those gitbuilder "version" URLs, I get something back.
$ curl -I http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version HTTP/1.1 200 OK Date: Thu, 14 May 2015 16:34:47 GMT Server: Apache/2.2.22 (Ubuntu) Last-Modified: Thu, 14 May 2015 06:52:20 GMT ETag: "e00dffb24-1d-516052b42d100" Accept-Ranges: bytes Content-Length: 29
I don't know what this implies for these teuthology failures.
It would be really great if we could have Teuthology log the full HTTP error that wget sees, instead of just checking wget's error code and logging "it failed".
#6 Updated by Yuri Weinstein almost 9 years ago
#7 Updated by Ken Dreyer almost 9 years ago
Every time I check the URLs in the logs here, the HTTP response is 200 OK
, and there are valid version strings available in the HTTP response body.
Filed #11630 to get more information here.
#8 Updated by Zack Cerza almost 9 years ago
- Status changed from New to Resolved
My fix for #11645 should prevent this from happening; if it continues we can reopen.
#9 Updated by Yuri Weinstein almost 9 years ago
- Status changed from Resolved to New
Run: http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-24_05:00:01-smoke-master-distro-basic-typica/
Jobs: ['25050', '25057', '25059', '25060', '25063']
Log for one: http://typica002.front.sepia.ceph.com/teuthology-2015-05-24_05:00:01-smoke-master-distro-basic-typica/25050/
2015-05-27T07:46:33.362 INFO:teuthology.orchestra.run.typica054.stdout:Unpacking ceph-fuse (9.0.0-971-gd49d816-1trusty) ... 2015-05-27T07:46:37.818 ERROR:teuthology.parallel:Exception in parallel execution Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__ for result in self: File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next resurrect_traceback(result) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install raise VersionNotFoundError("%s/version" % base_url) VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/d49d8166a02750e5b6ff962a5813d670f848ff34/version 2015-05-27T07:46:37.821 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested vars.append(enter()) File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 833, in install install_packages(ctx, install_info, config) File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 578, in install_packages ctx, remote, pkgs[system_type], config) File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 89, in __exit__ raise VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/d49d8166a02750e5b6ff962a5813d670f848ff34/version
#10 Updated by Yuri Weinstein almost 9 years ago
- ceph-qa-suite smoke added
#11 Updated by Yuri Weinstein almost 9 years ago
- ceph-qa-suite upgrade/giant added
#12 Updated by Zack Cerza almost 9 years ago
In the smoke suite, no os_type
was provided so we don't have a way to check for packages up front.
In the upgrade suite, we aren't currently scanning for every instance of the install task to see if we're doing an upgrade in order to check for those additional packages.
I do see some strangeness though: It looks like two out of three remotes is fetching the URL successfully; at this point I think it makes sense to use the requests module locally, and call raise_for_status()
on the response object so that we can see what HTTP error is occurring.
#13 Updated by Zack Cerza almost 9 years ago
- Status changed from New to Need More Info
This should be done now:
https://github.com/ceph/teuthology/commit/aa267edf8f65ed7d39aa6e36863fd39de51fabb6
#14 Updated by Sage Weil about 8 years ago
- Status changed from Need More Info to Can't reproduce