Project

General

Profile

Bug #11591

"VersionNotFoundError: Failed to fetch package version" in rados-hammer-distro-basic-typica run

Added by Yuri Weinstein almost 9 years ago. Updated about 8 years ago.

Status:
Can't reproduce
Priority:
Urgent
Assignee:
-
Category:
-
Target version:
-
% Done:

0%

Source:
Q/A
Tags:
Backport:
Regression:
No
Severity:
1 - critical
Reviewed:
Affected Versions:
ceph-qa-suite:
rados, rgw, smoke, upgrade/giant
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Run: http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-11_21:02:36-rados-hammer-distro-basic-typica/
Jobs: many
Logs for one: http://typica002.front.sepia.ceph.com/teuthology-2015-05-11_21:02:36-rados-hammer-distro-basic-typica/14184/teuthology.log

basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main librados2-dbg amd64 0.94.1-111-gb69fb89-1trusty [18.4 MB]
2015-05-12T02:11:11.824 INFO:teuthology.orchestra.run.typica019.stdout:Get:4 http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main librbd1-dbg amd64 0.94.1-111-gb69fb89-1trusty [22.1 MB]
2015-05-12T02:11:12.653 INFO:teuthology.orchestra.run.typica019.stdout:Get:5 http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/ trusty/main radosgw-dbg amd64 0.94.1-111-gb69fb89-1trusty [22.4 MB]
2015-05-12T02:11:13.919 INFO:teuthology.orchestra.run.typica019.stderr:
Extracting templates from packages: 76%
Extracting templates from packages: 100%
2015-05-12T02:11:13.966 INFO:teuthology.orchestra.run.typica019.stdout:Fetched 464 MB in 19s (23.9 MB/s)
2015-05-12T02:11:17.427 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install
    raise VersionNotFoundError("%s/version" % base_url)
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version

History

#1 Updated by Yuri Weinstein almost 9 years ago

Run: http://pulpito.ceph.com/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/
Jobs: almost all
Logs for one: http://qa-proxy.ceph.com/teuthology/teuthology-2015-05-13_15:55:43-rgw-hammer-distro-basic-multi/890375/

2015-05-13T16:20:24.061 INFO:teuthology.task.install:Pulling from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b
2015-05-13T16:20:24.061 INFO:teuthology.orchestra.run.plana83:Running: 'wget -q -O- http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version'
2015-05-13T16:20:24.062 INFO:teuthology.orchestra.run.burnupi56:Running: 'lsb_release -sc'
2015-05-13T16:20:24.147 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install
    raise VersionNotFoundError("%s/version" % base_url)
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
2015-05-13T16:20:24.148 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 833, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 578, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 89, in __exit__
    raise
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version

#2 Updated by Yuri Weinstein almost 9 years ago

  • ceph-qa-suite rgw added

#4 Updated by Yuri Weinstein almost 9 years ago

  • Severity changed from 3 - minor to 1 - critical

#5 Updated by Ken Dreyer almost 9 years ago

Every time I load one of those gitbuilder "version" URLs, I get something back.

$ curl -I http://gitbuilder.ceph.com/ceph-deb-precise-x86_64-notcmalloc/sha1/b69fb89122d6f989152a29124cc7ed54b5e4d43b/version
HTTP/1.1 200 OK
Date: Thu, 14 May 2015 16:34:47 GMT
Server: Apache/2.2.22 (Ubuntu)
Last-Modified: Thu, 14 May 2015 06:52:20 GMT
ETag: "e00dffb24-1d-516052b42d100" 
Accept-Ranges: bytes
Content-Length: 29

I don't know what this implies for these teuthology failures.

It would be really great if we could have Teuthology log the full HTTP error that wget sees, instead of just checking wget's error code and logging "it failed".

#7 Updated by Ken Dreyer almost 9 years ago

Every time I check the URLs in the logs here, the HTTP response is 200 OK, and there are valid version strings available in the HTTP response body.

Filed #11630 to get more information here.

#8 Updated by Zack Cerza almost 9 years ago

  • Status changed from New to Resolved

My fix for #11645 should prevent this from happening; if it continues we can reopen.

#9 Updated by Yuri Weinstein almost 9 years ago

  • Status changed from Resolved to New

Run: http://pulpito-rdu.front.sepia.ceph.com/teuthology-2015-05-24_05:00:01-smoke-master-distro-basic-typica/
Jobs: ['25050', '25057', '25059', '25060', '25063']
Log for one: http://typica002.front.sepia.ceph.com/teuthology-2015-05-24_05:00:01-smoke-master-distro-basic-typica/25050/

2015-05-27T07:46:33.362 INFO:teuthology.orchestra.run.typica054.stdout:Unpacking ceph-fuse (9.0.0-971-gd49d816-1trusty) ...
2015-05-27T07:46:37.818 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 322, in _update_deb_package_list_and_install
    raise VersionNotFoundError("%s/version" % base_url)
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/d49d8166a02750e5b6ff962a5813d670f848ff34/version
2015-05-27T07:46:37.821 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 833, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 578, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 89, in __exit__
    raise
VersionNotFoundError: Failed to fetch package version from http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/sha1/d49d8166a02750e5b6ff962a5813d670f848ff34/version

#10 Updated by Yuri Weinstein almost 9 years ago

  • ceph-qa-suite smoke added

#12 Updated by Zack Cerza almost 9 years ago

In the smoke suite, no os_type was provided so we don't have a way to check for packages up front.

In the upgrade suite, we aren't currently scanning for every instance of the install task to see if we're doing an upgrade in order to check for those additional packages.

I do see some strangeness though: It looks like two out of three remotes is fetching the URL successfully; at this point I think it makes sense to use the requests module locally, and call raise_for_status() on the response object so that we can see what HTTP error is occurring.

https://github.com/ceph/teuthology/pull/505

#13 Updated by Zack Cerza almost 9 years ago

  • Status changed from New to Need More Info

#14 Updated by Sage Weil about 8 years ago

  • Status changed from Need More Info to Can't reproduce

Also available in: Atom PDF