Project

General

Profile

Actions
Copy link

Bug #13204

closed

gitbuilder broken symlink leading to 403 for 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'

Added by Vasu Kulkarni over 8 years ago. Updated over 8 years ago.

Status:
Resolved
Priority:
High
Assignee:
Dan Mick
Category:
-
Target version:
-
% Done:

0%

Source:
other
Tags:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
rados
Pull request ID:
Crash signature (v1):
Crash signature (v2):

Description

Rados hammer singleton-nomsgr test failure, either the version should be modified in config.yaml( 0.80.8 to firefly ) or the permission issue should be fixed for this issue

http://magna002.ceph.redhat.com/teuthology-2015-09-22_21:00:01-rados-hammer-distro-basic-magna/201397/teuthology.log

2015-09-22T22:34:10.725 INFO:teuthology.task.install:Installing packages: ceph-debuginfo, ceph-radosgw, ceph-test, ceph-devel, ceph, ceph-fuse, cephfs-java, libcephfs_jni1, libcephfs1, librados2, librbd1, python-ceph, rbd-fuse, librbd1, librados2 on remote rpm x86_64
2015-09-22T22:34:10.726 INFO:teuthology.orchestra.run.magna067:Running: 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
2015-09-22T22:34:10.805 INFO:teuthology.orchestra.run.magna067.stderr:--2015-09-22 22:34:10--  http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm
2015-09-22T22:34:10.807 INFO:teuthology.orchestra.run.magna067.stderr:Resolving gitbuilder.ceph.redhat.com (gitbuilder.ceph.redhat.com)... 10.8.135.248
2015-09-22T22:34:10.808 INFO:teuthology.orchestra.run.magna067.stderr:Connecting to gitbuilder.ceph.redhat.com (gitbuilder.ceph.redhat.com)|10.8.135.248|:80... connected.
2015-09-22T22:34:10.968 INFO:teuthology.orchestra.run.magna067.stderr:HTTP request sent, awaiting response... 403 Forbidden
2015-09-22T22:34:10.969 INFO:teuthology.orchestra.run.magna067.stderr:2015-09-22 22:34:10 ERROR 403: Forbidden.
2015-09-22T22:34:10.970 INFO:teuthology.orchestra.run.magna067.stderr:
2015-09-22T22:34:10.971 ERROR:teuthology.parallel:Exception in parallel execution
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 250, in _update_rpm_package_list_and_install
    remote.run(args=['wget', base_url, ],)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on magna067 with status 8: 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
2015-09-22T22:34:10.974 ERROR:teuthology.contextutil:Saw exception from nested tasks
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 638, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 381, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 250, in _update_rpm_package_list_and_install
    remote.run(args=['wget', base_url, ],)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on magna067 with status 8: 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
2015-09-22T22:34:10.975 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/teuthology_master/teuthology/run_tasks.py", line 56, in run_tasks
    manager.__enter__()
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 1211, in task
    lambda: ship_utilities(ctx=ctx, config=None),
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/contextutil.py", line 28, in nested
    vars.append(enter())
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 638, in install
    install_packages(ctx, install_info, config)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 381, in install_packages
    ctx, remote, pkgs[system_type], config)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 82, in __exit__
    for result in self:
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 101, in next
    resurrect_traceback(result)
  File "/home/teuthworker/src/teuthology_master/teuthology/parallel.py", line 19, in capture_traceback
    return func(*args, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/task/install.py", line 250, in _update_rpm_package_list_and_install
    remote.run(args=['wget', base_url, ],)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/remote.py", line 156, in run
    r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 378, in run
    r.wait()
  File "/home/teuthworker/src/teuthology_master/teuthology/orchestra/run.py", line 114, in wait
    label=self.label)
CommandFailedError: Command failed on magna067 with status 8: 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
2015-09-22T22:34:11.025 ERROR:teuthology.run_tasks: Sentry event: http://sentry.ceph.com/redhat/teuthology/search?q=a4f4cc4a23864a44a729ea1b44c19797
CommandFailedError: Command failed on magna067 with status 8: 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
Actions
Copy link
 #1

Updated by Zack Cerza over 8 years ago

http://gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/ yields a 403 instead of a 404, which would be correct since the path does not exist.

Actions
Copy link
 #2

Updated by Zack Cerza over 8 years ago

On gitbuilder-archive:

root@gitbuilder-archive:/var/log/apache2# grep Log /etc/apache2/sites-enabled/gitbuilder.ceph.com
ErrorLog /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/error.log
CustomLog /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/access.log "%h %l %u %t \"%r\" %s %O \"%{Referer}i\" \"%{User-agent}i\" " 
root@gitbuilder-archive:/var/log/apache2# ls -lh /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/access.log
-rw-r--r-- 1 root root 74G Sep 23 13:49 /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/access.log
root@gitbuilder-archive:/var/log/apache2# ls -lh /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/error.log
-rw-r--r-- 1 root root 39G Sep 23 13:49 /home/gitbuilder/logs/gitbuilder.ceph.com/http.20106984/error.log

Also on there:

[Wed Sep 23 13:51:40 2015] [error] [client 161.98.8.1] Symbolic link not allowed or link target not accessible: /home/gitbuilder/gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8


root@gitbuilder-archive:/var/log/apache2# ls -l /home/gitbuilder/gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8
lrwxrwxrwx 1 gitbuilder pg1532828 48 Sep 23 11:26 /home/gitbuilder/gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8 -> ../sha1/69eaad7f8308f21573c604f121956e64679a52a7
root@gitbuilder-archive:/var/log/apache2# ls -l /home/gitbuilder/gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/../sha1/69eaad7f8308f21573c604f121956e64679a52a7
ls: cannot access /home/gitbuilder/gitbuilder.ceph.com/ceph-rpm-centos7-x86_64-basic/ref/../sha1/69eaad7f8308f21573c604f121956e64679a52a7: No such file or directory

Summary: broken symlink along with strangely-located, unrotated logfiles.

I'm not sure what a "real" fix would entail.

Actions
Copy link
 #3

Updated by Zack Cerza over 8 years ago

  • Project changed from Ceph to devops
  • Subject changed from gitbuilder permission denied for 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm' to gitbuilder broken symlink leading to 403 for 'wget http://gitbuilder.ceph.redhat.com/ceph-rpm-centos7-x86_64-basic/ref/v0.80.8/noarch/ceph-release-1-0.el7.noarch.rpm'
Actions
Copy link
 #4

Updated by Dan Mick over 8 years ago

So the upshot here is that the build was broken and never fixed in v0.80.8; rpmbuild now complains because of a change in autobuild-ceph.

A possible solution is to roll back that change (from issues whirling around #11955), rebuild the version, and then put the rpmbuild change back. v0.80.8 should not need rebuilding anytime soon.

Actions
Copy link
 #5

Updated by Dan Mick over 8 years ago

(and we still want to test from v0.80.8 -> later versions, because v0.80.8 had a specific sort of pg breakage)

Actions
Copy link
 #6

Updated by Dan Mick over 8 years ago

even more awesomeness: v0.80.8 had a submodule that pointed to ceph.com/git.

Actions
Copy link
 #7

Updated by Dan Mick over 8 years ago

  • Status changed from New to Resolved
  • Assignee set to Dan Mick

So I had an inspiration, and it worked:

1) set git.ceph.com's IP addr in /etc/hosts, so that git://ceph.com really goes to git://git.ceph.com

2) temporarily add back the --define "_unpackaged_files_terminate_build 0" to build-ceph-rpm.sh

With those done, the build completed, and is available on gitbuilder.ceph.com now.

Actions
Copy link

Also available in: Atom PDF